Description

You will build and maintain scalable data pipelines and API integrations to manage increasing data volume and complexity.

Responsibilities

• Develop Spark, Spark SQL, and Hive SQL jobs to populate defined data models.
• Design data integrations and frameworks for monitoring data quality.
• Implement processes to ensure production data accuracy and availability for stakeholders.
• Collaborate with frontend/backend engineers, product managers, and analysts to develop long-term data platform architecture.
• Perform data analysis to troubleshoot and resolve data-related issues.

Required Skills

• Hands-on experience with Spark.
• Proficiency in Spark SQL.
• Experience writing Hive SQL jobs.
• Ability to design data models and integrations.
• Experience designing data quality frameworks.
• Experience evaluating open source and vendor tools for data lineage.
• Ability to work within a cross-functional engineering team.
• Write unit and integration tests while maintaining engineering documentation and wikis.

Preferred Skills

• Degree in any field.

Key Skills

Education

Any graduate