Description

You will design, develop, and test data ingestion pipelines and end-to-end ETL processes for big data platforms.

This role is hybrid.

Responsibilities

• Lead the design and development of validation frameworks and automated test suites for data pipeline flows and integrity.
• Perform data profiling, analysis, and discovery to identify data types, formats, and quality issues.
• Develop tools to measure data quality and visualize anomaly patterns in source and processed data.
• Collaborate with architects and product managers to establish physical application frameworks, libraries, and modules.

Required Skills

• 10+ years of software development and testing experience.
• 4+ years of hands-on experience with Spark, HBase, Hive, Sqoop, Impala, Kafka, Flume, Oozie, MapReduce, or similar tools.
• 4+ years of programming experience in Scala, Java, or Python.
• Strong proficiency in Spark SQL and Scala development within Hadoop environments.
• Expertise in shell scripting and Web Services API development and testing.
• Experience building automated frameworks within CI/CD environments using Git and Jenkins.
• Proven experience in technical leadership and mentoring engineering teams.

Preferred Skills

• Experience with large-scale data migration projects.
• Hands-on experience with Google Cloud Platform (GCP).

Key Skills

Education

Any Graduate