Description

Design, develop, and maintain scalable data pipelines using PySpark for batch processing.

This role is remote.

Responsibilities

• Build ETL/ELT workflows to process large volumes of structured and unstructured data.
• Implement data transformation logic using Python and Spark SQL, including data validation and quality checks.
• Optimize Spark jobs for performance, scalability, and reliability.
• Troubleshoot failures and bottlenecks in production environments.
• Collaborate with cross-functional teams to deliver data solutions.

Required Skills

• 6+ years of experience in Data Engineering or Big Data development.
• Strong proficiency in PySpark for distributed data processing and pipeline development.
• Strong Python fundamentals for data engineering workflows.
• Experience with Spark SQL and ETL development for scalable transformations.
• Performance tuning, debugging, and reliability practices for production Spark jobs.
• Hands-on experience with MySQL for relational data integration.
• Knowledge of Hive for data warehousing and querying.

Preferred Skills

• Deep expertise within the Apache Spark ecosystem.
• Degree in Computer Science, Engineering, or related field (or equivalent experience).

Key Skills

Education

Any Graduate