Description

Key Skills: Pyspark, MySQL, Hive, Spark

Roles and Responsibilities:

• Design, develop, and maintain scalable data pipelines using PySpark for batch processing.
• Build ETL/ELT workflows to process large volumes of structured and unstructured data.
• Implement data transformation logic using Python and Spark SQL, including data validation and quality checks.
• Optimize Spark jobs for performance, scalability, and reliability; troubleshoot failures and bottlenecks.
• Collaborate with cross-functional teams to deliver data solutions and support production operations.

Skills Required:

• 6-12 years of experience in Data Engineering or Big Data development.
• PySpark for distributed data processing and pipeline development.
• Strong Python fundamentals for data engineering workflows.
• Spark SQL and ETL development for scalable transformations.
• Performance tuning, debugging, and reliability practices for production Spark jobs.

Good to Have:

• Apache Spark ecosystem experience (Spark).
• Hive for data warehousing and querying.
• MySQL for relational data integration.

Education: Degree in Computer Science, Engineering, or related field (or equivalent experience)

Key Skills

Education

Any Graduate