Description

You will design and maintain distributed computing systems and large-scale data processing workflows.

This role is hybrid.

Responsibilities

• Implement ETL pipelines and data transformation processes using PySpark.
• Troubleshoot and resolve issues within PySpark applications and workflows.
• Ensure data quality, integrity, and compliance with security and privacy regulations.
• Document code lineage and map data flows from converted PySpark code.
• Integrate PySpark code with existing frameworks like Ingestion Framework and DataLens.

Required Skills

• 5+ years of experience in big data engineering.
• Proficiency in Python and SQL.
• Hands-on experience with PySpark and Spark frameworks.
• Practical knowledge of Hadoop, Hive, and Kafka.
• Experience building and managing ETL pipelines.
• Understanding of data warehousing concepts and relational databases.
• Familiarity with CI/CD pipelines and DevOps practices.
• Ability to manage source dependencies and data flows.

Any Graduate

Key Skills

Education

Any Gradute