Description

You will build and maintain big data processing systems using PySpark and Python.

Responsibilities

• Implement ETL pipelines and data transformation processes using big data frameworks.
• Troubleshoot and resolve issues within PySpark applications and workflows.
• Ensure data quality and integrity across all data processing workflows.
• Integrate PySpark code with frameworks like Ingestion Framework and DataLens.
• Document code lineage and ensure compliance with data security standards.

Required Skills

• 5+ years of experience in big data development.
• Expert proficiency in PySpark and Python programming.
• Strong command of SQL and relational database concepts.
• Hands-on experience with Hadoop, Hive, and Spark Framework.
• Familiarity with ETL Pipelines and distributed computing systems.
• Experience implementing data flows using Kafka.
• Working knowledge of CI/CD Pipelines and DevOps practices.
• Understanding of data warehousing concepts.

Key Skills

Education

Any Graduate