Description

You will build and maintain big data processing systems using PySpark, ensuring data quality and integrity across ETL pipelines. You will troubleshoot application issues, document code lineage, and integrate with ingestion frameworks while adhering to security and privacy standards.

This role is hybrid.

Responsibilities

• Implement and optimize ETL pipelines and data transformation processes using PySpark.
• Troubleshoot and resolve issues within PySpark applications and distributed computing workflows.
• Document code lineage and understand data flow dependencies from converted PySpark code.
• Integrate PySpark code with frameworks like Ingestion Framework and DataLens.
• Ensure compliance with data security, privacy regulations, and organizational standards.

Required Skills

• 4+ years of experience in big data development using Hadoop, Hive, and Spark frameworks.
• Strong programming skills in Python, PySpark, and SQL.
• Experience with big data technologies including Hadoop, Hive, and Kafka.
• Understanding of data warehousing concepts and relational databases.
• Knowledge of CI/CD pipelines and DevOps practices.
• Ability to demonstrate and document code lineage.

Preferred Skills

• Certification in big data or cloud technologies.
• Experience with SAS.

Key Skills

Education

Any Graduate