Description

You will build and maintain data pipelines using PySpark and orchestration tools.

Responsibilities

• Process large volumes of structured and unstructured data from disparate sources.
• Develop and deploy data transformations using PySpark.
• Orchestrate data workflows using Airflow.
• Apply software engineering best practices to data solutions.

Required Skills

• 7+ years of professional experience in data engineering.
• Expert proficiency in Python and PySpark.
• Strong understanding of Spark architecture and implementation.
• Advanced SQL skills, with preference for Oracle.
• Experience with Airflow for workflow management.
• Solid grasp of OOP and Functional Programming paradigms.
• Proficiency with Git (Github) for version control.
• Experience processing data from multiple integration points.

Preferred Skills

• Knowledge of AWS data services.
• Familiarity with Jenkins for CI/CD.

Key Skills

Education

Any Graduate