Description

Develop and maintain ETL pipelines and batch data workflows using Databricks notebooks on AWS.

This role is on-site.

Responsibilities

• Build data processing jobs using PySpark and Spark SQL, including joins, aggregations, filtering, and validation.
• Read from files and write to Databricks Delta tables, applying Delta Lake versioning and time travel.
• Ingest and process data using AWS services such as S3, AWS Glue, AWS Lambda, and Amazon Redshift.
• Implement reusable notebooks and utility components within the Databricks workspace.
• Troubleshoot failures and performance issues under guidance.

Required Skills

• 5-11 years of relevant experience in data engineering and ETL development.
• Hands-on experience with AWS and Databricks on AWS for building data workflows.
• Proficiency in PySpark and Spark SQL for batch processing and transformations.
• Knowledge of Delta Lake fundamentals, including managed vs. external tables, versioning, and time travel.
• Experience with ETL fundamentals and AWS data services integration (S3, AWS Glue, AWS Lambda, Amazon Redshift).
• Familiarity with basic IAM, Git, Jenkins, and CI/CD basics.

Key Skills

Education

Any Graduate