Description

You will design and implement data pipelines and ETL/ELT workflows within the AWS ecosystem.

Responsibilities

• Design and implement data pipelines and ETL/ELT workflows using Databricks notebooks and AWS services.
• Develop scalable, reusable components for data ingestion, transformation, and validation.
• Optimize Spark jobs for performance, scalability, and cost-efficiency on Databricks.
• Build and maintain Delta Lake tables, streaming jobs, and time-based window aggregations.
• Ensure data governance, quality, lineage, and security compliance across all pipelines.

Required Skills

• 5+ years of experience in big data engineering.
• 1–2 years of hands-on experience using Databricks on AWS.
• Proficiency in PySpark, Spark SQL, and Delta Lake.
• Strong experience with AWS services including S3, Glue, Lambda, Athena, EMR, and Redshift.
• Familiarity with Databricks Workspaces, Jobs, Clusters, and DBFS.
• Experience building streaming and batch pipelines using Spark Structured Streaming or Kafka.
• Solid understanding of data modeling, data lakehouse architecture, and distributed system performance tuning.
• Knowledge of CI/CD pipelines, Git, and Infrastructure as Code tools like Terraform or CloudFormation.
• Strong SQL skills for working with large-scale datasets.

Key Skills

Education

Any Graduate