Description

You will design and maintain scalable data pipelines using Databricks and AWS for big data processing.

This role is on-site.

Responsibilities

• Build and optimize ETL/ELT workflows on AWS to handle large-scale structured and unstructured datasets.
• Implement efficient data ingestion from multiple sources using batch and streaming architectures.
• Optimize Spark jobs and Databricks clusters to meet reliability and latency requirements.
• Automate and manage data workflows using orchestration tools like Airflow or AWS Step Functions.
• Ensure data quality, governance, and security best practices across all pipelines.

Required Skills

• 5+ years of experience in data engineering and big data processing.
• Hands-on expertise with Databricks and Apache Spark.
• Proficiency in PySpark for distributed data processing and SQL for querying.
• Strong experience with AWS services including S3, EMR, Glue, Lambda, and Redshift.
• Experience building and optimizing ETL/ELT pipelines.
• Familiarity with workflow orchestration tools like Apache Airflow or AWS Step Functions.
• Understanding of data warehousing concepts, data modeling, and data governance.
• Bachelor's or Master's degree in Computer Science, IT, or a related field.

Preferred Skills

• Experience troubleshooting and debugging production issues to improve system performance.
• Ability to collaborate effectively with data scientists, analysts, and business stakeholders.

Key Skills

Education

Any Graduate