Description

You will design, build, and maintain scalable data pipelines and models using Python, PySpark, and SQL on AWS and Databricks.

This role is on-site.

Responsibilities

• Develop and maintain data ingestion, transformation, and loading pipelines using Python, PySpark, and SQL.
• Build and optimize scalable ETL/ELT jobs in Databricks for large datasets.
• Design data models and write efficient SQL for analytics consumption and reporting.
• Implement data quality checks, monitoring, and troubleshooting for batch and streaming workflows.
• Integrate data sources and sinks on AWS to ensure secure and reliable data movement.

Required Skills

• 5-10 years of relevant experience in data engineering.
• Proficiency in Python for building data processing and automation components.
• Experience with AWS services for data engineering workloads and cloud-based data pipelines.
• Strong expertise in PySpark for distributed transformations and scalable ETL/ELT processing.
• Advanced SQL skills for querying, data modeling, and performance tuning.
• Hands-on experience with Databricks for running Spark-based workloads and managing data pipelines.

Preferred Skills

• Experience with Hadoop ecosystems.
• Knowledge of Kafka for streaming ingestion and event-driven data pipelines.

Key Skills

Education

Any Graduate