Description

Build and maintain scalable data pipelines on AWS using Python, PySpark, and SQL.

This role is on-site.

Responsibilities

• Develop and maintain ETL/ELT jobs in Databricks for ingestion, transformation, and loading.
• Design data models and write efficient SQL for analytics and reporting.
• Implement data quality checks, monitoring, and troubleshooting for batch and streaming workflows.
• Integrate data sources and sinks on AWS to ensure secure and reliable data movement.
• Optimize pipeline performance for large datasets.

Required Skills

• 5-10 years of relevant experience in data engineering.
• Python for building data processing and automation components.
• PySpark for distributed transformations and scalable ETL/ELT processing.
• SQL for querying, data modeling, and performance tuning.
• AWS services for data engineering workloads and cloud-based data pipelines.
• Databricks for running Spark-based workloads and managing data pipelines.

Preferred Skills

• Hadoop for working with Hadoop-based data ecosystems.
• Kafka for streaming ingestion and event-driven data pipelines.

Education: Bachelor's Degree in Computer Science or related field

Key Skills

Education

Any Graduate