Description

• Design, develop, and maintain scalable data pipelines using Python and PySpark.
• Build and optimize ETL/ELT processes on AWS cloud platforms.
• Develop batch and real-time data processing solutions.
• Work with large-scale structured and unstructured datasets.
• Create and maintain Data Lakes and Data Warehouse solutions.
• Perform data quality validation, monitoring, and troubleshooting.
• Collaborate with business stakeholders, architects, and development teams to gather requirements and deliver data solutions.
• Optimize Spark jobs for performance, scalability, and cost efficiency.
• Participate in code reviews and ensure adherence to coding standards.
• Support production deployments and resolve critical incidents.

Qualifications:
Relevant Experience:

• 6–12 years of overall IT experience.
• Minimum 4+ years of hands-on experience in Python and PySpark development.
• Experience in AWS-based data engineering projects.
• Strong exposure to Data Warehousing and Data Lake architectures.

Education:

• BE / B.Tech / MCA / M.Tech or equivalent.

Must-Have Skills:

• Strong programming experience in Python
• Hands-on expertise in PySpark
• Experience with AWS services such as:
  • S3
  • EMR
  • Glue
  • Lambda
  • IAM
  • Athena
  • Redshift
• Strong SQL and Data Warehousing concepts
• Experience in building ETL/ELT pipelines
• Data Lake/Data Warehouse implementation experience
• Git/GitHub version control
• Performance tuning and optimization of Spark applications
• Understanding of Agile development methodologies

Good-to-Have Skills:

• Databricks
• Apache Airflow
• dbt
• Snowflake
• Terraform
• Kafka
• CI/CD Pipelines
• Docker/Kubernetes

Key Skills

Education

Bachelor's degree