Description

You will build and maintain scalable data pipelines on the AWS cloud.

Responsibilities

• Develop and maintain scalable data pipelines using Python, PySpark, and SQL.
• Design and implement data processing workflows using Databricks and distributed computing frameworks.
• Build and manage data solutions on AWS cloud services including S3, Glue, EMR, Redshift, and Lambda.
• Optimize data processing performance and SQL query efficiency for large datasets.
• Implement and manage data pipeline orchestration using Airflow or AWS Step Functions.

Required Skills

• Strong proficiency in Python for data engineering and pipeline development.
• Advanced knowledge of SQL, including writing complex queries and performance tuning.
• Hands-on experience with PySpark for large-scale data processing.
• Strong expertise in Databricks for data engineering and distributed processing.
• Experience working in AWS cloud environments (S3, Glue, EMR, Redshift, Lambda).
• Knowledge of data pipeline orchestration tools like Apache Airflow or AWS Step Functions.
• Solid understanding of data modeling, data warehousing concepts, and ETL/ELT processes.
• Experience using version control systems like Git and CI/CD practices.
• Bachelor's degree in Computer Science, IT, Engineering, or related field.

Key Skills

Education

Any Graduate