Description

Key Skills: Data Engineer, Spark, Python, AWS, Pyspark, Airflow, SQL, Snowflake, DBT

Roles and Responsibilities:

• Design and implement data pipelines for data migration to AWS
• Utilize Python and PySpark for data processing and transformation
• Optimize data workflows using DBT and Apache Airflow
• Ensure data integrity and security during migration processes
• Collaborate with cross-functional teams to understand data requirements
• Monitor and troubleshoot data pipeline performance issues
• Stay updated with the latest technologies in the Big Data tech stack

Skills Required:

• Strong experience in designing and building scalable data pipelines
• Hands-on experience with Python and PySpark for data processing
• Strong understanding of Apache Spark architecture and performance tuning
• Experience in AWS data services (S3, Glue, EMR, Redshift, etc.)
• Proficiency in SQL for data querying and transformation
• Experience with workflow orchestration tools like Apache Airflow
• Knowledge of DBT for data transformation and modeling
• Experience working with Snowflake or similar cloud data warehouses
• Strong understanding of data migration strategies (on-prem to cloud)
• Knowledge of data quality, validation, and governance practices
• Strong debugging and performance optimization skills
• Ability to work in Agile teams and collaborate with stakeholders

Education:

• B.E. / B.Tech / BCA / B.Sc. / M.Tech / M.E. / M.Sc. / MCA in Information Technology or related fields
• Equivalent practical experience in Data Engineering will also be considered

Key Skills

Education

Any Graduate