Description

You will build and maintain data pipelines using Python and PySpark within an AWS environment.

Responsibilities

• Develop ETL processes and technical specifications (HLD/LLD).
• Perform coding, unit testing, and SQL database programming.
• Execute data mining, analysis, and profiling tasks.
• Implement workload automation and CI/CD processes.

Required Skills

• 5+ years of experience in Data Engineering.
• Proficiency in Python and PySpark for ETL development.
• Hands-on experience with AWS services: Glue, Lambda, MSK (Kafka), S3, Step functions, RDS, and EKS.
• Strong SQL skills including performance tuning, stored procedures, views, functions, and triggers.
• Experience with databases: SQL Server, Oracle, and Sybase.
• Experience working with REST APIs.
• Knowledge of UNIX scripting and Oracle SQL/PL-SQL.
• Experience with CI/CD tools such as Bitbucket, GitHub, and Jenkins.
• Practical application of Agile/SCRUM methodologies.

Preferred Skills

• Experience with ERWIN, Tableau, or Power BI.
• Knowledge of MDM, Data Warehousing, or Data Science concepts.
• Familiarity with ETL tools like DataStage, Informatica, or Pentaho.

Key Skills

Education

Bachelor's degree