Description

Build and maintain batch or real-time data pipelines in data platform environments.

This role is on-site.

Responsibilities

• Develop ETL processes to extract and manipulate data from multiple sources.
• Automate data workflows including ingestion, aggregation, and ETL processing.
• Monitor and optimize data infrastructure for accurate extraction, transformation, and loading.
• Perform feature engineering and data transformations to enable machine learning models.
• Ensure data accuracy, integrity, privacy, security, and compliance through quality control.

Required Skills

• 8+ years of relevant experience in Data Engineering.
• Strong proficiency in Python, PySpark, and SQL.
• Experience building data pipelines and ETL processes.
• Knowledge of Big Data technologies and AWS.
• Understanding of Hive and data warehousing concepts.
• Proficiency in object-oriented programming and data analysis.
• Experience with data ingestion tools (Talend or Ab Initio).
• Scripting skills in Bash or UNIX Shell.

Preferred Skills

• Experience with data lakehouse architecture (AWS Cloud, Airflow, Starburst, Iceberg).
• Working knowledge of Impala, S3, and HDFS.
• Ability to apply machine learning and AI concepts for predictive analysis.

Key Skills

Education

Any Graduate