Description

You will design, build, and maintain data pipelines in an AWS-centric environment, ensuring performance optimization and reliable production support.

This role is on-site.

Responsibilities

• Design and implement ETL and ELT processes with robust error handling and re-processing capabilities.
• Manage end-to-end data pipelines and provide ongoing production operations support.
• Develop stored procedures and User Defined Functions (UDF) using Python and SQL.
• Perform code reviews to maintain engineering best practices and code quality.
• Collaborate with BI Engineers to ensure efficient and accessible data delivery.

Required Skills

• 8+ years of total professional experience in data engineering.
• Extensive experience with AWS native services: S3, Glue, Lambda, EMR, Athena, DynamoDB, CloudFormation, and Step Functions.
• Strong expertise in Apache Spark and PySpark for big data processing.
• Hands-on experience with Databricks and big data ecosystems.
• Proficiency in Python and SQL for data processing and function development.
• Experience with InfoSphere DataStage and ETL workflows.
• Familiarity with cloud data integration tools such as Matillion, NiFi, Fivetran, HVR, SnapLogic, or Talend.

Preferred Skills

• Advanced SQL programming capabilities.
• Experience with Amazon Redshift.
• Proficiency in code versioning using Git or SVN.

Key Skills

Education

Any Graduate