Description

You will build and manage data pipelines within an AWS-centric environment.

Responsibilities

• Design and implement ETL and ELT processes with a focus on error handling, re-processing, and performance optimization.
• Manage data pipelines and provide ongoing production and operations support.
• Perform code reviews to establish and maintain engineering best practices.
• Collaborate with BI Engineers to ensure efficient and accessible data delivery.
• Develop stored procedures and User Defined Functions (UDF) using Python and SQL.

Required Skills

• 8+ years of total professional experience in data engineering.
• Extensive experience with AWS native services including S3, Glue, Lambda, EMR, Athena, Dynamo, CloudFormation, and Step Functions.
• Proficiency in Python and SQL for data processing and function development.
• Hands-on experience with Databricks and Big Data ecosystems.
• Strong expertise in Apache Spark and PySpark.
• Experience with InfoSphere DataStage and ETL workflows.
• Familiarity with cloud data integration tools such as Matillion, NiFi, Fivetran, HVR, SnapLogic, or Talend.

Preferred Skills

• Advanced SQL programming capabilities.
• Experience with Amazon Redshift.
• Proficiency in code versioning using Git or SVN.

Key Skills

Education

Any Graduate