Description

Design, develop, and optimize data pipelines and ETL processes for an AWS Redshift-based data lake house.

Responsibilities

• Architect and manage end-to-end data pipelines to ensure data accuracy, reliability, and performance.
• Design ETL processes incorporating Change Data Capture (CDC) and Slowly Changing Dimension (SCD) logic.
• Implement efficient orchestration workflows using Airflow to coordinate complex processes.
• Optimize Redshift database performance through query tuning and stored procedure refinement.
• Automate repetitive tasks and develop reusable frameworks for data ingestion and transformation.

Required Skills

• 3+ years of experience designing and maintaining ETL/ELT pipelines on AWS Redshift.
• Proficiency in SQL programming and Redshift stored procedures.
• Hands-on experience with AWS DMS for ingesting data from on-premises databases.
• Strong Python scripting skills, specifically for developing Airflow DAGs and operators.
• Experience using AWS Glue, AWS Lambda, and Amazon S3.
• Practical knowledge of CDC and SCD strategies within complex ETL scenarios.
• Ability to convert Oracle scripts and stored procedures to Redshift equivalents.
• Experience with Git for version control and repository management.

Preferred Skills

• Bachelor’s Degree in Computer Science, Information Technology, or a related field.

Key Skills

Education

Bachelor’s Degree