Description

You will design, develop, and optimize data pipelines for an AWS Redshift-based data lakehouse.

Responsibilities

• Architect and manage end-to-end data pipelines to move data from Workday to AWS Redshift.
• Optimize Redshift performance through query tuning and database optimization.
• Build and implement workflows using Airflow.
• Automate tasks and develop reusable frameworks for data ingestion and movement.
• Maintain documented code repositories and configurations using Git/Bitbucket.

Required Skills

• 5+ years of experience with advanced hands-on AWS data lake solutions.
• Proficiency in Python, specifically for developing Airflow DAGs and working within complex object-oriented platforms.
• Experience with PySpark and Glue ETL scripting, including dataframe transformations.
• Competency in developing CloudFormation templates and YAML-defined IAM policies.
• Hands-on experience with AWS services including Redshift, DMS, Glue, Lambda, S3, and Athena.
• Experience with event-based data processing using SQS.
• Ability to debug serverless applications using CloudWatch Logs, Log Insights, and CloudTrail.
• Strong understanding of ETL best practices, data modeling, and data transformation.
• Proficiency in SQL and Redshift stored procedures.

Preferred Skills

• Experience with Docker, Parquet file formats, and Jupyter Notebooks.
• Knowledge of API best practices, API Gateway, and standard authentication protocols.
• Experience with CDC and SCD logic or converting Oracle stored procedures to Redshift.

Key Skills

Education

Any Graduate