Description

Design and implement data lake solutions and orchestration pipelines within an AWS environment.

Responsibilities

• Provide architectural solutions and technical recommendations to clients.
• Implement AWS Data Lakes and data publication workflows using Databricks, Airflow, and S3.
• Build data engineering pipelines and lake solutions using Databricks and AWS services.
• Manage job orchestration, dependency setup, and scheduling via Airflow.
• Configure and manage Databricks clusters, SQL warehouses, and Unity Catalog consumption patterns.

Required Skills

• 8+ years of experience in data engineering or architecture.
• Expert-level proficiency in Databricks and PySpark.
• Advanced SQL skills including PL/SQL, complex joins, and aggregation functions.
• Strong experience with Airflow for orchestration and scheduling.
• Hands-on experience with DBT using Python and SQL.
• Proficiency in Python, DataFrames, and Spark.
• Experience handling Delta and Parquet file formats.
• Knowledge of AWS infrastructure including S3, IAM roles, and Secrets Manager.
• Experience with GitHub and CI/CD pipelines.

Preferred Skills

• Bachelor's degree.

Key Skills

Education

Bachelor's degree