Description

Design, develop, and maintain data pipelines within an AWS and Databricks ecosystem.

Responsibilities

• Design and monitor data pipelines using Databricks, Delta Lake, Python, SQL, and Starburst.
• Automate testing for data pipelines and configure CI/CD workflows using GitLab.
• Integrate Databricks with AWS services including SNS, SQS, and MSK.
• Optimize data processing and query performance for large-scale datasets.
• Establish data quality checks and troubleshoot complex issues on the AWS Databricks platform.

Required Skills

• 5+ years of experience in data engineering roles.
• Proficiency with Databricks, Spark, PySpark, and Starburst/Trino.
• Strong Python and SQL skills for data analysis and pipeline development.
• Hands-on experience with AWS services: S3, RDS, Lambda, SQS, SNS, and MSK.
• Experience using GitLab for CI/CD.
• Knowledge of data pipeline orchestration tools.
• Bachelor’s or Master’s degree in Computer Science, Information Technology, or a related field.

Preferred Skills

• Experience with ETL tools such as Informatica PowerCenter.

Key Skills

Education

Bachelor’s or Master’s degree