Description

You will design, build, and maintain scalable data pipelines within the AWS and Databricks ecosystems.

This role is remote.

Responsibilities

• Build and maintain data pipelines using Delta Live Tables and PySpark.
• Manage infrastructure provisioning and configuration with Terraform.
• Implement and monitor data migration processes, including Change Data Capture (CDC) from on-premises sources to AWS.
• Optimize AWS services and Databricks jobs for performance and cost efficiency.
• Develop and maintain CI/CD pipelines using Azure DevOps principles.

Required Skills

• 5+ years of experience in a Data Engineering role.
• Expert proficiency with AWS core services: EC2, S3, RDS, Lambda, and VPC.
• Strong analytical skills in PySpark and Spark SQL for big data processing.
• Deep experience with Databricks, including Notebooks and Workflow setup.
• Proficiency in CI/CD automation using Azure Pipelines and Git/Azure Repos.
• Experience migrating databases using AWS DMS and implementing CDC patterns.
• Familiarity with Infrastructure as Code using Terraform for AWS/Azure.
• Scripting capability using PowerShell, Bash, or Python.
• Understanding of AWS security best practices, IAM, and encryption.

Key Skills

Education

Any Graduate