Description

You will lead the migration of AWS-hosted PostgreSQL workloads to a Databricks Lakehouse architecture.

Responsibilities

• Lead the migration of PostgreSQL workloads from AWS to Databricks.
• Design and implement scalable ETL/ELT pipelines using PySpark, SQL, and Delta Lake.
• Process complex healthcare data formats including X12 837, EBCDIC, and HL7.
• Implement data masking, profiling, and governance to ensure HIPAA compliance.
• Optimize data pipelines for performance and cost-efficiency within the AWS ecosystem.
• Drive data quality monitoring and reconciliation in high-volume, sensitive environments.

Required Skills

• 8+ years of data engineering experience with a focus on AWS and Databricks.
• Databricks Certified Data Engineer (Associate or Professional).
• Proven experience migrating PostgreSQL workloads to lakehouse platforms.
• Proficiency in PySpark, SQL, and Delta Lake.
• Experience with AWS ingestion services including S3, Glue, and Lambda.
• Expertise in data masking and governance within HIPAA-regulated environments.
• Working knowledge of healthcare data formats such as X12 837 and EBCDIC.
• Familiarity with CI/CD processes in a cloud ecosystem.

Preferred Skills

• Experience with Unity Catalog and Lakehouse governance frameworks.
• Familiarity with FHIR or clinical data standards.
• Prior experience working with healthcare payers, providers, or clearinghouses.

Key Skills

Education

Any Graduate