Description

Lead the architecture and delivery of large-scale data Lakehouse solutions on Databricks and AWS.

This role is remote.

Responsibilities

• Design reference architectures for batch, real-time, and streaming data pipelines using Databricks, Delta Lake, and AWS.
• Build, optimize, and manage large-scale PySpark and SQL pipelines for high-volume data ingestion and curation.
• Establish standards for data lineage, metadata management, and compliance while implementing security best practices such as RBAC and encryption.
• Lead and mentor data engineering teams, reviewing designs to ensure adherence to architectural principles and driving DevOps adoption.
• Collaborate with data scientists and stakeholders to enable advanced analytics and AI/ML use cases.

Required Skills

• 5+ years of experience in Data Engineering and Databricks.
• Strong proficiency in building pipelines with PySpark and SQL.
• Experience with real-time data processing using Kafka, Kinesis, or Event Hubs.
• Hands-on experience implementing CI/CD and DevOps practices for data engineering.
• Ability to architect scalable, secure, and cost-optimized solutions on AWS.
• Experience collaborating with Data Scientists and supporting AI/ML workflows.
• Bachelor's degree in Computer Science or related field.

Preferred Skills

• 3+ years in a lead or architectural role within data engineering teams.

Key Skills

Education

Bachelor's degree