Description

Lead the architecture and delivery of large-scale data Lakehouse solutions on Databricks and AWS. You will design end-to-end data pipelines, mentor engineering teams, and ensure platform scalability, security, and cost-efficiency.

Responsibilities

• Design reference architectures for batch, real-time, and streaming data pipelines using Databricks, Delta Lake, and AWS cloud-native services.
• Build, optimize, and manage large-scale PySpark and SQL pipelines to handle high-volume data ingestion and curation.
• Establish standards for data lineage, metadata management, and compliance while implementing security best practices such as RBAC, data masking, and encryption.
• Lead and mentor a team of data engineers, reviewing designs to ensure adherence to architectural principles and driving the adoption of DevOps and CI/CD practices.
• Collaborate with data scientists and business stakeholders to enable advanced analytics and AI/ML use cases.

Required Skills

• 5+ years of experience in Data Engineering and Databricks.
• Strong proficiency in building pipelines with PySpark and SQL.
• Experience with real-time data processing using Kafka, Kinesis, or Event Hubs.
• Hands-on experience implementing CI/CD and DevOps practices for data engineering.
• Ability to architect scalable, secure, and cost-optimized solutions on AWS.
• Experience collaborating with Data Scientists and supporting AI/ML workflows.
• Bachelor's degree in Computer Science or related field.

Preferred Skills

• 3+ years in a lead or architectural role within data engineering teams.

Key Skills

Education

Bachelor's degree