Description

You will design and build real-time, high-throughput data pipelines using Databricks across Azure, AWS, and GCP.

Responsibilities

• Design and implement real-time streaming and batch data pipelines using Databricks and Spark.
• Architect and optimize data lakehouse platforms on Azure, AWS, and GCP.
• Lead end-to-end architecture for ingestion, transformation, enrichment, and publishing of large-scale structured and unstructured data.
• Drive infrastructure-as-code adoption using Terraform and CI/CD to automate platform provisioning.
• Apply best practices for data quality, lineage, governance, and security including RBAC and encryption.

Required Skills

• 5+ years of experience in data architecture or engineering.
• Deep expertise in Databricks (Azure, AWS, GCP) including cluster management, Delta Lake, and Unity Catalog.
• Strong proficiency in Scala for high-performance data processing.
• Hands-on experience with Apache Spark and Structured Streaming.
• Experience with cloud services including Azure (ADF, Synapse, Event Hubs), AWS (S3, Glue, Kinesis, Redshift, IAM), and GCP (GCS, BigQuery, Dataflow).
• Knowledge of Delta Lake architecture, schema evolution, and ACID transactions.
• Proficiency in SQL and Python for scripting and notebooks.
• Experience with orchestration tools such as Airflow, dbt, or Dagster.
• Infrastructure-as-Code experience using Terraform, Pulumi, or ARM templates.
• Implementation of security protocols including Private Link, VNet, and encryption at rest/in transit.

Preferred Skills

• Experience working with financial institutions or in highly regulated environments.
• Familiarity with machine learning pipelines on Databricks.

Key Skills

Education

Any Graduate