Description

Key Responsibilities:

• Design, develop, and maintain scalable data pipelines using Databricks and Apache Spark.
• Build ingestion frameworks and pipelines for batch and near real-time data processing.
• Implement Medallion Architecture (Bronze, Silver, Gold layers) using Delta Lake.
• Develop and optimize PySpark and SQL-based transformations for large-scale datasets.
• Implement data quality checks, monitoring, and observability within data pipelines.
• Design and implement reusable data engineering frameworks and CI/CD deployment processes.
• Enable self-service analytics using semantic and business metric layers.
• Participate in performance tuning, troubleshooting, and production support activities.

Primary Skills (Must Have)

• Databricks (Data Engineering)
• PySpark & Spark SQL
• Delta Lake & Medallion Architecture
• Lakeflow Connect / DLT (Lakeflow Pipelines)

Secondary Skills (Preferred)

• Delta Sharing
• Metric Views
• Lakebase

Key Skills

Education

Bachelor's degree