Description

Key Responsibilities
Design and develop scalable ETL/ELT pipelines using Scala, Apache Spark, and Databricks.
Build batch and real-time streaming solutions with Apache Kafka and Spark Structured Streaming.
Implement Change Data Capture (CDC) and incremental data ingestion using Delta Lake.
Develop and maintain Databricks notebooks, jobs, workflows, and Lakehouse solutions.
Optimize Spark workloads for performance, scalability, and cost efficiency.
Implement data governance, security, and data lineage using Unity Catalog.
Troubleshoot production data pipelines, data quality issues, and performance bottlenecks.
Collaborate with architects and cross-functional teams to deliver production-ready data solutions

Key Skills

Education

Bachelor's degree