Description

You will manage data ingestion and systems integration activities specifically within the Databricks platform.

Responsibilities

• Integrate data from diverse source systems by tailoring approaches to specific system requirements.
• Design, maintain, and oversee ETL/ELT pipelines on Databricks.
• Automate and schedule ingestion jobs using Apache Airflow.
• Document all ingestion and integration flows, pipelines, and metadata.
• Monitor pipeline performance and ensure adherence to SDLC best practices and security guidelines.

Required Skills

• 5+ years of experience in data engineering, ingestion pipelining, and ETL/ELT.
• Hands-on experience with Databricks.
• Proficiency in Python, PySpark, SQL, Spark, or Scala.
• Experience with Apache Airflow and Mibo.
• Practical knowledge of CI/CD pipelines.
• Working experience with Git and version control.
• Experience operating within Agile environments.
• Understanding of data quality guidelines and PII management.

Preferred Skills

• Experience managing metadata and master data in technical data catalogs.

Key Skills

Education

Any Graduate