Description

You will own ingestion and systems integration activities on the Databricks platform.

Responsibilities

• Integrate data from various source systems and tailor integration approaches on a per-system basis.
• Design, maintain, and oversee ETL/ELT pipelines on the Databricks platform.
• Schedule and automate ingestion jobs using Airflow.
• Document ingestion and integration flows and pipelines.
• Manage metadata and master data within the technical data catalog.
• Monitor pipelines and ensure adherence to SDLC best practices and security guidelines, including PII management.

Required Skills

• 2+ years of experience in data engineering, ingestion pipelining, and ETL/ELT.
• Hands-on experience with Databricks.
• Proficiency in Python, PySpark, SQL, Scala, or Spark.
• Experience with Apache Airflow and Mibo for pipeline management.
• Knowledge of CI/CD pipelines and Git version control.
• Experience working within Agile methodologies.
• Understanding of data quality guidelines and data lake ingestion processes.
• Bachelor's degree in Computer Science, Engineering, Statistics, or a related field.

Key Skills

Education

Any Graduate