Description

You will own data ingestion and systems integration activities on the Databricks platform.

Responsibilities

• Integrate data from various source systems and tailor integration approaches per system.
• Design, maintain, and oversee ETL/ELT pipelines on the Databricks platform.
• Schedule and automate ingestion jobs using Apache Airflow.
• Document ingestion and integration flows and pipelines.
• Manage metadata and master data within the technical data catalog.

Required Skills

• 2+ years of experience in data engineering, ingestion pipelining, and ETL/ELT.
• Hands-on experience with Databricks.
• Proficiency in Python, PySpark, SQL, Scala, or Spark.
• Experience with Apache Airflow and Mibo for pipeline management.
• Knowledge of CI/CD pipelines and Git version control.
• Experience working within Agile methodologies.
• Understanding of data quality guidelines and data lake ingestion processes.

Preferred Skills

• Experience with PII management and security guidelines.

Key Skills

Education

Any Graduate