Description

You will lead the design and implementation of data platforms and pipelines.

Responsibilities

• Ingest and model data from APIs, files/SFTP, and relational sources using PySpark/SQL and dbt.
• Design and operate data pipelines using Prefect (or Airflow), managing scheduling, retrieval, and SLAs.
• Build on cloud data platforms, leveraging S3/ADLS/GCS for storage and Spark for compute.
• Publish governed data services, managing lifecycle via APIM authentication and policies.
• Enforce data quality and governance through contracts, validations, lineage, and alerting.

Required Skills

• 10+ years of professional data engineering experience.
• Strong proficiency in Python, SQL, and Spark (PySpark).
• Experience with Snowflake/Databricks, including Delta formats and workflows.
• Hands-on building of ETL/ELT processes using Prefect (or Airflow) and dbt.
• Familiarity with AWS data services (EMR/Glue, S3, Lambda) and Azure services.
• Experience onboarding datasets to cloud data platforms (storage, compute, security).
• Proficiency with Spark, PySpark, and SQL.
• Knowledge of data governance practices, including PII handling and encryption.
• Experience with Git-based workflows and CI/CD practices.

Key Skills

Education

Any Graduate