Description

You will design, build, and optimise large-scale data pipelines for batch and streaming workloads using PySpark and Python.

This role is remote.

Responsibilities

• Develop production-grade PySpark applications for complex transformations, window functions, and efficient multi-dataset joins.
• Build Structured Streaming pipelines with Kafka, Azure Event Hubs, or GCP Pub/Sub, handling late-arriving data and delivery semantics.
• Engineer Python-based data services, automation scripts, and REST API integrations for SAP OData, Salesforce, and third-party sources.
• Implement and manage lakehouse architectures on Azure Databricks, GCP Dataproc, and AWS EMR using Delta Lake or Apache Iceberg.
• Apply advanced Spark performance tuning, including partition optimisation, skew mitigation, and AQE tuning, while ensuring data quality with Great Expectations.

Required Skills

• 5+ years of experience in PySpark and Python development for data engineering.
• Deep expertise in PySpark DataFrame operations, UDFs, Pandas UDFs, and Structured Streaming.
• Proficiency with cloud data platforms: Azure Databricks, GCP Dataproc, or AWS EMR.
• Strong knowledge of Delta Lake features (ACID, Time Travel, Z-Order) and medallion architecture patterns.
• Experience with orchestration tools like Airflow and Databricks Workflows.
• Solid software engineering practices: pytest, type hints, modular design, and dependency management.
• Familiarity with data ingestion from SAP S/4HANA, Salesforce, and various streaming/file formats.

Preferred Skills

• Experience with Unity Catalog for governance and lineage.
• Knowledge of Apache Iceberg or Hudi alongside Delta Lake.

Key Skills

Education

Any Graduate