Description

You will build complex, scalable end-to-end pipelines to modernize business data into a cloud-native Azure structure.

Responsibilities

• Design and develop scalable ETL pipelines in Azure Data Factory for data ingestion into Azure Data Lake Storage.
• Build and implement robust ETL processes to maintain data integrity across environments.
• Optimize pipelines to handle time-series data and support real-time analytics needs.
• Build curated datasets in Databricks using PySpark and SQL, applying complex transformations.
• Collaborate with BI and AI teams to ensure curated data meets advanced analytics requirements.

Required Skills

• 10+ years of professional experience in data engineering.
• Expert-level experience with Databricks, including PySpark and SQL.
• Proven ability to design and implement scalable ingestion pipelines using Azure Data Factory.
• Deep experience writing high-performance SQL queries from scratch.
• Experience curating datasets and applying complex pivot logic in Spark.
• Background working with time-series data and real-time use cases, preferably in Oil & Gas.
• Experience migrating legacy digital data to a modern Azure ecosystem.
• Proficiency in data ingestion, curation, and transformation using Python/PySpark.

Key Skills

Education

Any Graduate