← Back to jobs

New York Technology Partners Logo
Azure Data Engineer
Posted On: 30+ days ago
Experience: 10+ years
Availability: Hybrid
Openings: 1
Category: Azure Data Engineer
Tenure: No Preference/Any
Related Jobs

No related jobs found

Description

You will build complex, scalable end-to-end pipelines to modernize business data into a cloud-native Azure structure.

Responsibilities

  • Design and develop scalable ETL pipelines in Azure Data Factory for data ingestion into Azure Data Lake Storage.
  • Build and implement robust ETL processes to maintain data integrity across environments.
  • Optimize pipelines to handle time-series data and support real-time analytics needs.
  • Build curated datasets in Databricks using PySpark and SQL, applying complex transformations.
  • Collaborate with BI and AI teams to ensure curated data meets advanced analytics requirements.

Required Skills

  • 10+ years of professional experience in data engineering.
  • Expert-level experience with Databricks, including PySpark and SQL.
  • Proven ability to design and implement scalable ingestion pipelines using Azure Data Factory.
  • Deep experience writing high-performance SQL queries from scratch.
  • Experience curating datasets and applying complex pivot logic in Spark.
  • Background working with time-series data and real-time use cases, preferably in Oil & Gas.
  • Experience migrating legacy digital data to a modern Azure ecosystem.
  • Proficiency in data ingestion, curation, and transformation using Python/PySpark.

Education

Any Graduate

Related Jobs

No related jobs found

← Back to jobs