Description

Lead the design and implementation of high-performance data ingestion pipelines using Azure Databricks and Apache Spark, managing the migration from Cloudera Public Cloud Hadoop to Azure.

This role is on-site.

Responsibilities

• Design and implement data ingestion pipelines from multiple sources using Apache Spark and Azure Databricks.
• Migrate data from Cloudera Public Cloud Hadoop to Azure Cloud environments.
• Integrate multiple databases including Oracle and SQL Server.
• Deliver and present proofs of concept for key technology components to project stakeholders.
• Execute programs using an onsite/offshore model and manage offshore teams.

Required Skills

• 12 to 15 years of professional experience in data engineering.
• Extensive experience with Azure Databricks, Apache Spark, and Databricks Spark.
• Proficiency in Python development and PySpark.
• High proficiency in SQL development.
• Hands-on experience with Oracle and SQL Server integration.
• Experience with SQL-on-Hadoop technologies such as Hive, Pig, Impala, Spark SQL, or Presto.
• Deep understanding of database engineering and design principles.
• Experience handling large, complex datasets from diverse sources.

Preferred Skills

• Familiarity with CI/CD methods.

Key Skills

Education

Any Graduate