Description

You will translate client requirements into technical designs and implement big data engineering components on Azure, owning the end-to-end data pipeline lifecycle from ingestion to analytics.

Responsibilities

• Build data ingestion functionality from heterogeneous sources for batch and real-time processing.
• Design and implement components for data analytics, search, and aggregation.
• Develop data integration, transformation, and storage frameworks.
• Optimize data pipelines for performance and computational efficiency.
• Drive design discussions to ensure the health and performance of the overall data solution.

Required Skills

• 5+ years of experience in data engineering or related roles.
• Hands-on experience with the Hadoop stack: HDFS, Sqoop, Kafka, Pulsar, NiFi, Spark, Spark Streaming, Flink, Storm, Hive, Oozie, and Airflow.
• Strong programming proficiency in Java, Scala, and Python (Java preferred).
• Experience with NoSQL and MPP data platforms: HBase, MongoDB, Cassandra, AWS Redshift, Azure SQLDW, or GCP BigQuery.
• Working knowledge of data platform-related services on Azure.
• Experience building end-to-end data pipelines, including real-time pipelines.

Preferred Skills

• Knowledge of traditional ETL tools like Informatica or Talend.
• Experience with relational databases: Oracle, MySQL, SQL Server, or Postgres.
• Familiarity with data governance processes and tools such as Collibra or Alation.

Key Skills

Education

Any Graduate