Description

You will translate client requirements into technical designs and implement big data engineering components on Azure. You own the end-to-end data pipeline lifecycle, from ingestion to analytics.

Responsibilities

• Build data ingestion functionality from heterogeneous sources for batch and real-time processing.
• Design and implement components for data analytics, search, and aggregation.
• Develop data integration, transformation, and storage frameworks.
• Optimize data pipelines for performance and computational efficiency.
• Drive design discussions to ensure the health and performance of the overall data solution.

Required Skills

• 5+ years of experience in data engineering or related roles.
• Hands-on experience with the Hadoop stack including HDFS, Sqoop, Kafka, Pulsar, NiFi, Spark, Spark Streaming, Flink, Storm, Hive, Oozie, and Airflow.
• Strong programming proficiency in Java, Scala, and Python, with a preference for Java.
• Experience with NoSQL and MPP data platforms such as HBase, MongoDB, Cassandra, AWS Redshift, Azure SQLDW, or GCP BigQuery.
• Working knowledge of data platform-related services on Azure.
• Experience building end-to-end data pipelines, including real-time pipelines.
• Bachelor's degree or equivalent combination of education and experience.

Preferred Skills

• Knowledge of traditional ETL tools like Informatica or Talend.
• Experience with relational databases including Oracle, MySQL, SQL Server, or Postgres.
• Familiarity with data governance processes and tools such as Collibra or Alation.

Key Skills

Education

Any Graduate