Description

You will lead the engineering of big data platforms and resilient data pipelines.

Responsibilities

• Build and manage modern, secure data pipelines for the collection, integration, and transformation of structured and unstructured data.
• Implement automated data controls, logging, monitoring, and alerting within all pipelines.
• Orchestrate data workflows to meet operational SLAs.
• Engineer scalable data lake and data warehouse architectures.

Required Skills

• 7+ years of experience in Big Data Platform (data lake) and data warehouse engineering.
• 5+ years of strong Python and functional programming skills.
• 3+ years of hands-on experience building resilient data pipelines.
• Expertise with the Hadoop stack, including HDFS, Hive, SQL, Spark, Spark Streaming, Spark SQL, HBase, Kafka, Sqoop, and Airflow.
• 1+ years of experience with Google Cloud data services, specifically Cloud Storage, Cloud Proc, Cloud Flow, and BigQuery.
• Experience managing pipeline orchestration using Airflow, DAGs, and connector plugins.
• Ability to work with various source data connectors.

Preferred Skills

• Experience with Cloudera Manager, Atlas, Flink, Impala, Tez, and Hue.

Key Skills

Education

Any Graduate