Description

Design and build scalable data pipelines for batch and near real-time ingestion, processing, and integration.

Responsibilities

• Design and build scalable data pipelines for batch and near real-time ingestion and processing.
• Coordinate the creation and documentation of ETL pipelines and testing for BI and Data Science needs.
• Own the process of manipulating and merging large datasets.
• Act as the technical point of contact between API teams and functional leads.
• Propose and develop data solutions that enable effective business decision-making.

Required Skills

• Expert-level Python, Databricks, and Azure Data Factory.
• Hands-on experience with Spark and PySpark.
• Experience configuring Azure Event Hub, Event Grid, Stream Analytics, and Logic/Function apps.
• Strong programming skills in C# / .NET and Logic Apps.
• Experience with Azure Functions using various protocols and triggers.
• Proficiency with JSON and Git/GitHub.
• Knowledge of CI/CD and ETL tools/techniques.
• Experience with message-oriented middleware or streaming technologies like Kafka, NiFi, or MQ.
• Experience with cloud NoSQL databases, specifically Azure Cosmos DB or equivalent.
• Knowledge of cloud infrastructure including load-balancing, auto-scaling, monitoring, and networking.

Preferred Skills

• Experience scaling data streaming technologies like Kafka or Spark Streaming.

Key Skills

Education

ANY GRADUATE