Description

You will design and build scalable data pipelines for near real-time and batch data ingestion, processing, and integration.

Responsibilities

• Design and build scalable data pipelines for near real-time and batch data ingestion and processing.
• Provide technical guidance to team members on the development and operations of streaming and event-driven applications.
• Identify potential project risks and implement mitigation strategies.
• Collaborate with business leadership to translate data requirements into technical solutions.
• Own the processes for manipulating and merging large datasets.

Required Skills

• 5+ years of experience in data engineering or related roles.
• Expert level proficiency in Python, Databricks, and Azure Data Factory.
• Strong programming experience in C# / .NET and Logic Apps.
• Hands-on experience with Azure Functions using various protocols and triggers.
• Proficiency in configuring Azure Event Hub, Event Grid, Stream Analytics, and JSON.
• Experience with message-oriented middleware and streaming technologies including Kafka, MQ, Azure Event Hub, Spark, and Spark Streaming.
• Practical knowledge of ETL tools, techniques, and CI/CD pipelines.
• Experience with cloud NoSQL databases, specifically Cosmos DB or equivalent.
• Degree in Information Systems, Computer Science, Engineering, or equivalent experience.

Key Skills

Education

Any Gradaute