Description

You will design and implement data engineering solutions focused on telemetry data and large-scale data processing.

Responsibilities

• Provide technical guidance and implement coding standards and best practices for the team.
• Debug and fine-tune notebook jobs to optimize memory consumption and processing speed.
• Conduct feasibility studies, evaluating technical alternatives, architectures, and risks.
• Manage process compliance and prepare status reports to mitigate project risks.
• Mentor team members to enhance technical capabilities and productivity.

Required Skills

• 10+ years of experience in data engineering roles.
• Hands-on experience with Kusto for telemetry data.
• Proficiency in PySpark and writing Python code within Azure Synapse Notebooks.
• End-to-end experience working with Azure Synapse.
• Experience with Spark structured streaming, specifically from Event Hubs.
• Proven ability to handle large data volumes in the hundreds of GBs.
• Strong knowledge of data modeling, data warehousing, and data management.
• Experience with SQL, Azure Data Lake, and Azure DevOps.

Preferred Skills

• Familiarity with Azure WebApps, Azure Functions, and Azure Cosmos DB.

Key Skills

Education

Not specified