Description

Manage data engineering workflows in Azure Databricks and Hadoop environments. Own optimized architectures, cost monitoring, and pipeline optimization.

Responsibilities

• Design and propose optimized architectures for new and existing projects to reduce latency and costs.
• Monitor Azure and Databricks resource usage and generate cost reports for existing sources.
• Implement configuration changes and manage the migration of data sources to Databricks.
• Optimize batch and streaming data processing pipelines.
• Manage Azure service administration.

Required Skills

• 5+ years of experience in data engineering.
• Proficiency with PySpark and Python.
• Strong SQL skills.
• Experience with ETL processes and SSIS.
• Hands-on experience with Azure services and Azure Databricks.
• Knowledge of Cosmos DB.
• Experience with streaming and batch data processing in Hadoop and Cloud Databricks environments.

Preferred Skills

• Experience in cost optimization and resource configuration for cloud data environments.

Key Skills

Education

ANY GRADUATE