Design, develop, and maintain data pipelines, models, and storage solutions within the Azure ecosystem.
Responsibilities
Design and maintain data pipelines using Azure Data Factory to ingest data from SQL Server, MongoDB, Blob Storage, Azure SQL Database, Cosmos DB, and Kafka.
Build transformation logic using Databricks and Spark, including tuning Databricks environments.
Implement data models for warehousing on Azure SQL DB and manage storage via Azure Blob and Data Lake.
Configure and manage Hadoop and Spark clusters in Azure HDInsight for big data processing.
Enforce data governance and security measures to protect sensitive information.
Required Skills
5+ years of experience managing data platforms and architecture using Azure services.
Proficiency in Python, SQL, and C#.
Hands-on experience with Azure Data Factory, Azure SQL Database, Azure Cosmos DB, and Azure HDInsight.
Strong background in data warehousing, data integration, and ETL tools.
Experience with NoSQL databases including MongoDB and Cassandra.
Working knowledge of SQL Server, Kafka, and Spark.
Proficiency with version control tools such as Git and Bitbucket.
Degree in Computer Science or a related field.
Preferred Skills
Familiarity with Aveva/OSISoft PI systems and SCADA systems.