Description

You will design, develop, and maintain scalable data pipelines to move and transform large datasets across cloud and on-premise environments.

Responsibilities

• Design, develop, and maintain ETL pipelines to load data from multiple sources into data warehouses or data lakes.
• Build, orchestrate, and automate complex workflows using Azure Data Factory.
• Develop and optimize complex SQL queries, stored procedures, and triggers to ensure data consistency and performance.
• Integrate data from various sources, including external APIs and third-party applications, into centralized storage.
• Monitor, troubleshoot, and optimize data pipelines with implemented error handling and logging.

Required Skills

• 6+ years of professional experience in data engineering and ETL development.
• 3+ years of hands-on experience with Azure Data Factory.
• Advanced proficiency in T-SQL, including complex queries, stored procedures, triggers, and functions.
• Experience with SQL Server tools such as SSMS, Profiler, and SSIS.
• Strong knowledge of data warehousing, data lake concepts, and SSRS.
• Experience with Azure cloud services including Azure SQL Database, Azure Data Lake, and Azure Blob Storage.
• Proficiency in Python, Java, or similar programming languages.
• Experience with SQL, NoSQL, and NewSQL databases.
• Familiarity with version control using Git and Agile methodologies.

Preferred Skills

• Experience with Azure Databricks and Azure Synapse Analytics.
• Knowledge of Spark and Python for data processing.
• Familiarity with data governance and security best practices.

Key Skills

Education

Bachelor's degree in Computer Science