Description

You will design, implement, and maintain data pipelines for data ingestion, processing, and transformation in Azure.

Responsibilities

• Utilize Azure Databricks to create and maintain ETL operations.
• Implement data validation and cleansing procedures to ensure data quality.
• Improve the scalability, efficiency, and cost-effectiveness of data pipelines.
• Monitor and resolve data pipeline problems to guarantee data consistency.
• Collaborate with data scientists and analysts to define data requirements and workflows.

Required Skills

• 8+ years of experience with Azure Databricks.
• Hands-on experience processing JSON or XML files as data sources.
• Proficiency in PySpark or Spark using data frames.
• Experience loading Delta live tables.
• Familiarity with autoloaders for incremental loading and data flattening options.
• Experience with ETL, BI, and DW technologies.
• Knowledge of data storage solutions including Azure SQL Database, Azure Data Lake, and Azure Blob Storage.
• Strong oral and written communication skills.

Key Skills

Education

Any Gradute