Description

You will own the design, development, and optimization of data ingestion and wrangling pipelines within the MS Azure ecosystem.

Responsibilities

• Develop and maintain scalable data pipelines using PySpark DataFrames, Spark SQL, and Delta Lake formats.
• Optimize Spark jobs through partitioning, clustering, and performance tuning to ensure efficient data processing.
• Integrate Azure Databricks, Azure Data Factory (ADF), and Azure SQL Database to build robust data solutions.
• Collaborate with stakeholders across time zones to define requirements and deliver customer-centric data products.
• Produce clear technical and non-technical documentation while maintaining autonomy in problem-solving.

Required Skills

• 5+ years of hands-on experience with PySpark, specifically focusing on DataFrames, Partitioning, and SQL Optimization.
• Deep knowledge of Azure Cloud services including Databricks, ADF, Azure SQL DB, Storage Accounts, Key Vault, Application Gateways, and VNets.
• Proficiency in handling Delta file tables, Parquet, and CSV file formats.
• Experience with Power BI integration and data visualization.
• Strong ability to work with minimal direct guidance and navigate cross-functional obstacles.
• Excellent oral and written communication skills in English.
• Bachelor’s degree in a relevant field.

Key Skills

Education

Any Gradute