PySpark Developer

You will develop Data Stores and Data Warehouse solutions within Azure.

Responsibilities

• Design and develop ADF data pipelines to move data from various sources into Azure Databases and Azure Data Lake using self-integration runtime.
• Collaborate with Product Owners and Solution Architects to capture requirements and deliver solutions on the Azure platform.
• Estimate, plan, and execute Proof of Concept (PoC) projects using Agile methodology.
• Design and deliver quality solutions independently or lead a team of Data Engineers.

Required Skills

• Minimum 5 years of PySpark Development experience, focusing on Spark SQL and complex transformations.
• Minimum 3 years of Python development experience.
• Minimum 2 years of experience with Azure Databricks, Azure Databases, PySpark, Databricks, Azure ADLS, and Azure Cloud.
• Hands-on experience with Delta Lake, SQL, and complex data handling scenarios.
• Proficiency in PySpark transformations, aggregations, window functions, and schema handling.
• Strong capability in Big Data handling, performance tuning, and complex SQL problem-solving.
• Demonstrated ability to handle tricky programming scenarios and share solutioning skills.