Description

Key Responsibilities

•        Design, build, and support end-to-end data pipelines (ingestion, transformation, validation, publishing).

•        Develop and optimize SQL, PySpark, Databricks transformations for large datasets.

•        Build production-grade Python/Pyspark components (reusable modules, logging, error handling, testing).

•        Create and maintain Azure Data Factory (ADF) pipelines (triggers, parameterization, monitoring, failure handling).

•        Work within Azure environments (ADLS Gen2, Azure SQL, resource groups, portal operations).

•        Provision and maintain Azure components using Pulumi (Infrastructure as Code).

•        Participate in code reviews, documentation, and operational support (triage + root cause analysis). Must-have Skills (Required)

•        ETL / ELT Concepts: Strong understanding of pipeline patterns, incremental loads, data validation, and troubleshooting.

•        SQL: Advanced querying (CTEs, views, joins, complex query logic) and performance tuning for transformations and validation.

•        Python: Production-quality development (modular code, testing, logging, integration with APIs/files, CICD, Unit Test/Integration test automation, Code Coverage).

•        PySpark: Distributed transformations and performance optimization (joins, partitions, debugging), CICD, Unit Test/Integration test automation, Code Coverage.

•        Azure Data Factory (ADF): Build/operate ADF pipelines, parameterization, triggers, monitoring, retry/error handling; integrate with Databricks/ADLS.

•        Databricks: Develop and operationalize notebooks/jobs/workflows; Delta Lake patterns; basic cluster/job configuration. New features - Genie Spaces, Delta Share, Unity Catalog, AI functions, Agent bricks and advanced optimization techniques.

•        Azure Fundamentals + Pulumi: Hands-on with ADLS Gen2, Azure Portal, Storage Explorer, Resource Groups, Azure SQL, Azure Batch and familiarity integrating with Azure OpenAI. Able to use/maintain Pulumi scripts for provisioning and managing Azure resources across environments.

•        Experience with Azure API Services and Azure Function Apps using Python. Nice-to-have Skills

•        R: Ability to support/translate validation rules with SQL scripts and create data quality reports.

•        TypeScript: Useful for pulumi pipeline to create Azure components.

•        Java: Useful for integration with existing services/components

. •        .NET: Useful for integration with existing services/components.

•        Angular / Spring Boot: Minor troubleshooting or coordination with app teams. Domain : Clinical (Health care) – Nice to have. Interview Process Round 1: Initial Phone Screening (30 minutes)

•        A 30-minute screening discussion conducted over the phone. Round 2: In-Person Interview (Chicago Office | 1 to 1.5 hours)

•        30–45 minutes: Complete a coding exercise based on a provided use case and demonstrate the solution/output.

•        15–30 minutes: Technical discussion focused on the coding solution, approach, design decisions, and problem-solving methodology

Key Skills

Education

Any Gradute