Description

Responsibilities

• Design, develop, and maintain end-to-end data pipelines and ETL/ELT workflows using PySpark and Python.

• Ensure and lead the efforts to review Legacy Data Stage legacy code and migrated Data bricks code to ensure functionality is not deviated

• Implement, optimize, and monitor large-scale data processing workloads in Azure Databricks, including cluster configuration, autoscaling, and governance.

• Build and maintain data integration and orchestration solutions using Azure services to meet performance, availability, and security requirements.

• Collaborate with data consumers, thread authors/owners, and stakeholders to gather business requirements, prioritize needs, and translate analytical objectives into technical designs.

• Implement secure data access patterns using Azure Active Directory, Managed Identities, and service principals.

• Author Infrastructure-as-Code for Azure resources (ARM templates) and deploy consistent, repeatable environments.

• Configure and operate Azure components including Storage Account, Synapse, Key Vault, VMSS, Function Apps, Web Apps, Log Analytics Workspace, Azure Container Apps / container instances, and related services.

• Collaborate with networking and security teams to design and implement Azure networking for data solutions.

• Implement monitoring, alerting, and cost optimization for data workloads (Log Analytics, metrics, and dashboards).

• Use GitLab and Azure DevOps for source control, CI/CD pipelines, and release management.

• Follow Agile/Scrum practices and participate in sprint planning, standups, and retrospectives.

• Ensure solutions meet data governance, lineage, and compliance requirements.

• Operations Support and Oncall Support for Production Issues and Deployments

Key Skills

Education

Bachelor's degree