Description

Required:

• 7–10 years of experience in software development, data engineering, or a related technical field.
• 7+ years of hands-on experience with Python, SQL, ETL, and data processing.
• Strong experience with Databricks and Apache Spark.
• Approximately 5 years of experience developing Spark and Databricks solutions.
• Strong knowledge of Azure Cloud services and Azure ML.
• Experience designing and developing scalable data pipelines and application components.
• Advanced proficiency in Python, Scala, and/or SQL.
• Strong understanding of software development lifecycle practices.
• Experience designing and implementing CI/CD pipelines using GitHub Actions, Azure DevOps, Jenkins, or similar tools.
• Experience deploying code across multiple environments, including development, QA, and production.
• Approximately 5 years of experience with logging, monitoring, alerting, and production troubleshooting.
• Experience with data governance, access control, security, compliance, and data lineage.

Preferred:

• Experience with Java and Spring.
• Knowledge of additional Azure services.
• Experience with Unity Catalog.
• Experience working with Azure DevOps and GitHub Actions.
• Experience using AI tools to improve software development productivity.
• Experience supporting enterprise-scale data or application platforms.

Responsibilities:

• Participate in the end-to-end design and implementation of scalable data and application solutions using Python, Azure, Databricks, Spark, and Azure ML.
• Develop complex ETL/ELT pipelines to process and transform large-scale datasets from multiple sources.
• Use Python, Scala, SQL, Spark, and Databricks for data processing and application development.
• Optimize Python and Spark jobs for performance and cost efficiency through partitioning, caching, cluster tuning, and bottleneck analysis.
• Collaborate with data scientists, analysts, product teams, and business stakeholders to convert business requirements into technical solutions.
• Implement data security, access controls, encryption, compliance standards, and governance frameworks such as Unity Catalog.
• Establish data lineage and ensure data assets meet organizational security and governance requirements.
• Design and maintain CI/CD pipelines using Azure DevOps, GitHub Actions, Jenkins, or similar tools.
• Automate the deployment of code, data pipelines, infrastructure, and other data assets across development, test, and production environments.
• Develop monitoring, logging, and alerting solutions for production-grade data pipelines and applications.
• Troubleshoot performance, reliability, deployment, and production issues.
• Follow software development lifecycle best practices, including code reviews, testing, documentation, release management, and production support.
• Support Agile planning and help coordinate project deliverables across multiple project pods

Key Skills

Education

Bachelor's degree