Description

You will build and maintain scalable data pipelines within a cloud-based environment.

Responsibilities

• Design and build scalable ETL pipelines using Databricks and Spark SQL.
• Optimize Spark jobs for performance and scalability within Databricks.
• Manage complex structured and semi-structured data, specifically JSON and XML.
• Integrate AWS services including S3, IAM, Secrets Manager, and Glue for workflow orchestration.
• Deploy and operate containerized workloads using AWS ECS and EKS.
• Use Domino Data Lab for model development and collaboration workflows.

Required Skills

• 4+ years of experience with Databricks, including platform internals and configuration.
• 5+ years of experience with Apache Spark and expert-level Spark SQL.
• 3+ years of experience using Domino Data Lab.
• Proficiency in Python for ETL and data pipeline development.
• Hands-on experience with complex JSON and XML structures.
• Solid working knowledge of AWS services: S3, IAM, Secrets Manager, Glue, and ECS/EKS.
• Strong troubleshooting and debugging skills for large-scale data systems.

Preferred Skills

• 10+ years of experience in Java development.
• Familiarity with CI/CD tools and Git version control.

Key Skills

Education

Any Graduate