Description

You will design, build, and maintain large-scale data processing solutions and AI/ML platforms using Java and Databricks on AWS or Azure.

Responsibilities

• Design, develop, and maintain scalable ETL/ELT processes and data pipelines using Java and Apache Spark on Databricks.
• Tune Spark jobs for performance, stability, and cost-efficiency; troubleshoot data skew and memory errors.
• Build automation using Java APIs and Infrastructure-as-Code tools like Terraform; manage orchestration and monitoring.
• Implement data governance policies, RBAC, encryption, and compliance standards using Delta Lake and Unity Catalog.
• Partner with data scientists and ML engineers to gather requirements, define compute needs, and support production environments.

Required Skills

• 10+ years of experience in software or data engineering, with strong Java expertise (Java 8+).
• Hands-on experience with Databricks, including workspace management, clusters, jobs, Delta Lake, MLflow, and Unity Catalog.
• Deep understanding of cloud infrastructure, specifically AWS or Azure, including IAM, Storage, and Networking.
• Proficiency in Spark fundamentals: DataFrames, SQL, and RDDs.
• Experience with CI/CD pipelines (GitHub Actions, Azure DevOps), Terraform, and monitoring tools (Grafana, Prometheus).
• Strong background in Spring Boot, Microservices, and Kafka.
• Familiarity with databases such as Oracle, MySQL, MongoDB, HBase, or DB2.

Preferred Skills

• Databricks Certified Professional Data Engineer certification.
• Experience with large-scale distributed systems and AI/ML pipelines.

Key Skills

Education

Any Graduate