Description

You will architect, implement, and manage cloud-based data solutions using Databricks and cloud infrastructure.

Responsibilities

• Build and maintain scalable data pipelines using Apache Spark and Databricks for efficient data flow.
• Design and implement cloud-native solutions for data storage and processing on Azure or AWS.
• Develop ETL processes to integrate data from SQL, NoSQL, and REST APIs into data lakes.
• Optimize Spark and Databricks workflows for performance and cost efficiency.
• Monitor pipelines, troubleshoot bottlenecks, and enforce coding best practices and documentation.

Required Skills

• 5+ years of experience in data engineering or architecture.
• Hands-on expertise with Databricks and Apache Spark for large-scale data processing.
• Strong experience with cloud platforms, specifically Azure or AWS.
• Proficiency in SQL, Python, Scala, or Java.
• Experience with data storage solutions including Azure Data Lake or AWS S3.
• Knowledge of building data pipelines and using orchestration tools like Airflow.
• Experience with DevOps and CI/CD processes for data workflows.
• Bachelor's degree or equivalent experience.

Preferred Skills

• Familiarity with containerization technologies such as Docker.
• Experience with machine learning pipelines and frameworks like MLflow, TensorFlow, or scikit-learn.

Key Skills

Education

Any Graduate