Description

You will architect, implement, and manage cloud-based data solutions using Databricks and associated cloud environments.

Responsibilities

• Build and maintain scalable data pipelines using Apache Spark and Databricks to ensure efficient data flow.
• Design cloud-native solutions for data storage and processing on Azure or AWS.
• Implement ETL processes to integrate data from SQL, NoSQL, and REST APIs into data lakes.
• Optimize Spark and Databricks workflows for performance and cost efficiency.
• Act as a technical bridge between offshore teams and business stakeholders to translate requirements into data solutions.

Required Skills

• 6+ years of experience in data engineering.
• Hands-on experience with Databricks and Apache Spark for large-scale data processing.
• Proficiency in SQL, Python, Scala, or Java.
• Experience with cloud platforms, specifically Azure or AWS.
• Working knowledge of Delta Lake, Data Lakes, and Azure Data Lake Storage.
• Experience building data pipelines using Airflow.
• K knowledge of DevOps and CI/CD processes for data workflows.
• Degree in any graduate field.

Preferred Skills

• Familiarity with containerization using Docker or Kubernetes.
• Experience with machine learning pipelines and frameworks such as MLflow, TensorFlow, or scikit-learn.

Key Skills

Education

Any Graduate