Description

You will design, build, and optimize scalable data pipelines and architectures on Azure or AWS using Databricks.

This role is hybrid.

Responsibilities

• Build and maintain robust data pipelines and datasets using Python, Scala, and PySpark.
• Design structured and unstructured data models, ensuring efficient access and storage techniques.
• Troubleshoot, profile, and optimize complex queries and code performance within the Databricks/Spark environment.
• Deploy and manage data applications on cloud platforms (Azure or AWS) using orchestration tools like Airflow.

Required Skills

• 7+ years of experience in data analytics and engineering.
• Expertise in Azure or AWS Databricks environments.
• Strong proficiency in Python, Scala, PySpark, and SQL.
• Experience with data storage formats (Parquet) and tools (Hive, HBase, Impala).
• Familiarity with data ingestion tools such as Sqoop, Kafka, and Flume.
• Knowledge of CI/CD and version control (Jenkins, Bamboo, GitHub, Bitbucket, Nexus).
• Bachelor’s degree in IT, Computer Science, Software Engineering, or equivalent.

Preferred Skills

• Experience with RDBMS design and big data problem-solving.

Key Skills

Education

Any Graduate