Description

You will build and maintain scalable data pipelines using Databricks and Spark to support large-scale data processing and business analytics.

Responsibilities

• Design and implement ETL processes using Databricks and Spark.
• Develop and optimize data pipelines for large-scale processing and efficient storage retrieval.
• Build and maintain data warehouse solutions and perform data modeling.
• Troubleshoot and resolve performance issues within data infrastructure and pipelines.
• Automate data quality checks and implement security and data governance best practices.

Required Skills

• 5 years of experience in data engineering.
• Proven experience developing data pipelines with Databricks and Spark.
• Advanced Python programming skills for data processing and manipulation.
• Strong SQL skills for writing complex queries and data analysis.
• Deep understanding of big data technologies and distributed computing concepts.
• Experience with data modeling and optimizing data storage for performance.
• Knowledge of data governance, security, and compliance practices.
• Bachelor's degree in Computer Science, Engineering, or a related field.

Preferred Skills

• Experience with cloud platforms such as AWS, Azure, or GCP.
• Databricks and Spark-related certifications.
• Experience in Agile development methodologies and version control systems.

Key Skills

Education

Not specified