Description

You will design, build, and maintain scalable data pipelines and processing workflows using Databricks and Apache Spark.

Responsibilities

• Build and maintain data pipelines using Databricks and Apache Spark.
• Implement data processing workflows for efficient transformation and aggregation.
• Write and optimize complex SQL queries for data extraction and analysis.
• Automate pipeline jobs and implement monitoring and logging for continuous operation.
• Leverage cloud platforms including AWS, Azure, or GCP for data solutions.

Required Skills

• Databricks Certification (required).
• 5+ years of experience in data engineering.
• Expert-level experience with Databricks and Apache Spark.
• Advanced SQL skills for manipulating large datasets from PostgreSQL, MySQL, and SQL Server.
• Proficiency in Python for scripting and automation.
• Experience with cloud platforms such as AWS, Azure, or GCP.
• Hands-on experience with big data technologies including Hadoop and Flink.
• Knowledge of data warehousing concepts.

Preferred Skills

• Any Graduate degree.

Key Skills

Education

Any Graduate