Description

You will build and manage data ingestion pipelines and processing frameworks within a Databricks environment.

Responsibilities

• Design and implement performant data ingestion pipelines from multiple sources using Databricks on AWS.
• Develop scalable, reusable frameworks to move data through Bronze, Silver, and Gold layers.
• Integrate end-to-end pipelines from source systems to target repositories while maintaining data quality and consistency.
• Utilize event-based and streaming technologies to ingest and process data.
• Deliver proof of concepts and production implementations within an Agile/DevOps methodology.

Required Skills

• 5+ years of experience in data engineering.
• Proficiency in writing Spark code using Scala, Python, or SQL.
• Hands-on experience with AWS Databricks.
• Experience with cloud-based architectures in AWS and Azure.
• Practical knowledge of Big Data components including Kafka, Spark SQL, Dataframes, and Hive DB.
• Experience working with large-scale data processing services like Hadoop.
• Strong understanding of databases and data warehousing.
• Ability to work within iterative Agile sprints.

Preferred Skills

• Experience with pro-code development workflows.

Key Skills

Education

Any Graduate