Description

Design and implement large-scale data engineering solutions within Databricks on AWS.

Responsibilities

• Build and maintain data pipelines using Spark and PySpark in Databricks.
• Design data engineering solutions on AWS cloud from MVP to production-grade deployment.
• Manage data governance, compliance, and cataloging using Databricks Unity Catalog.
• Develop data models for 360 customer views and customer behavior analysis.
• Collaborate with stakeholders to define system specifications and strategic IT solutions.

Required Skills

• 13+ years of professional experience.
• Expertise in Spark and PySpark for pipeline development.
• Hands-on experience with Amazon S3, IAM, EC2, EMR, Kinesis, Lambda, Redshift, CloudWatch, SNS, SQS, Glue, and Athena.
• Deep understanding of Delta Lake, Medallion architecture, and Delta tables.
• Experience handling structured and unstructured data in formats like ORC and Parquet.
• Knowledge of enterprise systems, networking modules, and software integration.
• Proficiency in data validation, data modeling, and data visualization.
• Ability to manage projects independently from development to production.

Key Skills

Education

Any Graduate