Description

You will design and manage data architectures leveraging Databricks and AWS services.

Responsibilities

• Run ETL jobs using Databricks Unity Catalog with Delta Lake, MongoDB, and S3-based Parquet files.
• Manage Databricks integration with AWS Glue Catalog.
• Monitor and manage Databricks job operations and performance.
• Implement best practices for setting up nested ETL jobs.
• Schedule Databricks jobs and pass runtime parameters to ETL processes.

Required Skills

• 12+ years of experience in data engineering projects within AWS environments.
• Extensive experience with Databricks Unity Catalog and Delta Lake format.
• Proficiency in handling MongoDB and S3-based Parquet files.
• Experience running and monitoring Databricks ETL jobs.
• Knowledge of AWS Glue Catalog integration.
• Ability to architect nested ETL job structures.
• Experience with job scheduling and runtime parameter passing in Databricks.
• Any Graduate degree.

Key Skills

Education

Any Graduate