Description

You will design and implement data architectures leveraging Databricks and cloud infrastructure.

Responsibilities

• Implement Databricks ingestion methods to move data from various source systems.
• Automate data workflows and processes using Python.
• Orchestrate data pipelines using AWS, Airflow, or Control-M.
• Optimize Spark jobs to support reporting applications.
• Design and maintain data structures following medallion architecture principles.

Required Skills

• 10+ years of experience in data architecture or engineering.
• Expertise in Databricks ingestion methods.
• Proficiency in Python for automation.
• Experience with orchestration tools such as Airflow or Control-M.
• Hands-on experience with AWS orchestration.
• Knowledge of medallion architecture.
• Experience with Spark optimization for reporting.
• Experience with file-based data copying from source systems.
• Familiarity with Qlik ETL.

Preferred Skills

• Experience managing large-scale data migrations.

Key Skills

Education

Any Graduate