Description

You will lead the design, development, and evolution of AI and Data Products within a highly technical environment.

Responsibilities

• Lead technical design meetings, conduct tech breakdowns, and serve as the primary technical contact for Product Owners.
• Design and implement batch and real-time data ingestion pipelines using Databricks and Azure Data Factory.
• Transform, cleanse, and enrich raw data using DBT and PySpark to ensure analytics readiness.
• Optimize data storage through partitioning, indexing, and use of Parquet or Delta Lake formats.
• Mentor junior engineers, manage technical debt, and collaborate with data scientists to prepare data for ML model deployment.

Required Skills

• 5+ years of experience in a Data Engineering role within large corporate organizations.
• Strong proficiency in Python, including Pandas and dependency management tools (poetry or venv).
• Expertise in building and managing data pipelines within the Azure cloud ecosystem.
• Hands-on experience with Databricks and PySpark for big data processing.
• Experience using DBT for data transformation.
• Proficiency in SQL for query optimization and assisting data analysts.
• Practical knowledge of Azure Data Factory, Azure Data Lake Storage, and Blob Storage.
• Ability to implement role-based access controls and data encryption standards.
• Engineering Master’s degree or PhD.

Preferred Skills

• Experience working within a dedicated Data/AI environment.

Key Skills

Education

Engineering Master’s degree or PhD