Description

• Design and implement automated data pipelines to integrate data from public portals, APIs, and internal systems into Delta tables.
• Collaborate with program manager on platform architecture, naming standards, and comprehensive documentation.
• Partner with data governance teams to define architecture, data sharing protocols, data contracts, ownership, and quality standards.
• Advise on data protection strategies including row-level and column-level access controls for sensitive data.
• Establish monitoring, alerts, and quality checks to ensure reliability and integrity of the data.
• Prepare datasets and documentation to make data easily accessible and explorable for business users and AI tools.
• Develop and maintain instruction (skill) files for consistent pipeline creation and support AI coding assistants.
• Assist other departments in building pipelines using standardized templates and best practices.
• Maintain clear communication and thorough documentation throughout the project.
  
   

Requirements

 

• 7+ years of experience in data engineering or closely related roles.
• Proven background in large-scale, enterprise-wide data architecture or engineering projects.
• Strong skills in Python and SQL.
• Experience with Apache Spark or leading cloud data platforms (Databricks strongly preferred).
• Skilled in extracting data from REST APIs, databases, or other systems.
• Excellent communication and documentation abilities

Key Skills

Education

Not specified