Description

You will design, develop, and optimize enterprise metadata management and data lineage solutions using DataHub.

Responsibilities

• Design and develop metadata management systems and data lineage solutions.
• Develop ETL pipelines and custom frameworks using Apache Spark.
• Manage metadata ingestion from data lakes, databases, and ETL tools.
• Implement machine learning techniques for anomaly detection.
• Optimize data cataloging processes and AWS costs.

Required Skills

• 5+ years of experience in metadata management, data lineage, or data governance.
• Proven track record as a committer or active contributor to the DataHub open-source project.
• Proficiency in Java and Python.
• Experience with REST API development.
• Strong experience with Apache Spark for ETL pipeline design.
• Hands-on experience with AWS services and cost optimization.
• Experience with metadata ingestion from various data systems.
• Any Graduate degree.

Preferred Skills

• Knowledge of data governance regulations such as GDPR, CCPA, or HIPAA.
• Experience with infrastructure-as-code tools like Terraform or Helm.

Key Skills

Education

Any Graduate