Description

You will develop and maintain DataHub-based frameworks and metadata management systems.

Responsibilities

• Build Spark-based frameworks and machine learning models for anomaly detection.
• Design ETL pipelines and custom frameworks using Apache Spark.
• Develop metadata ingestion processes from data lakes, databases, and ETL tools.
• Manage metadata, data cataloging, lineage, and governance workflows.
• Optimize AWS service usage and associated costs.

Required Skills

• 5+ years of professional experience.
• Proficiency in Java and Python.
• Experience with REST API development.
• Strong expertise in metadata management, cataloging, and lineage.
• Hands-on experience with Apache Spark for ETL design.
• Experience with DataHub and open-source development practices.
• Working knowledge of AWS services.
• Degree in any graduate field.

Preferred Skills

• Experience with Terraform, Helm, or other infrastructure-as-code tools.
• Knowledge of data governance regulations such as GDPR, CCPA, or HIPAA.
• Familiarity with Amundsen, Collibra, or Alation.
• Understanding of CI/CD pipelines and version control.

Key Skills

Education

Any Graduate