Description

You will build and manage scalable data infrastructure and ETL pipelines to support large-scale data analysis.

Responsibilities

• Design, develop, and implement scalable data pipelines and ETL processes using Java, Python, and Spark.
• Manage and optimize Spark clusters to ensure performance and reliability.
• Develop and maintain data models, schemas, and validation checks to ensure data consistency.
• Build and integrate REST APIs using Flask or Spring to facilitate data access.
• Monitor pipeline performance, troubleshoot bottlenecks, and mentor junior engineers.

Required Skills

• 10+ years of professional experience in data engineering.
• Strong programming expertise in Java and Python.
• Extensive experience with Spark, including cluster management and performance tuning.
• Proficiency in SQL and relational databases such as Snowflake or Delta Tables.
• Experience building REST APIs with Flask or Spring frameworks.
• Hands-on experience with ETL processes and big data technologies.
• Familiarity with cloud-based data platforms like Azure.
• Solid understanding of data science concepts and data modeling.

Preferred Skills

• Experience with data warehousing tools like BigQuery or Snowflake.

Key Skills

Education

Any Graduate