← Back to jobs

Experis Logo
AI Big Data Engineer

Experis

 

Rockville, MD, USA

Posted On: Just posted
Experience: 5+ years
Availability: Hybrid
Openings: 1
Category: Big Data Engineer
Tenure: No Preference/Any
Related Jobs

No related jobs found

Description

You will design, build, and maintain large-scale data processing pipelines using distributed systems and modern big data technologies.

This role is hybrid.

Responsibilities

  • Develop scalable data ingestion, storage, transformation, and analysis solutions using Hadoop, Spark, and Python/Scala.
  • Optimize data pipelines for performance, handling petabyte-scale volumes and mitigating issues like data skew and resource limitations.
  • Implement automated testing frameworks and continuous testing to ensure data quality, writing unit, integration, and end-to-end tests.
  • Troubleshoot production pipeline failures and collaborate with data scientists to support data-driven decision-making.
  • Leverage AI development tools (GitHub Copilot, ChatGPT) and apply prompt engineering to enhance development workflows.

Required Skills

  • 5+ years of experience with distributed systems and big data technologies including Hadoop, Spark, Hive, and Trino.
  • Strong proficiency in object-oriented programming languages such as Python, Scala, or Java.
  • Deep understanding of Apache Spark internals: executors, tasks, stages, DAG, and performance tuning (partitioning, caching, broadcast joins).
  • Advanced SQL skills, including window functions, complex multi-table joins, and query optimization.
  • Experience with cloud platforms, specifically AWS services like S3, EMR, Glue, Lambda, and Athena.
  • Hands-on experience with AI coding assistants (GitHub Copilot, Q Developer, ChatGPT) and prompt engineering.
  • Proven experience in Agile environments (Scrum, Kanban) with a focus on test automation and configuration management.

Preferred Skills

  • Master's degree in Computer Science or related field, or prior experience in the Financial Services industry.
  • Experience leading teams through AI adoption and workflow transformation.

Education

Any Graduate

Related Jobs

No related jobs found

← Back to jobs