Description

What you’ll do

Build complex batch and streaming pipelines to ingest data from upstream Equifax cloud systems.

Design and implement data engineering frameworks to scale the development and deployment of data pipelines across the D&A organization.

Leverage AI-powered coding assistants to accelerate development, optimize code, and generate documentation for data pipelines and infrastructure.

Develop and refine prompts for Large Language Models (LLMs) to assist in data-related tasks such as data cleansing, transformation logic generation, and automated data documentation.

Design, build, and maintain scalable data pipelines that support AI/ML applications.

Explore and implement AI agents to automate repetitive data management tasks, monitor data quality, and orchestrate complex data workflows.

Play an active role in setting engineering standards and best practices in EWS D&A.

What experience do you need

At least 5 years of experience in data engineering, data architecture, or a related field.

A strong understanding of data engineering principles and best practices, including data modeling, data warehousing, and data integration.

At least 1 year of experience working in a GCP big data environment.

Experience building complex data pipelines and solutions using two or more of the following: BigQuery, DataFlow, DataProc, Pub/Sub, Cloud Functions.

Experience with Airflow or Cloud Composer.

Experience with Vertex AI.

Proficiency in Python development.

Professional experience with SQL.

Proven ability to effectively communicate complex technical concepts to both technical and non-technical stakeholders.

A Bachelor's degree or higher in Computer Science, Information Systems, or a related field.

What could set you apart

Demonstrated experience with prompt engineering to interact with and guide AI models for optimal outputs.

Hands-on experience with AI-assisted coding tools

Familiarity with frameworks for building AI applications or agents (e.g., LangChain, LlamaIndex).

Experience integrating AI services and models into data pipelines via APIs

DevOps/CICD experience (Jenkins and GitHub) and IaC experience using Terraform for managing GCP infrastructure.

GCP Associate Cloud Engineer or Professional Data Engineer certification.

Key Skills

Education

Any Graduate