← Back to jobs

eTeam Logo
Lead Data Engineer

eTeam

 

Raleigh, NC, USA

Posted On: Just posted
Experience: 10+ years
Availability: Hybrid
Openings: 1
Category: Lead Data Engineer
Tenure: Contract - Corp-to-Corp
Related Jobs

No related jobs found

Description

Key Responsibilities
AI & Intelligent Automation Strategy
Define and execute the AI-enabled Data Engineering strategy to improve engineering velocity, quality, reliability, scalability and operational efficiency.
Identify, assess and prioritize high-value AI and automation use cases across Data Engineering, Data Warehouse and BI teams.
Develop reusable patterns and accelerators for introducing AI into existing Data Engineering and platform operations.
Build an AI-first engineering culture by adopting driving, enablement, governance and measurable engineering outcomes.

AI-Assisted Data Engineering
Lead adoption of GitHub Copilot, Snowflake Cortex and Databricks Genie to accelerate engineering activities and improve developer productivity.
Implement AI-assisted approaches for generating SQL, Python, PySpark, pipeline code, unit tests, documentation and deployment artifacts.
Automate generation of standardized, reusable data pipelines and engineering components.
Target automation of approximately 30–50% of repetitive pipeline development activities, where technically and operationally appropriate.
Build reusable engineering templates, prompts, coding standards and AI-assisted development patterns.

Intelligent Data Operations & Self-Healing Pipelines
Implement AI-driven monitoring and anomaly detection to proactively identify:
o Data quality issues
o Pipeline failures
o Schema changes and drift
o Processing anomalies
o Performance degradation
Design self-healing data pipelines capable of detecting and automatically resolving known operational issues.
Apply intelligent automation to incident triage, troubleshooting and root-cause analysis.
Reduce manual L1/L2 operational effort through automated diagnosis, remediation and preventive monitoring.
Establish appropriate human-in-the-loop controls for scenarios where automated remediation could create downstream risk.

Snowflake & Databricks Optimization
Leverage Snowflake AI and Cortex capabilities to improve engineering productivity and platform intelligence.
Identify expensive or inefficient queries and workloads and implement appropriate optimization strategies.
Improve warehouse utilization, workload management, query performance and overall platform cost efficiency.
Utilize Databricks Genie and Databricks AI capabilities to improve data discovery, analytics accessibility and engineering productivity.
Work with platform teams to establish measurable FinOps outcomes around compute consumption and cloud cost optimization.

AI-Enabled SDLC
Champion the use of AI across the complete engineering lifecycle, including:
Requirements analysis and refinement
Solution and pipeline design
Code generation and refactoring
Unit and integration testing
SQL and PySpark optimization
Code reviews
Technical documentation
CI/CD and deployment automation
Production monitoring
Incident analysis
Root-cause analysis
Technical debt identification and remediation
Ensure AI-generated outputs are appropriately reviewed, tested and integrated into existing engineering standards.

Responsible AI & Governance
Define standards and guardrails for responsible use of Generative AI within Data Engineering.
Establish controls covering:
o Data privacy
o Information security
o Regulatory compliance
o Intellectual property
o Code quality
o AI-generated code validation
o Accuracy and hallucination risk
o Maintainability and auditability
Ensure sensitive enterprise information is protected when engineers use AI-assisted development tools.
Work with architecture, security, risk and governance teams to establish approved AI engineering patterns.

Metrics & Value Realization
Define and maintain measurable KPIs for AI adoption and engineering effectiveness, such as:
AI tool adoption across engineering teams
Developer productivity improvements
Percentage of repetitive activities automated
Development cycle-time reduction
Pipeline failure rate
Mean Time to Detect (MTTD)
Mean Time to Resolve (MTTR)
Data-quality incident reduction
Technical-debt reduction
Cloud and platform cost savings
Reuse of standardized engineering components
The role will be expected to demonstrate measurable business value from AI adoption rather than focusing only on tool deployment.

Required Skills & Experience
10 years of experience in Data Engineering, Data Platforms, Data Warehousing or related technology roles.
Strong hands-on experience with enterprise-scale Data Engineering and Data Warehouse platforms.
Strong expertise in Snowflake and/or Databricks.
Advanced knowledge of SQL, Python and/or PySpark.
Experience designing and optimizing ETL/ELT pipelines and large-scale data-processing solutions.
Experience with AI-assisted engineering tools such as:
o GitHub Copilot
o Snowflake Cortex
o Databricks Genie
o Other enterprise Generative AI/code-assistance capabilities
Strong understanding of Snowflake performance optimization, workload management and cost optimization.
Experience with CI/CD, DevOps, automated testing and modern Data Engineering SDLC practices.
Experience implementing data observability, monitoring, anomaly detection or automated remediation.
Understanding of Generative AI, LLMs, prompt engineering and enterprise AI governance.
Ability to translate AI capabilities into practical engineering solutions and measurable business outcomes.
Strong stakeholder management and communication skills, with the ability to influence engineering, architecture and leadership teams.

Preferred Skills
Experience working with Azure/AWS enterprise data platforms.
Experience with Azure Data Factory, dbt, Airflow or similar orchestration technologies.
Knowledge of data observability and data-quality frameworks.
Experience building reusable Data Engineering accelerators or frameworks.
Understanding of MLOps/LLMOps, RAG and AI agent/agentic workflow concepts.
Experience implementing enterprise AI governance and secure Generative AI adoption.
Exposure to DataOps, DevOps, FinOps and Platform Engineering practices

Education

Bachelor's degree

Related Jobs

No related jobs found

← Back to jobs