Description

Job Duties:

The Data Engineer will perform the following duties and responsibilities, including but not limited to:

• Design, develop, and maintain scalable data processing architectures for large volumes of structured and unstructured data utilizing Amazon Web Services (AWS).

• Develop and manage distributed data pipelines using Apache Spark and Databricks to support batch and near real-time data processing operations.

• Build and maintain ETL/ELT workflows utilizing Delta Lake technologies to support ACID-compliant lakehouse data architecture.


• Design, implement, and optimize logical and physical data models, schemas, and storage structures within Snowflake environments for analytics and reporting purposes.

• Perform data ingestion and integration from various sources, including relational databases, APIs, flat files, and streaming platforms.

• Implement data validation procedures, monitoring frameworks, and quality assurance controls to ensure accuracy, integrity, and reliability of enterprise data systems.


• Analyze and optimize performance of large-scale distributed processing jobs, SQL queries, and cloud-based data workloads.


• Apply data governance, security, and compliance standards, including role-based access controls, encryption protocols, and data protection measures.

• Automate infrastructure deployment and operational workflows using CI/CD pipelines and Infrastructure as Code (IaC) methodologies.

• Troubleshoot, diagnose, and resolve issues affecting production data pipelines, workflows, and cloud-based processing systems.

• Collaborate with cross-functional teams, including data analysts, software engineers, business stakeholders, and data scientists, to support enterprise data initiatives and business intelligence objectives.

• Perform additional duties and responsibilities as assigned by management.

Minimum Requirements:

• Bachelor’s degree in Computer Science, Information Technology, Engineering, Data Analytics, or a closely related field.


• Related experience in data engineering, data warehousing, ETL/ELT development, or cloud-based data processing environments.

• Experience working with Amazon Web Services (AWS), Databricks, Apache Spark, Delta Lake, and Snowflake technologies.

• Strong knowledge of SQL, database design, data modeling, and query optimization techniques.

• Experience processing and managing structured and unstructured datasets.

• Experience with data ingestion and integration from relational databases, APIs, and streaming platforms.

• Knowledge of data validation methodologies, monitoring frameworks, troubleshooting, and production support for enterprise data pipelines.

• Understanding of data governance principles, security protocols, encryption standards, and role-based access controls.

• Experience with CI/CD pipelines and Infrastructure as Code tools is preferred.

• Strong analytical, problem-solving, organizational, communication, and teamwork skills

Key Skills

Education

Any Graduate