← Back to jobs

E-Solutions Logo
AWS Data Engineer

E-Solutions

 

New York, NY, USA

Posted On: 11 days ago
Experience: 10+ years
Availability: Hybrid
Openings: 1
Category: AWS Data Engineer
Tenure: No Preference/Any
Related Jobs

No related jobs found

Description

Job Duties:

Must have below:

  • 10+ Years Experience
  • Great Communicator/Client Facing/ Attention to detail
  • Individual Contributor and ability to work as a team.
  • 100% Hands on in the mentioned skills

 

Programming Skills:

 

Python and Data Pipelines:

  • Advanced Proficiency in Python concepts like Code Structures, Modules, Packages, Class, SubClass, Inheritance, Multi-Threading and Functional Programming.
  • Experience in developing reusable Python packages for internal or public usage
  • Ability to write automating ETL processes and scheduling jobs like Airflow DAG.
  • Ability to track job pipeline runs to reprocess error records
  • Ability to orchestration different pipelines to run in sequence or parallel.
  • Troubleshoot data pipeline errors and fix issues
  • Export or Import data to/from various formats like CSV, JSON, XML etc preferably from S3 or other cloud storage.
  • Experience in using AI IDE tool

 

SQL:

  • Advanced SQL skills, including complex joins, CTE's and subqueries
  • Experience in optimizing SQL queries for performance and optimization in data warehouse technologies preferably Snowflake

 

Testing and documentation:

  • Proficiency in Python unit, integration, and system test.
  • Proficiency in implementing DBT tests for data validation and quality checks

 

Code Generation:

  • Experience in generating code using configurations using python and jinja templates
  • Version control:
  • Experience in GitHub, including implementing CI/CD process from scratch

 

AWS Expertise:

  • Data Storage solutions:
  • In depth understanding of AWS S3 for data storage, ECS, IAM including best practices for organization and security

 

Cloud Security:

  • Knowledge of AWS security best practices, including IAM roles, encryption standard, secure coding guidelines, DBT profiles access configurations and more.

 

Data Integration (nice to have):

  • Experience with AWS lambda for serverless data processing tasks

 

Workflow Orchestration (nice to have):

  • Proficiency in using Apache Airflow on AWS to design ,schedule and monitor complex data flows
  • Ability to integrate Airflow with AWS services and DBT models such as triggering a DBT model or EMR or reading from s3 writing to redshift or snowflake.
  • Experience in API integrations to Salesforce, SFTP, SharePoint, One Drive, and others.

 

DBT Core/ Cloud Proficiency (nice to have):

  • Experience in creating complex DBT models including full refresh, incremental models, snapshots, and documentation.
  • Ability to write and maintain DBT macros for reusable code.
  • Experience in creating custom DBT macros using jinja and Python allowing for reusable components within DBT models.

 

Monitoring and Logging (nice to have):

  • Familiarity with AWS cloud watch, Data Dog for monitoring the pipelines and setting up alerts for workflow failures

Education

Any Graduate

Related Jobs

No related jobs found

← Back to jobs