Description

You will build and maintain data pipelines using Apache Spark, PySpark, and AWS EMR, ensuring efficient data storage and retrieval from S3 and Elasticsearch.

You will own the ETL orchestration using Airflow and Autosys, while developing APIs and integrating services with a solid grasp of API functional requirements.

You will write robust Python and Scala code, leveraging Databricks for advanced data processing, and utilize SQL (Postgres-SQL) for complex querying and manipulation.

You will manage version control with Git and SVN, implement shell scripting on UNIX systems, and handle basic HTML tasks for web development needs.

Responsibilities

• Develop and optimize data pipelines using Apache Spark, PySpark, and AWS EMR.
• Orchestrate ETL workflows with Airflow and Autosys, integrating data from S3 and Elasticsearch.
• Write production-grade Python and Scala code, utilizing Databricks for large-scale data processing.
• Design and maintain APIs, ensuring seamless data integration and retrieval.
• Manage version control with Git/SVN and implement UNIX shell scripting for automation.

Required Skills

• 7+ years of experience in data engineering.
• Proficiency in Apache Spark, PySpark, and AWS EMR.
• Strong Python and Scala programming skills.
• Hands-on experience with Databricks.
• Advanced SQL skills (Postgres-SQL) for data manipulation.
• Experience with Apache Hadoop, Hive, and Unix/Shell scripting.
• Familiarity with ETL tools: Airflow and Autosys.
• Knowledge of Git, SVN, and basic HTML.

Preferred Skills

• Experience with Elasticsearch for data storage and retrieval.
• Functional understanding of API development and integration.

Key Skills

Education

Bachelor's or Master's degrees