Description

Design, build, and maintain scalable data pipelines to process large volumes of data.

Responsibilities

• Develop and maintain ETL/ELT processes to load data into warehouses and data lakes.
• Design and optimize PySpark jobs for performance and cost-efficiency.
• Integrate data from multiple sources while implementing validation and monitoring for data quality.
• Troubleshoot and resolve data-related issues and pipeline failures.

Required Skills

• 3+ years of experience in data engineering or a related field.
• Proficiency in Python and PySpark.
• Strong knowledge of big data technologies including Hadoop, Hive, and Spark.
• Experience with cloud platforms such as AWS, Azure, or GCP.
• Familiarity with data warehousing solutions like Amazon Redshift, Google BigQuery, or Snowflake.
• Experience with relational and NoSQL databases, including MySQL, MongoDB, and Cassandra.
• Hands-on experience with orchestration tools like Apache Airflow or Apache NiFi.

Preferred Skills

• Bachelor's degree in Computer Science, Information Technology, or Engineering.

Key Skills

Education

Bachelor's degree in Computer Science