Description

You will build and maintain scalable data pipelines to process large-scale datasets across cloud environments.

Responsibilities

• Design and develop scalable data pipelines using PySpark to process large volumes of data.
• Build and manage ETL processes to extract, transform, and load data into warehouses and data lakes.
• Optimize and tune data processing jobs for performance and cost-efficiency.
• Implement data validation and monitoring processes to ensure accuracy and consistency.
• Troubleshoot and resolve data-related issues and pipeline failures.

Required Skills

• 3+ years of experience in data engineering or a related field.
• Proficiency in Python and PySpark.
• Strong knowledge of Big Data technologies including Spark, Hadoop, and Hive.
• Experience with cloud platforms including AWS, Azure, and GCP.
• Hands-on experience with SQL and relational databases such as MySQL.
• Experience with NoSQL databases like MongoDB or Cassandra.
• Familiarity with data warehousing solutions such as Amazon Redshift, Google BigQuery, or Snowflake.
• Experience with ETL/ELT processes and orchestration tools like Apache Airflow or Apache NiFi.

Preferred Skills

• Bachelor's degree in Computer Science, Information Technology, or Engineering.

Key Skills

Education

Bachelor's degree in Computer Science