Description

You will design and build scalable data pipelines and manage distributed data systems in a part-time, remote capacity.

This role is remote.

Responsibilities

• Design and build scalable data pipelines using Apache Spark, Kafka, and Airflow with cloud services.
• Develop and optimize data lakes, warehouses, and real-time streaming solutions using Snowflake, Databricks, BigQuery, or Redshift.
• Implement ETL/ELT processes to ingest and transform structured and unstructured data from multiple sources.
• Optimize SQL queries, distributed computing frameworks, and data storage for performance and cost efficiency.

Required Skills

• 6+ years of experience in Data Engineering, Big Data, and Cloud Technologies.
• Expertise in Python, SQL, Scala, or Java for data processing.
• Strong knowledge of Apache Spark, Hadoop (HDFS, YARN), Kafka, and Airflow.
• Hands-on experience with AWS (Glue, Redshift, EMR), Azure (Synapse, Data Factory), or GCP (BigQuery, Dataflow).
• Experience with Snowflake, Databricks, or other data warehousing tools.
• Proficiency in database management with PostgreSQL, MySQL, MongoDB, or Cassandra.
• Experience with containerization and orchestration using Docker and Kubernetes.
• Knowledge of DevOps practices including CI/CD, GitHub Actions, or Jenkins.

Preferred Skills

• Experience with MLOps, including MLflow and Feature Engineering.

Key Skills

Education

Any Graduate