Description

Lead the design and optimization of data pipeline architectures and data flow collection for cross-functional teams. You will build scalable, stable, and available data solutions for multiple squads and products.

Responsibilities

• Expand and optimize data pipeline architectures and cloud-optimized data solutions from the ground up.
• Formulate and deliver optimal data architecture to support data scientists, analysts, and engineers.
• Build processes for data transformation, metadata management, dependency tracking, and workload management.
• Perform root cause analysis on internal and external data processes to identify improvement opportunities.

Required Skills

• Advanced SQL knowledge and experience with relational databases and query authoring.
• Experience building and optimizing big data pipelines, architectures, and datasets.
• Proficiency with AWS cloud services including EMR and EC2.
• Experience with stream-processing systems such as Storm, Spark-Streaming, or Kafka.
• Proficiency in object-oriented or functional scripting languages like Python, Java, C++, or Scala.
• Hands-on experience with big data tools including Hadoop, Spark, Kafka, and Kubernetes.
• Experience with relational and NoSQL databases, specifically Postgres and Cassandra.
• Familiarity with workflow management tools such as Azkaban, Luigi, or Airflow.
• Experience with data management tools like Collibra, Alation, BigID, or Alex Solutions.
• Experience working with Snowflake and Elastic-Search.

Preferred Skills

• Knowledge of agile methods.
• Experience working with unstructured datasets.

Key Skills

Education

Any Graduate