Description

You will own the end-to-end data lifecycle, from requirement gathering and technical design to ETL implementation and performance testing.

Responsibilities

• Analyze large transactional and behavioral datasets using statistical modeling and pattern analysis to identify customer insights.
• Design technical specifications, high-level and low-level design documents, and data mapping transformations.
• Build and maintain ETL flows and data quality frameworks within Hadoop environments.
• Optimize query performance and Spark jobs through efficient partitioning and bucketing schemas.
• Execute comprehensive testing cycles, including load, stress, volume, and database failover testing.

Required Skills

• 5+ years of experience in data analysis and ETL development.
• Hands-on experience with the Hadoop ecosystem, including Spark, MapReduce (MRv1/YARN), Sqoop, Hive, Oozie, Flume, Kafka, and Zookeeper.
• Proficiency in writing HiveQL queries and managing data in MySQL and HDFS.
• Extensive experience with IBM DataStage for ETL processes and designing transformations.
• Expertise in creating SSIS packages, including Execute SQL and Data Flow tasks.
• Ability to work with various file formats such as Avro, Parquet, Sequence, and ORC using Snappy and Gzip compression.
• Experience in designing and implementing partitioning/bucketing schemas in Hive for fast data retrieval.
• Competency in preparing test plans, scripts, and scenarios based on functional requirements.
• Knowledge of monitoring system performance metrics like CPU/memory utilization, throughput, and response times.

Key Skills

Education

Any Graduate