ENGINEERING BIG DATA WITH HADOOP презентация

ENGINEERING BIG DATA WITH HADOOP BY International School of Engineering {We Are Applied Engineering}

OVERVIEW WHAT IS BIG DATA? EXPLOSION OF DATA DATA CONTRIBUTIONS DATA EXPLOSION WHO ARE THE PLAYERS? BIG DATA–BIG PICTURE– LANDSCAPE BIG DATA– ENTERPRISE ROLES

WHAT IS BIG DATA? High-volume, high-velocity and high- variety information assets that demand cost- effective, innovative forms of information processing for enhanced insight and decision making. -Gartner

EXPLOSION OF DATA

DATA CONTRIBUTIONS

DATA EXPLOSION

WHO ARE THE PLAYERS?

BIG DATA–BIG PICTURE– LANDSCAPE

BIG DATA– ENTERPRISE ROLES

INTRODUCTION TO

WHAT IS HADOOP? Flexible Structured/Unstructured Text/Binary Schema/Schema less 100% Open Source Scalable – Petabytes of Data – Thousands of Nodes

EVOLUTION OF HADOOP

HADOOP ECOSYSTEM

HADOOP ECOSYSTEM MAP

Hadoop Evolution – Map Explained! How did it all start- huge data on the web! Nutch built to crawl this web data Huge data had to be saved- HDFS was born! How to use this data? Map reduce framework built for coding and running analytics – java, any language-streaming (Hadoop streaming) How to get in unstructured data – Web logs, Click streams, Apache logs, Server logs  – fuse,webdav, chukwa, flume, Scribe Hiho and sqoop for loading data into HDFS – RDBMS can join the Hadoop band wagon!

Continued High level interfaces required over low level map reduce programming– Pig, Hive, Jaql BI tools with advanced UI reporting- drilldown etc- Intellicus Workflow tools over Map-Reduce processes and High level languages: Oozie Monitor and manage hadoop, run jobs/hive, view HDFS – high level view- Hue, karmasphere, eclipse plugin, cacti, ganglia Support frameworks- Avro (Serialization), Zookeeper (Coordination) More High level interfaces/uses- Mahout, Elastic map Reduce OLTP- also possible – Hbase

BIG DATA & ANALYTICS Case Studies

YAHOO - PERSONALIZATION

YAHOO SEARCH ASSIST

For Detailed Description of HADOOP ECOSYSTEM components