Big Data Platform at interest презентация

Big Data Platform at interest Mao Ye

Data at Pinterest

Pinterest Data Architecture

Pinterest Data Architecture

Pinterest Data Architecture

Pinterest Data Architecture

Hadoop Platform Requirements Ephemeral clusters Access control layer Shared data store Easy deployment

Decoupling compute & storage

Centralized Hive Metastore

Multi-layered Packaging

Executor Abstraction Layer

Why Qubole? API for simplified executor abstraction Advanced support for spot instances Baked AMI customization

Pinball for Workflow Management

Scale of Processing Scale: 60 Billion Pins Hundreds of workflows Thousands of jobs 500+ jobs in a workflow 3 petabytes processed daily Support: Hadoop, Cascading, Hive, Spark …

Why Pinball? Requirements Simple abstractions Extensible in future Reliable stateless computing Easy to debug Scales horizontally Can be upgraded w/o aborting workflows Rich features like auto-retries, per-job emails, overrun policies… Options Apache Oozie, Azkaban, Luigi

Pinball Design

Workflow Model Workflow A directed graph of nodes called jobs Edge Run after dependence Node Job is a node

Job State Job state is captured in a token Tokens are named hierarchically

Job State Machine

Master Worker Interaction Master keeps the state Workers claim and execute tasks Horizontally scalable

Master Entire state is kept in memory Each state update is synchronously persisted before master replies to client Master runs on a single thread – no concurrency issues

Worker

Open Source Git repo: https://github.com/pinterest/pinball Mailing list: https://groups.google.com/forum/#!forum/pinball-users

Thank You