Taming Big Data! презентация

Ian Foster Argonne National Laboratory and University of Chicago foster@anl.gov ianfoster.org

Discovery is an iterative process

Discovery in the big data era: Resource-intensive, expensive, slow

Three big data challenges Channel massive flows Automate management Build discovery engines

Three big data challenges Channel massive flows Automate management Build discovery engines

Channel massive data flows Data must move to be useful. We may optimize, but we can never entirely eliminate distance. Sources: experimental facilities, sensors, computations Sinks: analysis computers, display systems Stores: impedance matchers & time shifters Pipes: IO systems and networks connect other elements

Transfer is challenging at many levels Speed and reliability GridFTP protocol Globus implementation Scheduling and modeling SEAL and STEAL algorithms RAMSES project

GridFTP protocol and implementations: Fast, reliable, secure 3rd-party data transfer

85 Gbps sustained disk-to-disk over 100 Gbps network, Ottawa—New Orleans

Transfer scheduling and optimization Science data traffic is extremely bursty User experience can be improved by scheduling to minimize slowdown Traffic can be categorized: interactive or batch Increased concurrency tends to increase aggregate throughput, to a point

A load-aware, adaptive algorithm: (1) Data-driven model of throughput

A load-aware, adaptive algorithm: (2) Concurrency-constrained scheduling Define transfer priority: Schedule transfers if neither source nor destination is saturated, using model to decide concurrency If source or destination is saturated, interrupt active transfer(s) to service waiting requests, if in so doing can reduce overall average slowdown Should a new transfer be scheduled? When scheduling a transfer, with what concurrency? When should active transfer be preempted? When change concurrency of active transfer?

Robust analytic models for science at extreme scales Gagan Agarwal1* Prasanna Balaprakash2 Ian Foster2* Raj Kettimuthu2 Sven Leyffer2 Vitali Morozov2 Todd Munson2 Nagi Rao3* Saday Sadayappan1 Brad Settlemyer3 Brian Tierney4* Don Towsley5* Venkat Vishwanath2 Yao Zhang2 1 Ohio State University 2 Argonne National Laboratory 3 Oak Ridge National Laboratory 4 ESnet 5 UMass Amherst (* Co-PIs)

How to create more accurate, useful, and portable models of distributed systems? Simple analytical model: T= α+ β*l [startup cost + sustained bandwidth] Experiment + regression to estimate α, β

Differential regression for combining data from different sources Example of use: Predict performance on connection length L not realizable on physical infrastructure E.g., IB-RDMA or HTCP throughput on 900-mile connection Make multiple measurements of performance on path lengths d: Ms(d): OPNET simulation ME(d): ANUE-emulated path MU(di): Real network (USN) Compute measurement regressions on d: ṀA(.), A∈{S, E, U} Compute differential regressions: ∆ṀA,B(.) = ṀA(.) - ṀB(.), A, B∈{S, E, U} Apply differential regression to obtain estimates, C∈{S, E} ��U(d) = MC(d) - ∆ṀC,U(d)

End-to-end profile composition

Three big data challenges Channel massive flows Automate management Build discovery engines

One researcher’s perspective on data management challenges

Tripit exemplifies process automation Me Book flights Book hotel

How the “business cloud” works

Process automation for science

Globus research data management services

Reliable, secure, high-performance file transfer and synchronization “Fire-and-forget” transfers Automatic fault recovery Seamless security integration Powerful GUI and APIs

Simple, secure sharing off existing storage systems

Extreme ease of use InCommon, Oauth, OpenID, X.509, … Credential management Group definition and management Transfer management and optimization Reliability via transfer retries Web interface, REST API, command line One-click “Globus Connect Personal” install 5-minute Globus Connect Server install

High-speed transfers to/from AWS cloud, via Globus transfer service UChicago  AWS S3 (US region): Sustained 2 Gbps 2 GridFTP servers, GPFS file system at UChicago Multi-part upload via 16 concurrent HTTP connections AWS  AWS (same region): Sustained 5 Gbps

Globus transfer & sharing; identity & group management, data discovery & publication

Globus under the covers

Globus under the covers

Globus Platform-as-a-Service

The Globus Galaxies platform: Science as a service

Three big data challenges Channel massive flows Automate management Build discovery engines

Discovery engines: Integrate simulation, experiment, and informatics

A discovery engine for metagenomics

DOE Systems Biology Knowledge Base (KBase)

A discovery engine for the study of disordered structures

Immediate assessment of alignment quality in near-field high-energy diffraction microscopy

New data, computational capabilities, and methods create opportunities and challenges

Big Data to Knowledge: bd2k.org

Three big data challenges Channel massive flows New protocols and management algorithms Automate management The Discovery Cloud Build discovery engines MG-RAST, kBase, Materials

My work is supported by:

Thank you! foster@anl.gov ianfoster.org