SparkML basics презентация

SparkML basics Dmitry Bugaychenko

RDD Basics

RDD Basics

RDD Basics

RDD Basics

DataFrames

Datasets

SQL vs. DataFrame vs. Dataset

Spark ML Pipelines

Spark ML Pipelines Transformer

Spark ML Pipelines Transformer

Spark ML Pipelines

Spark ML Pipelines

Spark ML Pipelines

Spark ML Core

Field Metadata and Attributes

Prediction Model

“My Spark ML Model”

Spark ML Features ETL SQLTransformer SqlFilter, ColumnsExtractor Numerization OneHotEncoder StringIndexer MultinomialExtractor Vectorization VectorAssembler FeatureHasher AutoAssembler

Spark ML Features Feature Engineering DCT ElementwiseProduct Interaction VectorIndexer PolynomialExpansion Feature Selection ChiSqSelector FoldedFeaturesSelector

Spark ML Features Texts extraction Tokenizer RegexTokenizer Ngram StopWordsRemover NLP in Pravada-ML LanguageDetectorTransformer LanguageAwareAnalyzer NGramExtractor URLElimminator

Spark ML Features Regression

Spark ML Features Recommendations ALS FPGrowth Evaluation BinaryClassificationEvaluator ClusteringEvaluator MulticlassClassificationEvaluator RegressionEvaluator

Thank you for your attention! Dmitry.Bugaychenko @corp.mail.ru