Understanding Feature Space in Machine Learning презентация

Understanding Feature Space in Machine Learning Alice Zheng, Dato September 9, 2015

My journey so far

Why machine learning?

The machine learning pipeline

Feature = numeric representation of raw data

Representing natural text

Representing natural text

Representing images

Representing images

Feature space in machine learning Raw data  high dimensional vectors Collection of data points  point cloud in feature space Model = geometric summary of point cloud Feature engineering = creating features of the appropriate granularity for the task

Crudely speaking, mathematicians fall into two categories: the algebraists, who find it easiest to reduce all problems to sets of numbers and variables, and the geometers, who understand the world through shapes. -- Masha Gessen, “Perfect Rigor”

Algebra vs. Geometry

Visualizing a sphere in 2D

Visualizing a sphere in 3D

Visualizing a sphere in 4D

Why are we looking at spheres?

The power of higher dimensions A sphere in 4D can model the birth and death process of physical objects Point clouds = approximate geometric shapes High dimensional features can model many things

Visualizing Feature Space

The challenge of high dimension geometry Feature space can have hundreds to millions of dimensions In high dimensions, our geometric imagination is limited Algebra comes to our aid

Visualizing bag-of-words

Visualizing bag-of-words

Document point cloud

What is a model? Model = mathematical “summary” of data What’s a summary? A geometric shape

Classification model

Clustering model

Regression model

Visualizing Feature Engineering

When does bag-of-words fail?

Improving on bag-of-words Idea: “normalize” word counts so that popular words are discounted Term frequency (tf) = Number of times a terms appears in a document Inverse document frequency of word (idf) = N = total number of documents Tf-idf count = tf x idf

From BOW to tf-idf

From BOW to tf-idf

Entry points of feature engineering Start from data and task What’s the best text representation for classification? Start from modeling method What kind of features does k-means assume? What does linear regression assume about the data?

That’s not all, folks! There’s a lot more to feature engineering: Feature normalization Feature transformations “Regularizing” models Learning the right features Dato is hiring! [email protected]