Hongyi-Lee Machine Learning Spring 2021 Mandarin

learning-note

字数统计: 1.2k阅读时长: 6 min

 2021/03/17  61  Share

Hongyi-Lee Machine Learning Spring Mandarin —Notes

Lesson 1Basic Conceptions OF ML

Different type of Functions

Regression: The function outputs a scalar

Classification: Given options(classes), the function outputs the correct one.

How to find the Function

Three steps

Function with Unknown Parameters
Define Loss from Training Data
- Loss is a function of parameters: L( b,w )
- Loss: how good a set of value is.
Loss: $L = \frac{1}{N}\sum{n}e{n}$

Types of Loss
- MAE: L is mean absolute error(MAE) $e = |y-\hat{y}|$
- MSE: L is mean square error(MSE) $e = (y-\hat{y})^{2}$
- if y and $\hat{y}$ both probability distributons —> Cross-entropy
Optimization

the way to find a best w and b: gradient descent
- randomly pick a initial value of w0
- compute $\frac{\partial{L}}{\partial{w}}|_{w=w^0}$
  - Negative —> increase w
  - Positiive —> decrease w
  - learn rate : hyperparameters —> can modify be human
- update w iteratively

disadvantage of gradient descent: may sometimes can not find the absolutely right best w* —> local minima

Same steps of the finding of b*

Above summary is called Train

**Gradually considering more days of a circle to compute the best w* (different models)**

Less2 Basic Conceptions OF ML (second)

Model Bias

Approximate cotinuous curve by a piecewise linear curve.

To have good approximation, we need sufficient pices.

using multiple lines to fitting the red curve

New Model: more Features

using /theta to represent all unknow parameters

Loss

Loss is a function of parameters $L\theta$

Loss means how good a set of values is.

Optimization of New Model

Activation function

ReLU = 2 times Sigmoid

complexity model

Why don’t we go deeper? —— Overfitting

Less3 Mission Strategy OF ML

how to improve the currection

56-layer can do the things 20-layer do

The way to solve overfitting

constrained model to solve overfitting

Bias-Complexity

good way to improve currection

Split Training Set into Training Set and Validation Set

Mismatch

Lesson4 local minima saddle point

Critical point

using Tayler Series Approximation to approximately represent the Loss Function

$(\theta-\theta^{‘})^T g$ will disappear when Loss is at critical point

eigen value

Example

Higher Dimension

saddle point in higher dimension

Lesson5 Batch && momentum

Large Batch need smaller time for one epoch while Small Batch need more time for one epoch

What’s wrong with large batch size? —> Optimization Fails

Small batch: when one batch cracked in a critical point, another somehow will over come it.

small batch is better on test data.

small batch is worse on test data. —> overfitting

Many differences on Small Batch V.S. Large Batch

Momentum

Obey the inverse orientation of Gradient Descent to update the Next New Gradient

Lesson6 Learn rate (Error surface is rugged)

Tips for training: Adaptive Learning Rate.

Training stuck != Small Gradient

Error surface is left side of above pic.

we can see the gradient vibrate between the two peak.

Learning rate too big or too small.

Learning rate adapts dynamically

RMSProp is unpublished optimization algorithm designed for neural networks, first proposed by Geoff Hinton in lecture 6 of online course.

Learning Rate Scheduling

Decay
Warm up

Lesson 7 Loss Function Influence

Mean Square Error(MSE) VS Cross-entropy

Lesson8 CNN - Image Classification

All weight == input x kernal x out.

Identifying some critical patterns to ensure a certain object.

Receptive field design

can different neurons have different sizes of receptive field.
cover only some channels.
not square receptive field.

Typical Setting of Receptive neural

kernal size usually is 3x3
one receptive neural typically has 64 neurons
stride
overlap
padding: when over the whole image

The same patterns appear in different regions.

does every region need a pattern detection?

Solution to above issue ==> Parameter Sharing

same weight in different regions

different weight in two neurons with the same receptive field == two neurons with the same receptive field would not share parameters.

Benefit of Convolutional Layer

Another good story to describe Convolutional Layer

Feature Map

1channel x 64 filter —> “image” with 64 channels x 3x3x64 filter —> “image” with 64x3x3x64 channels —>…

When with deeper Convolutional Layers.

Pooling

Max pooling: select the max num of a receptive filed.

pooling: do the things that makes the image smaller.

pooling: sometimes make some damages in the result of Image Identification.

The Whole CNN

Flatten: makes the latter matrix like vector.

Then do with a softmax.

Why CNN for Go Playing?

some patterns are much smaller the the whole image
The same patterns apear in different regions.

Alpha Go does not use Pooling…

CNN need data augmentation in Image Identification work.

Lesson9 Self-Attention (part1)

货拉拉拉不拉拉布拉多犬呢

When the input is a set of vectors ( not one vector )

What is the output?

N to N

N to One

N to somewhat. seq2seq

seq2seq:

translate
speech recognization

Sequence Labeling: N2N

Self-attention

Attention is all you need.

First of all, input into a self-attention layer; then it will output a series of different vectors; a series of different vectors input into Full Connection Layers.

FC and self-attention can be traded off to use. ( alternate use )