1967 ⟶ Stochastic Gradient Descent for Deep Learning

Shun'ichi Amari was the first to use stochastic gradient des...
Timelines Logo
Year
1965
1967
2015

🚀 Development of First Deep Learning Algorithm

Alexey Ivakhnenko and Valentin Lapa developed the first Deep learning algorithm for multilayer perceptrons in Soviet Union.⟶
Development of First Deep Learning Algorithm (1965)
Deep LearningAlexey IvakhnenkoValentin LapaMultilayer PerceptronsNeural NetworksEarly Deep Learning
Soviet UnionSoviet Union

🧠 Stochastic Gradient Descent for Deep Learning

Shun'ichi Amari was the first to use Stochastic gradient descent for Deep learning in multilayer perceptrons. In computer experiments conducted by his student Saito, a five layer MLP with two modifiable layers learned useful internal representations to classify non-linearily separable pattern classes.⟶
Stochastic Gradient Descent for Deep Learning (1967)
Deep LearningStochastic Gradient DescentMultilayer PerceptronsAmariNeural NetworksPattern Recognition
JapanJapan

➕ Highway and Residual Networks Developed for Deep Learning

Two techniques were developed concurrently to train very deep networks: Highway network, and the Residual neural network (ResNet). They allowed over 1000-layers-deep networks to be trained.⟶
Highway and Residual Networks Developed for Deep Learning (2015)
Deep LearningNeural NetworksHighway NetworksResidual NetworksResNetTraining Techniques