Fantastic lecture by Bill Dally (NVIDIA) on Deep Learning Hardware.
With one clear lesson: DEEP LEARNING WAS ENABLED BY HARDWARE.
The theory and the algorithms were known since few decades, but deep learning only started to work when computing hardware became sufficiently advanced to enable the training of a complex network in a reasonable time (the famous AlexNet paper).
Second lesson: hardware is not enough. A properly optimized software stack can lead to performance improvements of orders of magnitude.
Enjoy the video: