Update December 2020: I have published a major update to this post, where I cover TensorFlow, PyTorch, PyTorch Lightning, hyperparameter tuning libraries — Optuna, Ray Tune, and Keras-Tuner. Along with experiment tracking using Comet.ml and Weights & Biases.
The recent announcement of TPU availability on Colab made me wonder whether it presents a better alternative than GPU accelerator on Colab or training locally. So, I decided to take it for a spin using very simple notebook that trains a convnet to classify CIFAR10 images.
Without further ado, here are the results.
All three notebooks use the same network architecture, batch size and hyperparams.
However, it is not a fair comparison because the notebook perhaps doesn’t use the code optimized for target hardware, neither does it account for hardware differences between the local vs cloud VMs. Using local workstation with a good NVIDIA GPU works best but with Colab we are free from the troubles of cuda installations/upgrades, environment management or package management.
To me, I wanted to find out whether I’ll get any benefits if I just took my notebook from local machine to Colab?
If I am using my laptop, the silver colored one with AMD GPU, the anwser is yes I would definitely use Colab GPU. My code will run as is and without needing any wrappers. TPU Accelerator on the other hand does require wrapping the model around contrib.tpu and does not seem to support eager mode yet. But I expect these to go away as TPU moves from contrib to core.
Conclusion
We live in great times where compute and infra such as Google Colaboratory is freely accessible to anyone with internet connection. All you need is a laptop, desktop or even a piTop (haven’t tried it) and you get instant access to a GPU or TPU.