rclone-colab
·
Editors' PicksFeaturesDeep DivesGrowContribute
Why You Should Try Rclone with Google Drive and Colab
Wouldn’t it be nice if you never lost your trained ML models on the Google Colab? Rclone to the rescue.
Image by author.
Do you ever start training a machine learning model on the Google Colab and the runtime disconnects before you can save your results? It use to happen to me and I would get frustrated. I would leave my model training to come back a few hours later and find it finished without saving!
Image by author.
So what can you do about this? Use Rclone¹, Google Drive², and Google Colab³ together with the following 3 major steps:
1) Install Rclone.
2) Configure Rclone.
3) Export with Rclone.
Step 1. Install Rclone.
Do this in the very first cell of your Google Colab Jupyter Notebook.
Step 2. Configure Rclone.
Configure the newly installed Rclone.
Step 2.1. Make a new remote
Step 2.2. Name it “remote”
Step 2.3. Enter “15” for the Google Drive
Step 2.4. Press Enter for default on client_id
Step 2.5. Press Enter for default on client_secret
Step 2.6. Enter “1” for Full access all files on your Google Drive.
Step 2.7. Press Enter for default on root_folder_id.
Step 2.8. Press Enter for default on service_account_file.
Step 2.9. Enter “n” for default on advanced config.
Step 2.10. Enter “n” for non-default on Remote config.
Go to the link it generates for you.
Copy the code from your Google Drive Sign in for Rclone access.
Paste it where it tells you to Enter verification code
Step 2.11. Enter “n” for default on Shared Drive.
Step 2.12. Enter “y” for default on remote settings.
Step 2.13. Enter “q” for Quit config.
Step 3. Export with Rclone.
Add an export after your model finishes training.
Copy your current “/content/” directory in the Google Colab to wherever you want on your Google Drive through the Rclone remote path we just established.
Summary
So after following those 3 major steps outlined above, you should be able to back-up whatever models you are training with Google Colab (assuming they finish).
Here is an example Jupyter Notebook that implements theses steps from start to finish.
You can see for that example, my Google Drive stored the trained models as such under content:
Image by author.
You can find more examples in this Github repository I made for various TensorFlow machine learning problem notebooks.
I hope you found this article helpful! Thank you for reading and hit the like if so. Follow me on here for more machine learning based content coming soon!
You can also follow me on LinkedIn: https://www.linkedin.com/in/stevensmiley1989/
References
Rclone.[ https://rclone.org/]
Google Drive. [ https://g.co/kgs/qu7aAY]
Google Colab. [ https://colab.research.google.com/]
TensorFlow. MartĂn Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis,Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Rafal Jozefowicz, Yangqing Jia,Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan ManĂ©, Mike Schuster,Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Jonathon Shlens,Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker,Vincent Vanhoucke, Vijay Vasudevan, Fernanda ViĂ©gas,Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke,Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large-scale machine learning on heterogeneous systems, 2015. Software available fromtensorflow.org.
Writing about Data Science, CV, DL, ML, AI, Python https://www.linkedin.com/in/stevensmiley1989/
45
45
45
Your home for data science. A Medium publication sharing concepts, ideas and codes.
Read more from Towards Data Science
More From Medium
Dmytro Nikolaiev (Dimid)in Towards Data Science
Arun Ramanujapuramin logistimo
Tariq Munirin Towards Data Science
Last updated
