DEV Community

Cover image for Running computational tasks indefinitely on the cloud
Andrew Garcia
Andrew Garcia

Posted on • Edited on

Running computational tasks indefinitely on the cloud

I have been running my most resource-intensive computations on the cloud (that is, on commercial external servers) rather than on using the CPUs from my personal computer. This is advantageous for many reasons such as liberating your computer from resources and in doing so, avoiding lagging or other undesired effects. These 2 reasons are most unique to the application:

  1. Computational tasks can be kept running in the cloud even after your computer is turned off.
  2. If more cloud resources are needed, you can upgrade your resources while keeping your same virtual environment at all times (i.e. programs / applications installed on the cloud space)

I have outlined the process to sending your calculations to the cloud with DigitalOcean as well as other useful features you may use for cloud computing below.

Outline

  1. Sign-up to DigitalOcean
  2. Python Users
  3. Installing Git on droplet
  4. Transferring files between droplet and computer
  5. Run processes in the cloud even after turning off your computer
  6. How to take a snapshot of your created droplet
  7. Machine Learning: Installing Tensorflow

Let's begin.

Sign-up to DigitalOcean

Referral link: Sign up with a $100 credit over 60 days through here

I have used DigitalOcean virtual machines known as droplets for this cloud computing activity. I consider the DigitalOcean platform quite user-friendly and their pricing of droplets is also very reasonable and flexible; the cheapest droplet, for instance, will only cost about $5 a month and if you decide to only use it for 2 days, the price would be around $(2/30) x 5 = 33 cents TOTAL and not $5 (as long as the droplet is destroyed after the 2 days, see more on this below) You may click on the link above to sign up

DigitalOcean Referral Badge

Python users: Installing Anaconda python on droplet

https://www.digitalocean.com/community/tutorials/how-to-install-the-anaconda-python-distribution-on-ubuntu-20-04

Once you have the droplet, I recommend installing the Anaconda suite which comes with several python modules pre-packaged. The tutorial above allows you to do so with command lines alone.

Installing Git on droplet

https://www.digitalocean.com/community/tutorials/how-to-install-git-on-ubuntu-20-04#setting-up-git

Git may already be installed in your DigitalOcean droplet, but above is the tutorial nonetheless. Git can be used to communicate with your Github repository, so that Python programs developed in your computer can be pulled to your droplet, and viceversa. (P.S. You may also need a Github account)

Transferring files between droplet and computer

https://docs.digitalocean.com/products/droplets/how-to/transfer-files/

This approach uses Filezilla to do this. I use this in the case my programs generate files and I need to transfer them from droplet to computer.

Use screen to run process on droplet even after turning off your computer or exiting ssh

https://www.digitalocean.com/community/tutorials/how-to-install-and-use-screen-on-an-ubuntu-cloud-server

Screen is a program which allows you to do just that. You may need to install it INSIDE your droplet (i.e. ssh protocol) and then call "screen". After calling screen you can run your program (e.g. "python myprogram.py") and then press "Ctrl+A", followed by letting go off the keys, followed by pressing "D" to detach the screen.

Then you may exit the droplet or even turn off the computer and the python process will keep on running! To access again, access the droplet, then type "screen -r" to re-attach the screen and see how far your process has been running on the server.

How to take a snapshot of created droplet

https://docs.digitalocean.com/products/images/snapshots/how-to/snapshot-droplets/

In case you have been using an expensive droplet and you do not want to be charged for it for a period of time, you may need to destroy it. No worries! If you want to "revive" your droplet later on preserving all its installed programs and environment*, you may do so by taking a snapshot. Click on the link above for this option.

How to install TensorFlow on droplet

https://www.digitalocean.com/community/tutorial_collections/how-to-install-and-use-tensorflow

If you're ready to take your droplet to the next level, machine learning, then this tutorial may be just for you. The current recommendations are to install TensorFlow in a droplet with at least 4GB RAM.

Let's say you created your whole development environment on a smaller droplet. That's ok. You may take a snapshot of the smaller droplet and then re-execute it on a larger memory droplet. This strategy may also be useful if you would want to take time setting up your development platform on a smaller droplet to spend less money before moving to the larger one.

Conclusion

DigitalOcean droplets are designed to be suitable for developers and small applications. Their application to cloud computing goes beyond python, as it can be extended to as many programming languages as can be run on a computer.

Top comments (0)