Having learned my lesson, I'm going through video 1.4.1 of https://github.com/DataTalksClub/data-engineering-zoomcamp in a more orderly way. First, I watched the whole video, "Setting up the Environment on Google Cloud". Now I'm going through it at half speed, taking notes. When I get to a good stopping point, I go through my notes and implement the instructions. Google cloud scares me. I can never find anything on it, so this is necessary.
To start, I go to console.cloud.google.com and make sure my project is selected in the upper left. How to set this up, I can't tell you. I set up my project two years ago. First time users can get a $300 credit. Otherwise you're going to pay. Just make sure you have a project set up, and make sure you're in it.
In the left menu, choose "Compute Engine->VM Instances". You have to click on "Enable the API".
First you have to generate an ssh key to use to log in to the instance. Go to https://cloud.google.com/compute/docs/connect/create-ssh-keys. There's a line of code there that you enter in the terminal window on your computer. Open a terminal window and type:
ssh-keygen -t rsa -f ~/.ssh/KEY_FILENAME -C USER -b 2048
(You should have a dot ssh directory in your home directory - create one first if you don't.)
Substitute something like gcp for KEY_FILENAME and your name for USER. Enter nothing for the passphrase. ssh-keygen will create two keys. In my case, gcp and gcp.pub. gcp is the secret secret private key that you keep on your computer. gcp.pub will now go into the settings. Type "cat gcp.pub" and copy the key. Not gcp which is your private key. Back to Google Cloud in the browser, navigate to settings->metadata. Choose to add an ssh key, not a metadata. Paste it into the ssh key box and hit SAVE.
Next, still in the browser, go to VM Instances. Choose: CREATE INSTANCE. You'll have to set some configuration variables. First, name the instance, and select a region. I used de-zoomcamp and US-east4.
Next, choose an instance. I chose e2-standard-4 with 4vCPU and 16GB memory. This is what the instructor used, and I'm sticking with his recommendation.
Finally, change the Boot disk. He chose Ubuntu 20.04LTS, and changed the size from 10G to 30G. I did the same for consistency. Then I hit SELECT and at the bottom, CREATE.
Once it's created, you'll see a row of data. Find External IP and copy it. Now you'll try to connect.
Back to your computer. In the terminal window, go to your home directory. Type: $ssh -i ~/.ssh/gcp name@External IP.
Here, you use your private key. You should connect right away. If you type $ls, you'll see there is nothing. We will have to provision the instance. One good thing is that we have the google cloud sdk. You can type $gcloud --version to see it. In my next post I'll write about adding everything we need to the Instance, so we can work from there rather than from our computers.
Top comments (0)