'No More Space' and Can't connect SSH to the Server

Hello, I want to share my Experiment of this 'Little Nightmare' about insufficient Storage Space in running live Server, in this case I use Google Cloud Services : Compute Engine and Ubuntu as the Operating System.

Note :
this post contain some Story, Trial & Error, Skip to the Solution Section if you want to see the Answer.

The Problem

I've got message from my friend, he sent me this Error Message :

and he also can't execute some ssh command on the Server, it always return this :

cd /va-bash: cannot create temp file for here-document: No space left on device

then, i told him to run this command for check directory file size :

du -h --max-depth=1

this is example of the output from command above, picture taken from Twitter

at this moment, he can running rm -rf to cleanup some Files / Directory and free up some space.

now the Server running well and the website up again.

The next day, he sent message to me, the disk on the server running out of space again, and he asked me to increase the size of the disk.

Everything getting worse here, and I've made my mistake, I'm in rush and ask him permission to shutdown the instance for a while, then I increase the size of the Disk from Google Cloud Console page.

after I boot up the instance again, we can't connect to the SSH (using Putty / WinSCP), the server refuse our public key and user profile :

the journey begin...

Attempt to Solve

1. Restart the Instance

The first thing I do is Restart the server, and hope with this little magic will fix the SSH Connect issue, but unfortunately everything still not working.

2. Connect to SSH in Google Cloud Console

Then I try to connect the SSH from Google Cloud Console Instance Page, and Still no luck, Connection Failed

3. Connect to Serial Console

So I start to Google my problem, and found some answer from Serverfault and Stackoverflow to turn on the Serial Console and try to login from there using this Startup Script :

#!/bin/bash
adduser USERNAME
echo USERNAME:PASSWORD | chpasswd
usermod -aG google-sudoers USERNAME

Unfortunately, the startup script Failed to Running, and I see this error from Serial Console :

GCEGuestAgent[1121]: 2022-02-28T14:40:50.0037+07:00 GCEGuestAgent Error non_windows_accounts.go:144: Error creating user: useradd: /etc/passwd.1520: No space left on device#012useradd: cannot lock /etc/passwd; try again later.#012.

Start to PANIC... then I think that we locked to the Server and almost can't do anything.

Moment of Silent...

4. Create new 'restore' disk with old disk snapshoot

After read some related answer on the forum that I've got from google, I try to create new Disk with bigger Storage Size (from 10GB to 20GB) and using old disk snapshoot as the source

I hope with this new Disk and bigger size everything will run smoothly when I attach this new disk as Boot disk to the Instance without losing any existing data. But Unfortunately, when the server run, we still Refused from the Server by using Putty / WinSCP and also Google Cloud Console SSH.

Solution

Almost gave up, but I will try my last Shoot. and YEAH!! it's working.

here is the step that I do to solve this problem

1. Create new Instance with new Disk and new OS Image

Creating new Instance with new Disk with Ubuntu and start it, make sure I can connect using SSH.

the idea is to Attach the old disk to this new Instance as additional disk

2. Attach old 'fully' disk as additional disk to the new Instance

go to Compute Engine Instance List, and choose the new instance then click the edit button.

find the Additional Disk Section and Click Attach Existing Disk, then choose the old disk.

Make sure to detach the old disk from Boot Disk at the main instance

3. Mount disk to filesystem folder

After attach the old disk, run this command to check if the disk Exist / loaded in the instance :

sudo lsblk

and it will show the list of the disk partition

in my case, the old disk identified by sdb and the partition that we need to fix is at sdb1 to filesystem, let's say we mount it to /myOldDisk in the new instance

• create myOldDisk folder by execute this command ```

sudo mkdir -p /myOldDisk

- mount sdb1 to /myOldDisk

sudo mount -o discard,defaults /dev/sdb1 /myOldDisk

- configure permission

sudo chmod a+w /myOldDisk

- run sudo lsblk again, now I can see my sdb1 mounted to /myOldDisk


### 4. Clean up or backup to local computer some files for free up more space in the mounted folder

since our old Disk mounted to **/myOldDisk** I can easily go to the directory then cleanup some files / folder to get more free space

cd /myOldDisk

then I can find some big files to delete or archive it as zip and download it to my local computer

I get 500MB+ free space and I thing it's save to go

### 5. Umount the old disk
unmount the sdb1 from filesystem by running this command :

umount /dev/sdb1

detach my old disk using Google Cloud Console at this new Instance Detail page, then we can shutdown and delete this new Instance & new Disk, we don't need that anymore.

### 6. Increase the old disk size at Google Cloud Console
Go to Google Cloud Console Disk Page, and select the old Disk, and increase size, in my case is from 10GB to 30GB, and save it.

### 7. Attach the old disk to the old Instance
I need to Attach my old Disk to my old Instance as Boot Disk, go to Compute Engine Instance Detail, edit the instance and add the Boot Disk, select my old Disk.

### 8. Run the instance and Growpart the Partition
Turn on the Instance, Finally now I can login SHH to the Server,
when I check the free storage by running **df -h** command, the storage space still 10GB.

my next step is to Update the disk partition by using Growpart from 10GB (existing) to the new size 30GB

I can check the new size by running :

sudo lsblk

It will show new **dev/sda** storage size, all i Need to do is Grow the Partition by running this command :

growpart /dev/sda 1
CHANGED: partition=1 start=2048 old: size=419428319 end=419430367 new: size=2097149919
,end=2097151967

then I need to Resize the Partition by execute this command :

sudo resize2fs /dev/sda1

it will return something like this :

resize2fs 1.42.13 (28-Feb-2022)
Filesystem at /dev/sda1 is mounted on /; on-line resizing required
old_desc_blocks = 13, new_desc_blocks = 63
The filesystem on /dev/sda1 is now 262143739 (4k) blocks long.

Finally Done ! now the Disk have extra space, I can verify the available / free storage by running **df -h**

_It's time to rest and drink some Coffee.._

## Summary

from this Experience I Learn : 

- Never ever Shutdown your 'full storage' server, Free up some space before increasing the disk size, and make sure everything is fine before the shutdown.

- You don't need to shutdown the server for increasing the storage Disk

I'm so sorry for my bad English, and I hope this post will be guide Me or Someone else when facing the same problem in the future.

_if you have suggestion about this post feel free to write it down in the comment section below :)_