Freeing Up Disk Space (in my mind!)

I haven’t been as good at writing up daily sysadmin stuff as I should have, but I do reference this blog regularly for issues I have when playing a sysadmin on TV. One article I go to time and gain is the fix account permissions for a cPanel user, it has proven to be an indispensable reference, although by now I have the ‘fixperms’ command committed to memory. So, in hopes that I get better at this practice, I wanted to quickly chronicle some commands yesterday that helped me out with a storage issue one of our server’s was having.

We use Digital Ocean for a large swath of our servers now, and one of the key reasons we switched was block storage. Block storage enables us to mount significantly more storage then we would get with a normal virtual private server.  For example, the server options at Digital Ocean provide 40Gb, 40GB, 80GB, 160GB etc. of storage space with their increasingly more powerful CPU/Memory options for servers. But if you are running a shared hosting server, you need anywhere from 500GB to 1 TB to meet the needs. Block storage allows us to add the additional 500+ GB to a server that may come stock with 80Gb. So, that’s what block storage is, additional storage you mount to your original server.

Now, yesterday a server was not responding reporting space issues. The dreaded 500 Internal Server errors rides again:

Internal Server Error 500:  The system failed to open the session file “/var/cpanel/sessions/raw/johnaste:r15mndEoycCpsOQo” because of an error: No space left on device at /usr/local/cpanel/Cpanel/Session.pm line 269.

When this happens it is usually a daily backup plugin someone installed for a fairly large site that is eating through the storage. To confirm that, we use the following command:

df -h

or to include types

df -Th

This will give you a list of your file systems and the space they take up in a human readable form (hence the -h):

The /dev/sda is the mounted storage block, and you’ll notice this has over 163GB free. No problem there, which confused me at first because the mounted disk is where all the new cPanel account are created and files are stored.  This file system is mapped on top of the /home directory of the server where all new cPanel accounts are created and stored. But you will also notice there is no storage left on /dev/sda1 which is where all the core cPanel files are. That’s the issue.

The next thing I tried to do is install the tool NCDU, which is disk usage analyzer which will run through all your various folder and let you know where any big files are so you can start removing them.  This is crucial when we have hundreds of cPanel account in distinct folders that we need to establish where the system storage drain is located. So, I tried to run

NCDU

But it was not installed yet. I then tried to install it

yum install ncdu

But given there was no more space I could not. Hmmm, so I found this article with yet another disk usage command that was quite helpful:

du -sh *

This did the trick, a directory within the backup folder (2017-09-13) on the main file directory with has 53 GB , which was a flag. I deleted the directory contents using this commands

rm -rf /backup/2017-09-13

And the system was back up immediately. A pretty simple fix*, but having the right commands goes a long way. Errors are scary, but reading the error and Googling can reap remarkable results 🙂  The web resources are really amazing for this stuff. It is not that different than figuring out how to hack WordPress themes and plugins, but the more you do it the quicker you get, and only this time hundreds of people depend on the fix rather than the two who read bavatuesdays. 

*Turns out the backups were being mistakenly stored in this folder en route to AWS for another project.