Sunday, 12 April 2020

"No space left on device" Error while there are plenty of free space

One week back I came across the following clusterware error:

2020-04-09 05:17:38.726 [OCSSD(27899)]CRS-10000: CLSU-00100: operating system function: mkdir failed with error data: 28
CLSU-00101: operating system error message: No space left on device
CLSU-00103: error location: authprep6
CLSU-00104: additional error information: failed to make dir /u01/12.1.0/grid/auth/css/fzpson06pe1p/A1673086


When checking /u01 filesystem, I found that it already has much of free space :


[oracle@fzpson06pe1p fzpson06pe1p]$ df -hP /u01
Filesystem                          Size  Used Avail Use% Mounted on
/dev/mapper/VolumeGroup00-LogVol07   99G   59G   36G  63% /u01


I remember I had a similar issue many years back and it was due inodes exhaustion.
When checked the inodes with df -i it was totally exhausted:

[oracle@fzpson06pe1p ~]$ df -i /u01
Filesystem                         Inodes IUsed IFree IUse% Mounted on
/dev/mapper/VolumeGroup00-LogVol07   6.3M  6.3M   0  100% /u01


And yes, running out of inodes can paralyze the filesystem as similar as running out of space.

But what are the inodes?
In short, inodes are the records that hold the metadata of the directories/files which stored under the filesystem. Metadata like; on which blocks the file is stored, its owner and permissions and so on...
Each Linux filesystem has a limited number of inodes depending on the filesystem size. This limitation of inodes limits the number of directories/files that can be stored under the filesystem as well.

Now, how to fix this situation?
Just delete/cleanup the unwanted files under the impacted filesystem. In my case the impacted filesystem was /u01 where Grid Infrastructure and Oracle DB software were installed. So, I cleaned up old audit logs and trace files to release the inodes.

Once I cleaned the audit files alone the inodes utilization dropped dramatically:

[oracle@fzpson06pe1p ~]$ df -i /u01
Filesystem                         Inodes IUsed IFree IUse% Mounted on
/dev/mapper/VolumeGroup00-LogVol07   6.3M  1.6M  4.7M   26% /u01


Note: In this situation, I'm not concerned about cleaning up big files as I'm concerned to delete as much unneeded files as possible.

In the next few days I'll update the dbalarm script with a new feature to monitor the inodes utilization in the filesystems as well. Stay tuned and download the latest version from here:
http://dba-tips.blogspot.com/2014/02/database-monitoring-script-for-ora-and.html

For more reading about inodes:
https://www.howtogeek.com/465350/everything-you-ever-wanted-to-know-about-inodes-on-linux/