One week back I came across the following clusterware error:
2020-04-09 05:17:38.726 [OCSSD(27899)]CRS-10000: CLSU-00100: operating system function: mkdir failed with error data: 28
CLSU-00101: operating system error message: No space left on device
CLSU-00103: error location: authprep6
CLSU-00104: additional error information: failed to make dir /u01/12.1.0/grid/auth/css/fzpson06pe1p/A1673086
When checking /u01 filesystem, I found that it already has much of free space :
I remember I had a similar issue many years back and it was due inodes exhaustion.
When checked the inodes with df -i it was totally exhausted:
[oracle@fzpson06pe1p ~]$ df -i /u01
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/mapper/VolumeGroup00-LogVol07 6.3M 6.3M 0 100% /u01
And yes, running out of inodes can paralyze the filesystem as similar as running out of space.
But what are the inodes?
In short, inodes are the records that hold the metadata of the directories/files which stored under the filesystem. Metadata like; on which blocks the file is stored, its owner and permissions and so on...
Each Linux filesystem has a limited number of inodes depending on the filesystem size. This limitation of inodes limits the number of directories/files that can be stored under the filesystem as well.
Now, how to fix this situation?
Just delete/cleanup the unwanted files under the impacted filesystem. In my case the impacted filesystem was /u01 where Grid Infrastructure and Oracle DB software were installed. So, I cleaned up old audit logs and trace files to release the inodes.
Once I cleaned the audit files alone the inodes utilization dropped dramatically:
[oracle@fzpson06pe1p ~]$ df -i /u01
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/mapper/VolumeGroup00-LogVol07 6.3M 1.6M 4.7M 26% /u01
Note: In this situation, I'm not concerned about cleaning up big files as I'm concerned to delete as much unneeded files as possible.
In the next few days I'll update the dbalarm script with a new feature to monitor the inodes utilization in the filesystems as well. Stay tuned and download the latest version from here:
http://dba-tips.blogspot.com/2014/02/database-monitoring-script-for-ora-and.html
For more reading about inodes:
https://www.howtogeek.com/465350/everything-you-ever-wanted-to-know-about-inodes-on-linux/
2020-04-09 05:17:38.726 [OCSSD(27899)]CRS-10000: CLSU-00100: operating system function: mkdir failed with error data: 28
CLSU-00101: operating system error message: No space left on device
CLSU-00103: error location: authprep6
CLSU-00104: additional error information: failed to make dir /u01/12.1.0/grid/auth/css/fzpson06pe1p/A1673086
When checking /u01 filesystem, I found that it already has much of free space :
[oracle@fzpson06pe1p
fzpson06pe1p]$ df -hP /u01
Filesystem
Size Used Avail Use% Mounted on
/dev/mapper/VolumeGroup00-LogVol07 99G 59G 36G 63% /u01 I remember I had a similar issue many years back and it was due inodes exhaustion.
When checked the inodes with df -i it was totally exhausted:
[oracle@fzpson06pe1p ~]$ df -i /u01
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/mapper/VolumeGroup00-LogVol07 6.3M 6.3M 0 100% /u01
And yes, running out of inodes can paralyze the filesystem as similar as running out of space.
But what are the inodes?
In short, inodes are the records that hold the metadata of the directories/files which stored under the filesystem. Metadata like; on which blocks the file is stored, its owner and permissions and so on...
Each Linux filesystem has a limited number of inodes depending on the filesystem size. This limitation of inodes limits the number of directories/files that can be stored under the filesystem as well.
Now, how to fix this situation?
Just delete/cleanup the unwanted files under the impacted filesystem. In my case the impacted filesystem was /u01 where Grid Infrastructure and Oracle DB software were installed. So, I cleaned up old audit logs and trace files to release the inodes.
Once I cleaned the audit files alone the inodes utilization dropped dramatically:
[oracle@fzpson06pe1p ~]$ df -i /u01
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/mapper/VolumeGroup00-LogVol07 6.3M 1.6M 4.7M 26% /u01
Note: In this situation, I'm not concerned about cleaning up big files as I'm concerned to delete as much unneeded files as possible.
In the next few days I'll update the dbalarm script with a new feature to monitor the inodes utilization in the filesystems as well. Stay tuned and download the latest version from here:
http://dba-tips.blogspot.com/2014/02/database-monitoring-script-for-ora-and.html
For more reading about inodes:
https://www.howtogeek.com/465350/everything-you-ever-wanted-to-know-about-inodes-on-linux/