Solution For Running Out of Inodes

how-to-solve-running-out-of-inode-issue

Today in this article, we will see how can we solve running out of inode issue. Many times in our system engineering or site reliability engineering career, we encounter this issue. We find that the inode is full for a specific partition.

What should you do to resolve that problem?

Let’s check out!

Before we get into the solution, I prefer to elaborate inode in brief. I feel that it is essential for someone who is new to system engineering or site reliability engineering.

What is an inode?

An inode is a data structure in Linux that stores various information about a file, such as the access mode (read, write, execute permissions), ownership, file type, file size, group, number of links etc. Each inode is identified by an integer number. An inode is assigned to a file when it is created.
Some file systems (specifically ext3) create all inodes when the file system is created. This means that it is possible to run out of storage because all of the inodes are used, although there is available space on the disk. Other file systems (like xfs) create inodes as needed.

You can view a file’s inode number using the ls -il command:

[root@linux hsperfdata_jenkins]$ ls -li
total 32
23869 -rw------- 1 jenkins jenkins 32768 Nov  3 03:50 605
[root@linux hsperfdata_jenkins]$

The first number on the left represents the inode number of the file.

Important Facts!

An inode doesn’t store the content of the file and filename.

How to solve running out of inode issue?

Step 1:
The first step is for it to check whether your server have enough free disk space. Use the below given command for checking the available disk space on the server.

(1)[ROOT@adm10 ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/xvda1             15G  7.4G  6.5G  54% /
/dev/mapper/vgadm10-lvstaging
                      4.9G  370M  4.3G   8% /staging
/dev/md0               31G  9.3G   20G  33% /mnt
(2)[ROOT@adm10 ~]#

Step 2:

The second step is to check the available inodes on your server. For that use below command to check the available inodes on the server.

(2)[ROOT@adm10 ~]# df -i
Filesystem            Inodes  IUsed   IFree IUse% Mounted on
/dev/xvda1            983040 749627  233413   77% /
/dev/mapper/vgadm10-lvstaging
                      327680 327680       0  100% /staging
/dev/md0             1998848     23 1998825    1% /mnt
(3)[ROOT@adm10 ~]#

If the result has 100% or near of IUse% value, then the large number of files is the reason for this issue.

Step 3:

The next step is to find those files. For that we can use a small script which will list the directories and the number of files on them.

#  for i in /*; do echo $i; find $i |wc -l; done

From the output, you can see the directory which uses a large number of files, then repeat this script for that directory like below. Repeat it until you see the suspected directory.

#  for i in /staging/*; do echo $i; find $i |wc -l; done

Step 4:

When you find the suspected directory with large number of unwanted files. Just delete the unwanted files on that directory and free up some inode space by following command.

#  rm -rf /staging/bad_user/directory_with_lots_of_empty_files

If there are many that you want to clear at once, you may want to fork below command

# ls -lrt| grep pattern | awk {'print $9'}|xargs -I {} rm -rf {}

You have successfully solved the problem. Check the inode usage now with the df -i command again, you can see the difference like this.

/dev/mapper/vgadm10-lvstaging
                      327680 297540   30140   91% /staging

If you like to know more information about Linux or Open source module, please click here.

Amazon Kinesis AMI Automation AWS AWS AppStream 2.0 AWS Backup aws certifications aws certifications catalog AWS EBS AWS Elastic Transcoder AWS IAM AWS Sagemaker aws services AWS SES aws support aws swf AWS WorkSpaces AWS X-Ray Azure Cost Management Best Practices chage command Linux Cloud Computing Cloud Migration Data Science DNS Edge Computing Fog Computing Interview Preparation Jenkins Kubernetes Linux Linux User Management Microsoft Azure OSI Model Project Management Python Quantum Computing R sample questions Server Hardening Supercomputer WordPress

Tags: