Sometimes users notice that the Apache service fails in the Linux server with the error "semget: No space left on device". Generally, this error indicates that your server has run out of semaphores.
Here at Ibmi Media, as part of our Server Management Services, we regularly help our Customers to perform related Apache queries.
In this context, we shall look into methods to resolve this Apache error.
Recently, we received a support request where Apache fails and is not starting again. We immediately checked the "/usr/local/apache/error_log" and found the error as given below:
semget: No space left on device
This error generally indicates that the server has run out of semaphores and apache cannot be started.
To see how many semaphores are being used, access the server via SSH as root and run the following command:
# ipcs -s
For the Apache to get started again, we must clear the semaphores. Run this for-loop to flush them:
for whatever in `ipcs -s | awk '{print $2}'`; do ipcrm -s $whatever; done
On older servers, this command may not work. In these cases, we may need to do the following:
# /sbin/service httpd stop
# ipcs -s | grep nobody | gawk '{ print $2 }' | xargs -n 1 ipcrm sem
# /sbin/service httpd start
If this is a recurrent problem for us, we have to increase the semaphore limits on the server. We can do that by adding the following to the /etc/sysctl.conf file:
# Increases the semaphore limits & extend Apache's uptime.
kernel.msgmni = 512
kernel.sem = 250 128000 32 512
Then load the new settings into the kernel:
# sysctl -p
This article covers methods to resolve Apache Error: "semget: No space left on device". This error indicates that apache failed and will not start again, so check the error log If you see an error similar to the following, it could indicate that your server has run out of semaphores and apache cannot be started:
[emerg] (12)Cannot allocate memory: mod_fcgid: Create process manager error
[error] (28)No space left on device: Cannot create SSLMutex Configuration Failed
To solve this problem you can restart Apache, Postgres and other services that consumer many IPC resources or increase limit of the resources in the system using 'sysctl'. When you stop all services the semaphores and shared memory segments have to be removed, if not, and you still able to see them using 'ipcs' command, try to remove them manually using 'ipcrm' command.
1. For example to remove semaphore:
# ipcs -a
2. If this is a common problem for you, you may want to increase the semaphore limits on your VPS server. You can do that by adding the following to the /etc/sysctl.conf file:
# Increases the semaphore limits & extend Apache's uptime.
kernel.msgmni = 512
kernel.sem = 250 128000 32 512
3. Then load the new settings into the kernel:
sysctl -p