×


Apache Error: "semget: No space left on device" - Fix it Now ?

Sometimes users notice that the Apache service fails in the Linux server with the error "semget: No space left on device". Generally, this error indicates that your server has run out of semaphores.

Here at Ibmi Media, as part of our Server Management Services, we regularly help our Customers to perform related Apache queries.

In this context, we shall look into methods to resolve this Apache error.


Nature of Apache error "semget: No space left on device"

Recently, we received a support request where Apache fails and is not starting again. We immediately checked the "/usr/local/apache/error_log" and found the error as given below:

semget: No space left on device

This error generally indicates that the server has run out of semaphores and apache cannot be started.

To see how many semaphores are being used, access the server via SSH as root and run the following command:

# ipcs -s


How to resolve Apache error "semget: No space left on device" ?

For the Apache to get started again, we must clear the semaphores. Run this for-loop to flush them:

for whatever in `ipcs -s | awk '{print $2}'`; do ipcrm -s $whatever; done

On older servers, this command may not work. In these cases, we may need to do the following:

# /sbin/service httpd stop
# ipcs -s | grep nobody | gawk '{ print $2 }' | xargs -n 1 ipcrm sem
# /sbin/service httpd start

If this is a recurrent problem for us, we have to increase the semaphore limits on the server. We can do that by adding the following to the /etc/sysctl.conf file:

# Increases the semaphore limits & extend Apache's uptime.
kernel.msgmni = 512
kernel.sem = 250 128000 32 512

Then load the new settings into the kernel:

# sysctl -p

 

[Need urgent assistance to fixing Apache errors? – We're available 24*7. ]


Conclusion

This article covers methods to resolve Apache Error: "semget: No space left on device". This error indicates that apache failed and will not start again, so check the error log If you see an error similar to the following, it could indicate that your server has run out of semaphores and apache cannot be started:

[emerg] (12)Cannot allocate memory: mod_fcgid: Create process manager error
[error] (28)No space left on device: Cannot create SSLMutex Configuration Failed


To solve this problem you can restart Apache, Postgres and other services that consumer many IPC resources or increase limit of the resources in the system using 'sysctl'. When you stop all services the semaphores and shared memory segments have to be removed, if not, and you still able to see them using 'ipcs' command, try to remove them manually using 'ipcrm' command. 

1. For example to remove semaphore:

# ipcs -a

2. If this is a common problem for you, you may want to increase the semaphore limits on your VPS server. You can do that by adding the following to the /etc/sysctl.conf file:

# Increases the semaphore limits & extend Apache's uptime.

kernel.msgmni = 512
kernel.sem = 250 128000 32 512

3. Then load the new settings into the kernel:

sysctl -p