How to Dynamically Adjust Process Limit Quotas in Linux

Imagine that you have an Apache web server that acts as a proxy, for years everything worked well and smoothly.. but out of the blue you started getting log messages straight from hell:

[Fri Jan 17 02:11:43.271513 2025] [proxy:error] [pid 3979:tid 140012208011008] (24)Too many open files: AH00952: HTTP: error creating fam 2 socket for target network-pos

The proxy cannot longer serve requests towards to the backend because too many open files! fuck! what that means? what i should do?. Lets pause a bit.. this is actually overdramatization and the situation was not so bad but i wanted just to get your attention.. so in technical terms.. the Apache web server acting as a proxy runs with a process named httpd with user apache, in Unix and Linux each user/group has a Limit in resources such as number of open files. but not only limited to number of open files there are more even resources to set limits. In modern systems that uses systemd overwrites limits.conf because it allows a more fine granular control to the resources per service and not only by user, so if you want for example to change the max file number of process httpd which is under systemd enter

systemctl edit httpd.service

Probably the service in your system will not be httpd so you must replace it with the proper service file, now in the editor enter the following and save

[Service] 
LimitNOFILE=65535

This will set the Max Limit of Open files to 65535, adjust it to your needs!

For changes to take effect you will need to restart httpd , now you will thinking “hey! you said change quota dynamically” and yes.. i will keep my promise.. the previous step was necessary if you want to make the change persistent after a service / system restart.

The tool that allows to change the process quota limits on the fly is called prlimit and you can install it on ubuntu using the following command

sudo apt update 
sudo apt install util-linux

Next is to locate the PID of the process, here might be a problem! some processes like Apache web server might spawn many PID, so you need a way to locate them all and apply the new limits for the resources you want, to do this we can use the pidof tool which shows all the child PID and the main PID of a process, like in the following example:

pidof httpd 
24398 22644 19953 19391 17704 14028 13440 12970 12967 12791 9209 8705 4713

Now we know how to get all PID of a process lets see how prlimit works, in the following example we say PID to change the Max Open files limit to 16384 for a PID

prlimit --pid=<PID> --nofile=16384:16384

Lets see how we can make it to change the limits for all PID of a process, isnt that magic? we looped all PID of httpd and applied the new limits

for pid in $(pidof httpd); do prlimit --pid=$pid --nofile=16384:16384; done

Lets now verify the change, to do this we can create an oneliner that parses /proc/ pseudo file system for all the PIDs of httpd process

for pid in $(pidof httpd); do echo "PID $pid: $(cat /proc/$pid/limits | grep 'Max open files')"; done

the /proc/<PID>/limits is a file in this pseudo file system which shows for each PID details about its limits, for my system verifies that max open files are indeed now 16384 for each process

PID 24398: Max open files            16384                16384                files 
PID 22644: Max open files            16384                16384                files 
PID 19953: Max open files            16384                16384                files 
PID 19391: Max open files            16384                16384                files 
PID 17704: Max open files            16384                16384                files 
PID 14028: Max open files            16384                16384                files 
PID 13440: Max open files            16384                16384                files 
PID 12970: Max open files            16384                16384                files 
PID 12967: Max open files            16384                16384                files 
PID 12791: Max open files            16384                16384                files 
PID 9209: Max open files            16384                16384                files 
PID 8705: Max open files            16384                16384                files 
PID 4713: Max open files            16384                16384                files

Now with the combination of systemd settings and prlimit even in case of a service / server restart the number of open files setting will be persistent

Conclussion

Of course one very important thing yo must have in mind is why in the first place the number of open files increased over the predefined limit especially if the Apache web server worked for a long period without such issues, so you those instructions as a way to mitigate the problem but also start troubleshooting to find out why this happened