A couple of years ago, I switched our cPanel servers to use the more modern FastCGI Process Manager (PHP-FPM). Although requiring slightly more memory than other PHP FastCGI implementations, the appeal is superior performance, scalability, and security.
All was great, apart from one teeny, tiny issue… PHP-FPM would periodically and randomly go 100% CPU and lock-up, crashing the web server and requiring PHP-FPM to be restarted.
I eventually fixed it, and all was good and stable… until recently, when I migrated to a new server and found myself struggling to remember what I did to fix the issue first time around.
After much searching online, and deep diving into the various settings, I realised this was a common and long-standing problem, which nobody seems to have a viable working solution for.
So this blog is as much to remind my future self as it is to share my simple solution.
Let’s dive in…
Hurt Lockers
My digging around kept coming back to a couple of known bugs in PHP-FPM, which have been kicking around since 2016/2017!
Bug #72590 Opcache restart with kill_all_lockers does not work as expected
Bug #74709 PHP-FPM process eating 100% CPU attempting to use kill_all_lockers
The issue, it seems is caused by the cPanel implementation of PHP-FPM.
On a cPanel server, all pools for a particular version of PHP share the same master process (owned by root), and therefore share the same pool of memory allocated for Opcache.
The actual pools themselves are, however, owned by the user account they serve.
When Opcache reaches its forced restart time, it issues a kill_all_lockers() command.
Opcache then tries to force quit (kill) all the PHP-FPM processes holding locks so it can restart, but because the FPM pools are owned by specific users, it doesn’t have permission… which freaks the server out and causes the crash.
However, the advice to add kill_all_lockers to disable functions just didn’t seem to work for me, and disabling/removing Opcache seemed like a backwards step.
(And no, restarting PHP-FPM every few minutes really didn’t seem like a good option either).
So what’s the solution?
Simple really… stop Opcache issuing a kill_all_lockers() command!
Looking at the code for the Opcache ZendAccelerator, we can see kill_all_lockers() is invoked on line 922 if (and only if) the opcache.force_restart_timeout is enabled.
So by setting opcache.force_restart_timeout=0 in our PHP ini settings, Opcache will never try to force a restart using kill_all_lockers(), and will never crash.
To do this, simply edit the INI settings of each PHP version using the WHM MultiPHP INI Editor and add an opcache section at the end as follows:
[opcache]
; Disable restarting long-running scripts, as this does not work with cPanels implementation
; Enabling can cause PHP-FPM to crash with 100% CPU due to kill_all_lockers bug
; cPanel Default: 180
opcache.force_restart_timeout = 0 Save the file and check the setting is active by running the following command in WHM > Terminal:
php -ini | grep opcache.force_restart_timeoutYou should see the value 0 is set like so:
[root@host ~]# php -ini | grep opcache.force_restart_timeout
opcache.force_restart_timeout => 0 => 0Alternative Fix (and Bonus Info)
If you don’t see the setting change to 0, then your cPanel server may have the setting hard-coded into one of its other INI files.
These are all stored in the following directory (where XX is the PHP version):
# XX represents the PHP Version
/opt/cpanel/ea-phpXX/root/etc/php.d/So, for PHP 8.2, the directory is:
# The directory for PHP v8.2 is:
/opt/cpanel/ea-php82/root/etc/php.d/
# Here's what the directory contains:
[root@host ~]# ls /opt/cpanel/ea-php82/root/etc/php.d/
10-opcache.ini 20-exif.ini 20-imap.ini 20-posix.ini 20-xmlwriter.ini 30-xmlreader.ini
20-bcmath.ini 20-fileinfo.ini 20-intl.ini 20-simplexml.ini 20-xsl.ini opcache-default.blacklist
20-calendar.ini 20-ftp.ini 20-mbstring.ini 20-sodium.ini 20-zip.ini zzzzzzz-pecl.ini
20-ctype.ini 20-gd.ini 20-mysqlnd.ini 20-sqlite3.ini 30-mysqli.ini
20-curl.ini 20-gettext.ini 20-pdo.ini 20-tokenizer.ini 30-pdo_mysql.ini
20-dom.ini 20-iconv.ini 20-phar.ini 20-xml.ini 30-pdo_sqlite.iniThe INI files in this directory are read in ALPHABETICAL order, after the INI file in WHM MultiPHP INI Editor, so they override whatever you put in the WHM MultiPHP INI Editor INI file.
The Opcache settings are in the file 10-opcache.ini. For PHP 8.2, the file looks like this:
[root@host ~]# cat /opt/cpanel/ea-php82/root/etc/php.d/10-opcache.ini
; Enable Zend OPcache extension module
zend_extension=opcache.so
; Determines if Zend OPCache is enabled
opcache.enable=1
; Determines if Zend OPCache is enabled for the CLI version of PHP
;opcache.enable_cli=0
; The OPcache shared memory storage size.
opcache.memory_consumption=128
; The amount of memory for interned strings in Mbytes.
opcache.interned_strings_buffer=8
; The maximum number of keys (scripts) in the OPcache hash table.
; Only numbers between 200 and 100000 are allowed.
opcache.max_accelerated_files=4000
; The maximum percentage of "wasted" memory until a restart is scheduled.
;opcache.max_wasted_percentage=5
; When this directive is enabled, the OPcache appends the current working
; directory to the script key, thus eliminating possible collisions between
; files with the same name (basename). Disabling the directive improves
; performance, but may break existing applications.
;opcache.use_cwd=1
; When disabled, you must reset the OPcache manually or restart the
; webserver for changes to the filesystem to take effect.
;opcache.validate_timestamps=1
; How often (in seconds) to check file timestamps for changes to the shared
; memory storage allocation. ("1" means validate once per second, but only
; once per request. "0" means always validate)
;opcache.revalidate_freq=2
; Enables or disables file search in include_path optimization
;opcache.revalidate_path=0
; If disabled, all PHPDoc comments are dropped from the code to reduce the
; size of the optimized code.
;opcache.save_comments=1
; If enabled, a fast shutdown sequence is used for the accelerated code
;opcache.fast_shutdown=0
; Allow file existence override (file_exists, etc.) performance feature.
;opcache.enable_file_override=0
; A bitmask, where each bit enables or disables the appropriate OPcache
; passes
;opcache.optimization_level=0xffffffff
;opcache.inherited_hack=1
;opcache.dups_fix=0
; The location of the OPcache blacklist file (wildcards allowed).
; Each OPcache blacklist file is a text file that holds the names of files
; that should not be accelerated.
opcache.blacklist_filename=/opt/cpanel/ea-php82/root/etc/php.d/opcache*.blacklist
; Allows exclusion of large files from being cached. By default all files
; are cached.
;opcache.max_file_size=0
; Check the cache checksum each N requests.
; The default value of "0" means that the checks are disabled.
;opcache.consistency_checks=0
; How long to wait (in seconds) for a scheduled restart to begin if the cache
; is not being accessed.
;opcache.force_restart_timeout=180
; OPcache error_log file name. Empty string assumes "stderr".
;opcache.error_log=
; All OPcache errors go to the Web server log.
; By default, only fatal errors (level 0) or errors (level 1) are logged.
; You can also enable warnings (level 2), info messages (level 3) or
; debug messages (level 4).
;opcache.log_verbosity_level=1
; Preferred Shared Memory back-end. Leave empty and let the system decide.
;opcache.preferred_memory_model=
; Protect the shared memory from unexpected writing during script execution.
; Useful for internal debugging only.
;opcache.protect_memory=0
; Allows calling OPcache API functions only from PHP scripts which path is
; started from specified string. The default "" means no restriction
;opcache.restrict_api=
; Enables and sets the second level cache directory.
; It should improve performance when SHM memory is full, at server restart or
; SHM reset. The default "" disables file based caching.
; RPM note : file cache directory must be owned by process owner
; for mod_php, see /etc/httpd/conf.d/php.conf
; for php-fpm, see /etc/php-fpm.d/*conf
opcache.file_cache=""
; Enables or disables opcode caching in shared memory.
;opcache.file_cache_only=0
; Enables or disables checksum validation when script loaded from file cache.
;opcache.file_cache_consistency_checks=1
; Enables or disables copying of PHP code (text segment) into HUGE PAGES.
; This should improve performance, but requires appropriate OS configuration.
opcache.huge_code_pages=0
; Leads OPcache to check file readability on each access to cached file.
; This directive should be enabled in shared hosting environment, when few
; users (PHP-FPM pools) reuse the common OPcache shared memory.
opcache.validate_permission=1Removing all the blank and commented lines (starting with a ;), the active configuration is:
[root@host php.d]# cat /opt/cpanel/ea-php82/root/etc/php.d/10-opcache.ini | grep -v '^$' | grep -v ';'
zend_extension=opcache.so
opcache.enable=1
opcache.memory_consumption=128
opcache.interned_strings_buffer=8
opcache.max_accelerated_files=4000
opcache.blacklist_filename=/opt/cpanel/ea-php82/root/etc/php.d/opcache*.blacklist
opcache.file_cache=""
opcache.huge_code_pages=0
opcache.validate_permission=1We could just edit this file directly, but it could be overwritten by a cPanel or EasyApache upgrade, so I prefer to add a new file to override these settings.
So we can simply go to the WHM > Terminal and add a file which will be read last, like so:
# Create file with override setting
cat <<EOT > /root/zzzzzz-10-opcache.ini
; Disable restarting long-running scripts, as this does not work with cPanels implementation
; Enabling can cause PHP-FPM to crash with 100% CPU due to kill_all_lockers bug
; cPanel Default: 180
opcache.force_restart_timeout = 0
EOT
# Copy the file into each of the available ea-phpxx directories
/bin/find /opt/cpanel/ -name php.d -type d -exec /bin/cp /root/zzzzzz-10-opcache.ini {}/ \;
# Restart PHP-FPM
/scripts/restartsrv_apache_php_fpmYou can use this file to override any of the other active settings you can’t override via the WHM MultiPHP INI Editor.
For example, if you are running 30-150 websites on your server, you’ll DEFINITELY want to tweak the opcache.memory_consumption, opcache.interned_strings_buffer and opcache.max_accelerated_files settings (see this excellent tutorial for more info).
Thank you Rob – you’ve really done the deep-dive on what turns out to be an oversight with cPanel. Have given this a try and it sure has made a difference with no more 98% CPU on a specific thread. Still evaluating if it has solved the problem entirely.
Glad the info was useful, Lance.