Opcache with file_cache and JIT enabled crashes an application

[voodooism]

Description

On our production server running a Symfony application with PHP-FPM and Nginx, I've noticed an occasional strange behavior: after an HTTP request, we sometimes receive an empty response with a 200 HTTP status code. In the PHP-FPM logs, I see the following message:

Error: Shared memory lock not obtained

After conducting some tests, I realized that this error occurs only when the file_cache and JIT options are enabled. Additionally, I noticed that the max_accelerated_files option also plays a role in this behavior. Our application is quite large, so this setting is configured to 20,000 on our production server. However, if I turn off the file_cache, everything works fine, even with the max_accelerated_files set to its minimum value (200 instead of 20000). The same happens with JIT turned off - everything works well, with JIT disabled.

To reproduce this behavior locally, I created a simple project: I downloaded the same version of Symfony that we have in production (6.4), built the application using PHP and Nginx images from Docker Hub, and decreased the max_accelerated_files to its minimum value (200, according to the documentation).

Steps to reproduce:
First, let's make sure that everything works fine:

git clone [email protected]:voodooism/opcache-testing.git
cd ./opcache-testing
composer install
docker-compose up -d --build

Expected response:

curl http://localhost:8080/health
{"status":"ok"}

Logs from php-fpm container:

[19-Aug-2024 12:49:25] NOTICE: [pool www] 'user' directive is ignored when FPM is not running as root
[19-Aug-2024 12:49:25] NOTICE: [pool www] 'user' directive is ignored when FPM is not running as root
[19-Aug-2024 12:49:25] NOTICE: [pool www] 'group' directive is ignored when FPM is not running as root
[19-Aug-2024 12:49:25] NOTICE: [pool www] 'group' directive is ignored when FPM is not running as root
[19-Aug-2024 12:49:25] NOTICE: fpm is running, pid 1
[19-Aug-2024 12:49:25] NOTICE: ready to handle connections
172.18.0.4 -  19/Aug/2024:12:49:28 +0000 "GET /index.php" 200

Then, let's change the max_accelerated_files

echo "opcache.max_accelerated_files = 200" >> ./docker/php/conf.d/opcache.ini
docker-compose up -d --build

Bad behavior (no response body at all):

curl http://localhost:8080/health

Logs from php-fpm container:

[19-Aug-2024 12:50:38] NOTICE: [pool www] 'user' directive is ignored when FPM is not running as root
[19-Aug-2024 12:50:38] NOTICE: [pool www] 'user' directive is ignored when FPM is not running as root
[19-Aug-2024 12:50:38] NOTICE: [pool www] 'group' directive is ignored when FPM is not running as root
[19-Aug-2024 12:50:38] NOTICE: [pool www] 'group' directive is ignored when FPM is not running as root
[19-Aug-2024 12:50:38] NOTICE: fpm is running, pid 1
[19-Aug-2024 12:50:38] NOTICE: ready to handle connections
Mon Aug 19 12:50:40 2024 (7): Error Shared memory lock not obtained
172.18.0.4 -  19/Aug/2024:12:50:40 +0000 "GET /index.php" 200

Then let's override the existent file_cache option and turn it off:

echo "opcache.file_cache = null" >> ./docker/php/conf.d/opcache.ini

Everything works as expected then:

curl http://localhost:8080/health
{"status":"ok"}

Logs:

[19-Aug-2024 13:45:50] NOTICE: [pool www] 'user' directive is ignored when FPM is not running as root
[19-Aug-2024 13:45:50] NOTICE: [pool www] 'user' directive is ignored when FPM is not running as root
[19-Aug-2024 13:45:50] NOTICE: [pool www] 'group' directive is ignored when FPM is not running as root
[19-Aug-2024 13:45:50] NOTICE: [pool www] 'group' directive is ignored when FPM is not running as root
[19-Aug-2024 13:45:50] NOTICE: fpm is running, pid 1
[19-Aug-2024 13:45:50] NOTICE: ready to handle connections
172.18.0.4 -  19/Aug/2024:13:46:26 +0000 "GET /index.php" 200

More information for context

PHP version

PHP 8.2.22 (cli) (built: Aug 13 2024 02:05:25) (NTS)
Copyright (c) The PHP Group
Zend Engine v4.2.22, Copyright (c) Zend Technologies
    with Zend OPcache v8.2.22, Copyright (c), by Zend Technologies

php -m

[PHP Modules]
Core
ctype
curl
date
dom
fileinfo
filter
hash
iconv
json
libxml
mbstring
mysqlnd
openssl
pcre
PDO
pdo_pgsql
pdo_sqlite
Phar
posix
random
readline
Reflection
session
SimpleXML
sodium
SPL
sqlite3
standard
tokenizer
xml
xmlreader
xmlwriter
Zend OPcache
zlib

[Zend Modules]
Zend OPcache

php -i|grep opcache.

/usr/local/etc/php/conf.d/docker-php-ext-opcache.ini,
/usr/local/etc/php/conf.d/opcache.ini
opcache.blacklist_filename => no value => no value
opcache.consistency_checks => 0 => 0
opcache.dups_fix => Off => Off
opcache.enable => On => On
opcache.enable_cli => Off => Off
opcache.enable_file_override => Off => Off
opcache.error_log => no value => no value
opcache.file_cache => /var/www/html/var => /var/www/html/var
opcache.file_cache_consistency_checks => On => On
opcache.file_cache_only => Off => Off
opcache.file_update_protection => 2 => 2
opcache.force_restart_timeout => 180 => 180
opcache.huge_code_pages => Off => Off
opcache.interned_strings_buffer => 16 => 16
opcache.jit => tracing => tracing
opcache.jit_bisect_limit => 0 => 0
opcache.jit_blacklist_root_trace => 16 => 16
opcache.jit_blacklist_side_trace => 8 => 8
opcache.jit_buffer_size => 128M => 128M
opcache.jit_debug => 0 => 0
opcache.jit_hot_func => 127 => 127
opcache.jit_hot_loop => 64 => 64
opcache.jit_hot_return => 8 => 8
opcache.jit_hot_side_exit => 8 => 8
opcache.jit_max_exit_counters => 8192 => 8192
opcache.jit_max_loop_unrolls => 8 => 8
opcache.jit_max_polymorphic_calls => 2 => 2
opcache.jit_max_recursive_calls => 2 => 2
opcache.jit_max_recursive_returns => 2 => 2
opcache.jit_max_root_traces => 1024 => 1024
opcache.jit_max_side_traces => 128 => 128
opcache.jit_prof_threshold => 0.005 => 0.005
opcache.lockfile_path => /var/www/html/var => /var/www/html/var
opcache.log_verbosity_level => 1 => 1
opcache.max_accelerated_files => 200 => 200
opcache.max_file_size => 0 => 0
opcache.max_wasted_percentage => 5 => 5
opcache.memory_consumption => 2048 => 2048
opcache.opt_debug_level => 0 => 0
opcache.optimization_level => 0x7FFEBFFF => 0x7FFEBFFF
opcache.preferred_memory_model => no value => no value
opcache.preload => no value => no value
opcache.preload_user => no value => no value
opcache.protect_memory => Off => Off
opcache.record_warnings => Off => Off
opcache.restrict_api => no value => no value
opcache.revalidate_freq => 2 => 2
opcache.revalidate_path => Off => Off
opcache.save_comments => On => On
opcache.use_cwd => On => On
opcache.validate_permission => Off => Off
opcache.validate_root => Off => Off
opcache.validate_timestamps => Off => Off

PHP Version

PHP 8.2.22

Operating System

Ubuntu 22.04.4

[iluuu1994]

/cc @dstogov

[dstogov]

This error may be caught only in PHP-8.2.
PHP-8.3 debug build should lead to assertion.
PHP-8.3 release build misses the error condition because commit 10d43c4 has removed this error message and replaced it with ZEND_ASSERT(), but ZEND_ASSERT() is ignored in release builds. This means PHP-8.3 might do something completely wrong and should be fixed.

[dstogov]

The bug should be fixed now.
I was able to reproduce it without your application, by setting opcache.max_accelerated_files=200 and opcache.file_cache and running Symfony demo app.

$ php-cgi /.../symfony_demo/public/index.php

Please check if the the fix works for you.

It seems we have some different problem with PCRE cache...