Frequency Oscillation HHVM fastcgi + Nginx

Currently, we have started using HHVM in the production environment, and so far almost all the results are quite impressive. Our overall transaction rate has been significantly improved over PHP-FPM with APC. Almost all requests are less than 500 ms, but each pair of requests (from 5 to 10 or so) results in a request time of 2 to 5 seconds.

The requested page does not seem to make any difference, and repeating the same page again causes this behavior for several requests.

We run HHVM in server mode with the following command line options:

/usr/bin/hhvm --mode server -vServer.Type=fastcgi -vServer.FileSocket=/usr/local/php55/sockets/admin.sock -vPidFile=/var/run/hhvm/admin.pid -vEval.Jit=true -vServer.ThreadCount=24 -vServer.APC.EnableApc=true 

We run nginx for the web server with these corresponding configurations (sorry if I forgot something importand here).

 fastcgi_buffer_size 128k; fastcgi_buffers 256 16k; fastcgi_busy_buffers_size 256k; fastcgi_temp_file_write_size 256k; fastcgi_read_timeout 240; fastcgi_intercept_errors on; 

The server has 128 GB of memory and 24 cores (hyperthreading is actually 12).

We shared a bit of searching for https://github.com/facebook/hhvm/wiki/Runtime-options , however most of the options are not very well explained, so I have no idea what they are doing and testing them in a production environment is a bit scary.

If anyone here had a similar problem or could point me in the direction with some HHVM options, I would be very grateful.

Used version of HHVM - http://www.hop5.in/yum/el6/

 HipHop VM 3.0.1 (rel) Compiler: Repo schema: e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 

And the file /etc/hhvm/config.hdf

 Log { Level = Warning AlwaysLogUnhandledExceptions = true RuntimeErrorReportingLevel = 8191 } MySQL { TypedResults = false } 

We use supervisord to start HHVM, so here is the dispatcher configuration as well:

 [program:hhvm] stopasgroup=true killasgroup=true command=/usr/bin/hhvm --mode server -vServer.Type=fastcgi -vServer.FileSocket=/usr/local/php55/sockets/admin.sock -vPidFile=/var/run/hhvm/admin.pid -vEval.Jit=true -vServer.ThreadCount=24 -vServer.APC.EnableApc=true user=admin stdout_logfile=/var/log/hhvm/admin.log stderr_logfile=/var/log/hhvm/admin.error.log directory=/home/admin umask=000 

There is php.ini in the /etc/hhvm/php.ini file, but the content is empty. Also, the pages tried everything to make some database connections, but this is usually very minimal. For the full picture, also my.cnf. Used mysql version - percona

 [mysql] # CLIENT # port = 3306 socket = /var/lib/mysql/mysql.sock [mysqld] # GENERAL # user = mysql default-storage-engine = InnoDB socket = /var/lib/mysql/mysql.sock pid-file = /var/lib/mysql/mysql.pid [mysqld] # MyISAM # key-buffer-size = 32M myisam-recover = FORCE,BACKUP # SAFETY # max-allowed-packet = 128M max-connect-errors = 1000000 # DATA STORAGE # datadir = /var/lib/mysql/ # BINARY LOGGING # log-bin = /var/lib/mysql/mysql-bin expire-logs-days = 14 sync-binlog = 1 # CACHES AND LIMITS # tmp-table-size = 128M max-heap-table-size = 256M query-cache-size = 10G max-connections = 1000 thread-cache-size = 100 open-files-limit = 65535 table-definition-cache = 4096 table-open-cache = 4000 join-buffer-size = 1M # INNODB # innodb-flush-method = O_DIRECT innodb-log-files-in-group = 2 innodb-log-file-size = 512M innodb-flush-log-at-trx-commit = 1 innodb-file-per-table = 1 innodb-buffer-pool-size = 73G # LOGGING # log-error = /var/lib/mysql/mysql-error.log log-queries-not-using-indexes = 1 slow-query-log = 1 slow-query-log-file = /var/lib/mysql/mysql-slow.log 

Mysql Version:

 innodb_version 5.6.17-65.0 protocol_version 10 slave_type_conversions version 5.6.17-65.0-56-log version_comment Percona Server (GPL), Release 65.0, Revision 587 version_compile_machine x86_64 version_compile_os Linux 
+6
source share
1 answer

So, we managed to figure it out, it was a joint problem of HHVM and the application we use.

HHVM produces quite a lot of punched card files, and where there are also a bunch of sess_ files. In total, there are more than 8 million files in the / tmp directory. After fixing these issues, our problems disappeared, and almost all of our requests improved significantly in performance. In addition, the session files should not have been there in the first place, but it was a problem that was already fixed.

Now we have disabled the generation of perf files by adding -vEval.PerfPidMap to our launch arguments.

+2
source

Source: https://habr.com/ru/post/972254/


All Articles