Best file system for working with 1 GB files using nginx, with moderate write, performance reading?

Question

I'm going to create a large file server, and the stack overflow community tips will be needed to discuss file system (linux) choices.

The file server will serve static files 1-2 GB in size (mostly different with each request) through Nginx, with constant moderate write to disks (RAID5 SATA / 7200 arrays). The write-to-read ratio is about 1: 5-10, for every 1 byte per second, 5-10 is read. Reading is more important to me, I can live slower.

Which Linux file system would be the best solution for this task? And why :) Thank you!

Answer 1

I achieved 80 MB / s of random read performance on a "real" disk (spindle). Here are my findings.

So, first determine how much traffic you need to push users away and how much memory you need on the server.

You can skip the disk setup recommendations below because you already have RAID5 setup.

Let's look at an example of a dedicated 1 Gb / s bandwidth server with 3 * 2 TB drives. Save the first OS and tmp disk. For the other 2 disks, you can create a software raid (for me it worked better than on board the hardware raid). In addition, you need to split the files on independent disks. The idea is to use the same load of reading and writing to disk. Software Raid-0 is the best option.

Nginx Conf There are two ways to achieve a high level of performance using nginx.

• use directio

  aio on,
  directio 512; output_buffers 1 8m;

  "This parameter will require that you have a good amount of bar." Requires about 12-16 GB of RAM.

• userland io

  output_buffers 1 2m;

  "make sure you set readahead to 4-6MB to mount the software" blockdev --setra 4096 / dev / md0 (or independent disk mount)

  This option will optimally use the system file cache and require much less RAM. It takes about 8 GB of RAM.

General notes:

• save "sendfile off";

You may also like to use the bandwidth throttle to enable 100s of connections over the available bandwidth. Each download connection will use 4 MB of active bar.

limit_rate_after 2m; limit_rate 100k; 

Both of the above solutions easily scale to 1k + simultaneous user on 3 disk servers. Assuming you have 1 Gb / s bandwidth and each connection is throttled at 1 Mb / s, there is an additional setting needed to optimize writing to the disk, without affecting the reading much.

make all downloads to the os main disk on the mount / tmpuploads scanner. this will not result in intermittent disturbances when heavy readings occur. Then move the file from / tmpuploads using the dd command with the tolag = direct command. sort of

 dd if=/tmpuploads/<myfile> of=/raidmount/uploads/<myfile> oflag=direct bs=8196k 

Answer 2

To provide the best results when serving heavy content, there is something else to tune. Please see the comment, the kernel developers Nginx below:

• Disable sendfile , it does not work well under such loads on Linux due to the inability to control readahead (and, therefore, blocks are read from disk).

  sendfile off;

• Use large output buffers

  output_buffers 1 512k

• Try using aio to provide a better concurrency drive (and pay attention to linux, it also needs directio), i.e. something like this

  aio on; directio 512;

Other recommendations:

• File system sharing check not used

• The file system is ext4, xfs. It’s good to include parameters for connecting to data_writeback and noatime files.

Answer 3

Very large files, as a rule, do not depend on the file system used, modern file systems (i.e. not FAT!) Perform very well their allocation in large adjacent pieces of storage and, thus, minimize the search delay. Where you tend to see the differences between them, this is a small file performance, fragmentation resistance in situations outside the space, concurrency, etc. Storing large files is a relatively simple problem, and I doubt that you will see measurable differences.

But as always: if you really care, compare. There are no simple answers about file system performance.