{"id":133167,"date":"2023-04-17T08:00:00","date_gmt":"2023-04-17T08:00:00","guid":{"rendered":"https:\/\/fedoramagazine.org\/?p=38109"},"modified":"2023-04-17T08:00:00","modified_gmt":"2023-04-17T08:00:00","slug":"linux-bcache-with-writeback-cache-how-it-works-and-doesnt-work","status":"publish","type":"post","link":"https:\/\/sickgaming.net\/blog\/2023\/04\/17\/linux-bcache-with-writeback-cache-how-it-works-and-doesnt-work\/","title":{"rendered":"Linux bcache with writeback cache (how it works and doesn\u2019t work)"},"content":{"rendered":"
bcache is a simple and good way to have large disks (typically rotary and slow) exhibit performance quite similar to an SSD disk, using a small SSD disk or a small part of an SDD.<\/p>\n
In general, bcache is a system for having devices composed of slow and large disks, with fast and small disks attached as a cache.<\/p>\n
This article will discuss performance and some optimization tips as well as configuration of bcache.<\/p>\n
<\/span> <\/p>\n
The following terms are used in bcache to describe how it works and the parts of bcache:<\/p>\n
\n
\n\n
\n
backing device<\/em><\/td>\n
slow and large disk (disk intended to actually hold the data)<\/td>\n<\/tr>\n
\n
cache device<\/em><\/td>\n
fast and small disk (cache)<\/td>\n<\/tr>\n
\n
dirty cache<\/em><\/td>\n
data present only in the cache device<\/td>\n<\/tr>\n
\n
writeback<\/em><\/td>\n
writing to the cache device and later (much later) to the backing device<\/td>\n<\/tr>\n
\n
writeback rate<\/em><\/td>\n
cache write speed in the backing device<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/figure>\n
A disk data cache has always existed, it is the free RAM in the operating system. When data is read from the disk it is copied to RAM. If the data is already in RAM, it is read from RAM rather than being read from disk again. When data is written to the disk, it is written to RAM and after a few moments written to the disk as well. The time data spends only in RAM is very short since RAM is volatile.<\/p>\n
bcache is similar, only it has various modes of cache operation. The mode that is faster in writing data is writeback. It works the same as for RAM, only instead of RAM there is a SATA or NVME SSD device. The data may reside only in the cache for much longer, even forever, so it is a bit riskier (if you break the SSD, the data that resided only in the cache is lost, with a good chance that the whole filesystem becomes inaccessible).<\/p>\n
Performance Comparison<\/h2>\n
It is very difficult to gather reliable data from any tests, either with real cases or with special programs. They always give extremely variable, different, unstable values. The various caches present and the type of filesystem (btrfs, journaled, etc.), make the values very variable. It is advisable to ignore small differences (say 5-10%).<\/p>\n
The following performance data refers to the test below (random and multiple reads\/writes), trying to always maintain the same conditions and repeating three times in immediate sequence.<\/p>\n
The tables below show the performance of the separate devices:<\/p>\n
Performance of the backing device (RAID 1 with 1TB rotary disks)<\/strong><\/p>\n
\n
\n\n
\n
Throughput:<\/td>\n
<\/td>\n
<\/td>\n<\/tr>\n
\n
read, MiB\/s: 0.22<\/td>\n
read, MiB\/s: 0.23<\/td>\n
read, MiB\/s: 0.19<\/td>\n<\/tr>\n
\n
written, MiB\/s: 0.15<\/td>\n
written, MiB\/s: 0.16<\/td>\n
written, MiB\/s 0.13<\/td>\n<\/tr>\n
\n
Latency (ms):<\/td>\n
<\/td>\n
<\/td>\n<\/tr>\n
\n
max: 174.92<\/td>\n
max: 879.59 <\/td>\n
max: 1335.30<\/td>\n<\/tr>\n
\n
95th percentile: 87.56<\/td>\n
95th percentile: 87.56 <\/td>\n
95th percentile: 89.16<\/td>\n<\/tr>\n<\/tbody>\n<\/table>RAID 1 with 1TB rotary disks<\/figcaption><\/figure>\n
Performance of the cache device (SSD SATA 100GB)<\/strong><\/p>\n
\n
\n\n
\n
Throughput:<\/td>\n
<\/td>\n
<\/td>\n<\/tr>\n
\n
read, MiB\/s: 7.28 <\/td>\n
read, MiB\/s: 7.21<\/td>\n
read, MiB\/s: 7.51<\/td>\n<\/tr>\n
\n
written, MiB\/s: 4.86\u2003<\/td>\n
written, MiB\/s: 4.81<\/td>\n
written, MiB\/s 5.01<\/td>\n<\/tr>\n
\n
Latency (ms):<\/td>\n
<\/td>\n
<\/td>\n<\/tr>\n
\n
max: 126.55<\/td>\n
max: 102.39<\/td>\n
max: 107.95<\/td>\n<\/tr>\n
\n
95th percentile: 1.47<\/td>\n
95th percentile: 1.47 <\/td>\n
95th percentile: 1.47<\/td>\n<\/tr>\n<\/tbody>\n<\/table>Cache device (SSD SATA 100GB)<\/figcaption><\/figure>\n
The theoretical expectation that a bcache device will be as fast as the cache device is (physically) impossible to achieve. On average, bcache is significantly slower and only sometimes approaches the same performance as the cache device. Improved performance almost always requires various compromises.<\/p>\n
Consider an example assuming there is a 1TB bcache device and a 100GB cache. When writing a 1TB file, the cache device is filled, then partially emptied to the backing device, and refilled again, until the file is fully written. <\/p>\n
Because of this (and also because part of the cache also serves data when reading) there is a limit on the length of the file’s sequential data that are written to the cache. Once the limit is exceeded, the file data is written (or read) directly to the backing device, bypassing the cache.<\/p>\n
bcache also limits the response delay of the disks, but disproportionately so, especially for SSD SATA, degrading the performance of the cache.<\/p>\n
The dirty cache should be emptied to decrease the risk of data loss and to have cache available when it is needed. This should only be done when the devices exhibit little or no activity, otherwise the performance available for normal use collapses.<\/p>\n
Unfortunately, the default settings are too low, and the writeback rate adjustment is crude. To improve the writeback rate adjustment it is necessary to write a program (I wrote a script for this).<\/p>\n
The following commands provide the necessary optimizations (required at each startup) to get better performance from the bcache device.<\/p>\n