On November 18 a new major update of our mass storage system CephFS v20, Tentacle, was released. Only two days later on November 20 SubMIT upgraded to this version. Apart from various bug fixes and new features, Tentacles brings significant performance improvements.

SubMIT is using Erasure Coding in a 6+2 scheme. This allows us to have a 3-fold fault tolerance, i.e. 2 random drives or 2 of our 10 CephFS machines can fail at once without loss of data. This comes with a much reduced cost of (6+2)/6 = 133% storage compared to 300% storage needed for a complete 3 fold replication. From our total raw capacity of 1.6PB, more than 1.2PB are available for our users.

Every user has by default 1TB ceph space under ceph/submit/data/user/<first letter of username>/<username> with possibility for extension. More storage for groups is opened upon request and can be found under /ceph/submit/data/group/.

The Erasure Coding comes with better storage efficiency at the cost of additional computing, and some performance losses. With the recent upgrade, performance losses are mitigated and storage efficiency is further increased by partial read and writes, parity delta writes, and improved padding of small objects.

In the upcoming weeks and months we are planning to perform I/O benchmarks to verify the speed gain. What is your experience with the mass storage so far? Let us know!