The LEBRE scoring methodology was designed mainly as a quick way to evaluate the value of VPS offers based on paper specifications. LEBRE-Xtended (LEBRE-X) seeks to provide a real-world performance testing methodology based on three principles: validity, reliability, and parsimony.
LEBRE-X tests the performance of the CPU, RAM, disk and network speeds of a lowend box using the following tools:
- Geekbench 5 (CPU and RAM)
- fio (disk storage performance)
- ioping (disk latency)
- iperf3 (port throughput)
Geekbench 5 offers the most comprehensive measure of CPU and memory performance and aggregates the value of the tests into a single score for ease of comparison. Geekbench 5 does stress the CPU and RAM tremendously, so it provides a good representation of how the well the VPS performs under heavy load.
Disk performance on a VPS is better measured by fio instead of dd, which is found in many benchmark scripts. With fio, the results are less likely to be affected by OS caching, especially on SSD and NVMe disks. We currently do not have a method to remove the effect of hardware RAID controller caching but we will revise our testing methodology if a solution to address this issue is developed.
Also, dd primarily measures sequential writes, which is not a common scenario for VPS use. LEBRE-X runs random read-write tests (50% read and 50% write) using fio to measure IOPS and disk bandwidth, which is a more accurate representation of real-world disk performance. In addition, LEBRE-X runs tests with 4k, 64k and 256k block sizes. Increasing the block size stresses the disk and the rate of decline in IOPS provides an indication of how capable the disk is in handling heavy workloads.
Disk latency is measured using ioping. Thirty requests are made to derive the average response time to the requests.
To measure port throughput, we use iperf3 with 10 parallel connections. The goal of the test is to determine if the port speed promised by the provider performs as it should. Generally, we can expect port speeds to perform close to maximum to other servers in a similar geographic locale. Our iperf3 tests are performed on private servers with 1Gbps port speed and we cannot measure beyond that, but for most use cases sustained throughput at 1Gbps should be sufficient.
Also, most benchmarking scripts (except YABS) measure inbound traffic speeds by downloading test files onto the VPS, but with private iperf3 setups, we measure outbound traffic speeds by sending data out of the VPS. For many common use VPS cases such web servers, plex streaming etc, outbound traffic speeds are relatively more important.
LEBRE-X tests are run daily at random times for at least three to four weeks. There are two reasons for this:
- We want to eliminate patterns based on time (e.g. certain times of the day could be busier for the host node than other timeings)
- We want to capture longitudinal data because we are also interested in whether the VPS performs consistently well across time, which also suggests how well the VPS provider manages the host node.
While no amount of testing can reliably give a definite answer to a VPS performance, we believe that LEBRE-X is set up to give a real-world performance evaluation that is probably more correct than wrong most of the time.