Need help debugging terrible performance

[pretzelhammer]

Hi, I recently implemented the same RESTful API server once using r2d2 + Diesel + Rocket and another time using sqlx + actix-web. I benchmarked both servers using vegeta, a CLI HTTP load testing tool, and here are the results I got:

• The r2d2 + Diesel + Rocket (DR) server processed 198 requests per second over 1 minute.
• The sqlx + actix-web (SA) server processed 2845 requests per second over 1 minute.

The SA server's performance is pretty good but the DR server's performance is absolutely terrible. I think the issue with the DR server has something to do with how r2d2 and Diesel are configured/used but I can't figure out exactly what the problem is.

Complete source code for all DB operations in the DR server: https://github.com/pretzelhammer/kanban/blob/main/diesel-rocket/src/db.rs

If someone could take a look at the code above and tell me what I'm doing wrong I'd really appreciate it. At least I hope I'm doing something wrong, I would be shocked if r2d2's and Diesel's performance was just inherently this bad.

For comparison, if it's at all helpful, here's the complete source code for all DB operations in the SA server: https://github.com/pretzelhammer/kanban/blob/main/sqlx-actix-web/src/db.rs

[weiznich]

I fear it's not really possible to help you here, at least not with the information you've provided here. This is missing at least the following information:

• How to setup the benchmark?
• How did you exactly measure your numbers?
• Which tools did you use for that?
• Did you use a release build?
• Which endpoint was used by your benchmark?
• Which backend did you use?
• How did you configure your database?

Leaving these points beside: Are you even sure that this is caused by diesel and not by an other difference in your implementation? At least our benchmarks indicate that diesel on itself is in all situations at least as performant as sqlx, if not much more performanter. Therefore as general advice for benchmarks: If you try to measure the performance difference between only two crates that can serve the same propose, do only change that one piece in your code and don't couple it with other changes (like using rocket instead of actix_web here). If possible do not include any other code at all (so only measure the raw performance of sqlx vs diesel instead of including a full web framework), as this can greatly change the results for other reasons without that being a result of the actual thing you are trying to measure.

[pretzelhammer]

All those questions are answered in the readme.md of the benchmarks directory of my project: https://github.com/pretzelhammer/kanban/tree/main/benchmarks

How to setup the benchmark?

Have Docker, Just, Python3 (+ psutil), Vegeta, and Bash installed on a machine and then run bash run_bench.sh <project directory> <duration> <workers> <targets> from the benchmarks directory.

How did you exactly measure your numbers?

Vegeta measures request throughput & latencies. I measure CPU usage (in CPU seconds) and memory usage (in megabytes) using psutil. I mostly care about the request throughput & latencies though.

The specific vegeta command I run is

cat $targets_file | vegeta attack -duration=${duration}s -rate=0 -max-workers=${workers}

Where $targets_file is either reads.txt or mixed.txt.

Here's an example vegeta report. Here's an example CSV generated from my psutil script.

Which tools did you use for that?

Vegeta and psutil. How the process is monitored is done entirely by this 70-line Python3 script. And for Vegeta I just run the single command I showed above.

Did you use a release build?

Yes. I compile with:

RUSTFLAGS="-C target-cpu=native" cargo build --release

With profile:

[profile.release]
debug = 0
lto = true
codegen-units = 1
panic = "abort"

And run the binary from the targets/release directory.

Which endpoint was used by your benchmark?

I have 2 benchmarks, a read-only workload and mixed (reads + writes) workload. The read-only workload hits these targets:

GET http://localhost:8000/api/boards
Authorization: Bearer LET_ME_IN

GET http://localhost:8000/api/boards/1/summary
Authorization: Bearer LET_ME_IN

GET http://localhost:8000/api/boards/1/cards
Authorization: Bearer LET_ME_IN

The mixed workload hits these targets:

GET http://localhost:8000/api/boards
Authorization: Bearer LET_ME_IN

GET http://localhost:8000/api/boards/1/summary
Authorization: Bearer LET_ME_IN

POST http://localhost:8000/api/boards
Authorization: Bearer LET_ME_IN
Content-Type: application/json
@post-board.json

DELETE http://localhost:8000/api/boards/1000000
Authorization: Bearer LET_ME_IN

GET http://localhost:8000/api/boards/1/cards
Authorization: Bearer LET_ME_IN

POST http://localhost:8000/api/cards
Authorization: Bearer LET_ME_IN
Content-Type: application/json
@post-card.json

PATCH http://localhost:8000/api/cards/1
Authorization: Bearer LET_ME_IN
Content-Type: application/json
@patch-card.json

DELETE http://localhost:8000/api/cards/1000000
Authorization: Bearer LET_ME_IN

The best diesel+rocket results for getting hit with the read-only workload for 60 secs with up to 40 workers is this:

Requests      [total, rate, throughput]         13971, 231.22, 198.12
Duration      [total, attack, wait]             1m10s, 1m0s, 10.034s
Latencies     [min, mean, 50, 90, 95, 99, max]  6.178ms, 173.741ms, 27.678ms, 318.932ms, 326.678ms, 1.326s, 15.038s
Bytes In      [total, mean]                     2661222, 190.48
Bytes Out     [total, mean]                     0, 0.00
Success       [ratio]                           99.91%
Status Codes  [code:count]                      0:12  200:13959  

The exact same workload and test on the sqlx+actix-web server produced these results:

Requests      [total, rate, throughput]         170714, 2845.20, 2844.81
Duration      [total, attack, wait]             1m0s, 1m0s, 8.335ms
Latencies     [min, mean, 50, 90, 95, 99, max]  6.187ms, 14.026ms, 12.803ms, 17.904ms, 20.36ms, 27.41ms, 1.029s
Bytes In      [total, mean]                     32549337, 190.67
Bytes Out     [total, mean]                     0, 0.00
Success       [ratio]                           100.00%
Status Codes  [code:count]                      200:170714 

The best diesel+rocket results for getting hit with the mixed workload for 60 secs and up to 40 workers is this:

Requests      [total, rate, throughput]         15455, 253.26, 188.32
Duration      [total, attack, wait]             1m6s, 1m1s, 4.981s
Latencies     [min, mean, 50, 90, 95, 99, max]  35.528µs, 164.5ms, 26.017ms, 313.756ms, 323.253ms, 1.428s, 14.772s
Bytes In      [total, mean]                     118362905, 7658.55
Bytes Out     [total, mean]                     205473, 13.29
Success       [ratio]                           80.43%
Status Codes  [code:count]                      0:3025  200:12430 

However, the exact same workload and test on the sqlx+actix-web server produced these results:

Requests      [total, rate, throughput]         71522, 1191.92, 1189.90
Duration      [total, attack, wait]             1m0s, 1m0s, 101.888ms
Latencies     [min, mean, 50, 90, 95, 99, max]  6.706ms, 33.559ms, 21.452ms, 74.728ms, 96.839ms, 138.173ms, 1.169s
Bytes In      [total, mean]                     3515798166, 49156.88
Bytes Out     [total, mean]                     1385700, 19.37
Success       [ratio]                           100.00%
Status Codes  [code:count]                      200:71522 

Which backend did you use?

I ran these tests locally on my macOS machine, which has:

• CPU: 2.3 GHz 8-core Intel Core i9
• RAM: 32 GB 2667 MHz DDR4

And I ran the database in a docker container which has access to the entire CPU and 2 GB of memory.

How did you configure your database?

The database I used is PostgresQL v13. I create it with this command:

docker run --rm --name kanban-postgres --tmpfs=/pgtmpfs --env PGDATA=/pgtmpfs --env POSTGRES_HOST_AUTH_METHOD=trust --publish 5432:5432 --detach postgres:13

Which the exact command I use for every benchmark test for every server. All the PostgresQL configuration options are at their defaults (e.g. max_connections = 100).

Are you even sure that this is caused by diesel and not by an other difference in your implementation?

I'm not sure, I'm planning on making a similar issue to this one in the Rocket github repo later though.

Therefore as general advice for benchmarks: If you try to measure the performance difference between only two crates that can serve the same propose, do only change that one piece in your code and don't couple it with other changes (like using rocket instead of actix_web here).

Yes, I know I'm not following best practices here, and honestly I wasn't expecting the results to be super accurate or comparable in the first place, but I was still expecting them to be within the same ballpark of each other. The problem isn't that the Diesel + Rocket implementation is a little bit slower, the problem is that it is unbelievably unacceptably slow. It's so painfully slow that I'm hoping it's somehow my fault and I did something obviously wrong in the implementation or configuration that someone can point out for me, because it's literally hard to believe the numbers when they're this terrible.

[weiznich]

As already linked: Our benchmarks without any web framework on top indicate that diesel is at least as performant as sqlx. Therefore I suspect the issue more in the rocket part of the code. I do not use rocket by myself, but I would first try to adjust the number of worker threads via their config file to something larger.

[pretzelhammer]

I made an issue on the Rocket github repo and Sergio, like you, suggested I increase the worker threads, so I increased them from 32 to 256 and that doubled the throughput of the Diesel + Rocket server, which is nice, except that it's still significantly less than the throughput of the sqlx + actix-web server.

Is it at all possible that the issue is actually with the r2d2 connection pool, or no?

[weiznich]

I don't believe that this is an issue with the connetcion pool or any other concrete part of your application. As Sergio has already written: You are measuring two completely different implementations. One that is designed to handle a large number of connections (actix_web), and one that isn't (rocket 0.4). It is not unexpected that they behave that different under load.
To add a bit of perspective as a maintainer of diesel: I believe that this is mostly an question of how the web framework is designed, especially if they are using async or if they are using a fixed number of threads. As far as I can tell this is not really related to how you connect to database, but I believe that the main bottleneck lies in the implementation of the web frameworks. The reason for this is that database connections are a really scarce resource as soon as you scale out to more than a few hundred requests per second. Each and every request will wait that a database connection will be available then. This waiting is handled by the web framework and by a minor contribution by the database pool implementation. In these scenarios it matters to use an async pool implementation in combination with an async web framework.
In the end it just boils down to the point that you are trying to use a two different application designs for a very specific use case that is tailored to one of the designs, but is the antithesis of the other one.

If you still believe that's caused by something in the database layer, I invite you to provide a reproducing example that does not depend on two different web framework.

[pretzelhammer]

Okay, well you've convinced me it's probably the web framework, but even then, given my benchmark parameters, I don't understand why the performance difference is so drastic. I left my final comment in the Rocket issue thread here try to explain my confusion. Thanks for all your time.