Downloading ChRIS files from swift 10x efficiently by bypassing CUBE

[jennydaman]

chrs is the best way to download files from a production ChRIS deployment. However, in situations where we can access swift directly, we can sidestep CUBE to improve download speeds.

On our internal deployment cube-next, downloading 80,000 files (70 GB) is 10x faster using rclone connecting directly to Swift than chrs which downloads from CUBE.

Experiment Setup

We want to compare the performance of downloading from CUBE v.s. Swift directly. We're using chrs v0.2.3 as the CUBE client and rclone v1.62.2 as the Swift client. chrs is hard-coded to do 4 concurrent downloads so the --transfers=4 option is passed to rclone.

First I ran miniChRIS-docker and used chrisomatic to add dbg-bigfiles. Then I configured rclone and chrs for connection:

chrs login --address http://localhost:8000/api/v1/ --username chris --password chris1234

# rclone.conf
[minichris-swift]
type = swift
auth_version = 1
key = testing
user = chris:chris1234
auth = http://localhost:8080/auth/v1.0

I created some data in ChRIS by running dbg-bigfiles

chrs run-latest dbg-bigfiles -- --total 5GiB --size 100MiB
chrs run-latest dbg-bigfiles -- --total 5GiB --size 10MiB

The Benchmark

hyperfine -p 'rm -rf /tmp/outgoing' \
    'chrs download chris/feed_1 /tmp/outgoing' \
    'rclone copy --transfers=4 minichris-swift:users/chris/feed_1 /tmp/outgoing'

Results

File Size	Total Size	Client	Time	User	System
100MiB	5GiB	chrs	9.150	6.010	8.839
100MiB	5GiB	rclone	8.198	12.768	2.354
10MiB	5GiB	chrs	12.752	5.567	8.579
10MiB	5GiB	rclone	9.401	12.758	2.627

Conclusion

Connecting to swift directly using rclone is more efficient than downloading files from CUBE using chrs.
This is attributable to the inefficiency of CUBE (here's an experiment assessing the overhead of chrs as a client).
Inefficiency problems are exacerbated in situations where there are numerous small files.

Side Observation: CPU Time

chrs spends more system time than user time, which is different from rclone. This has to do with how chrs is written in async Rust using tokio.

Limitations

In this experiment, client and server were the same computer. In reality, the client and server are usually different computers. Also, results depend a lot on the server's specs and load.