File transfer performance issue when running on z/OS

[drimmeer]

Help please again!

The performance of using this sftp package (version 1.13.7) is very strange:

It is about 10 times faster than Java when transferring the same file, to the local host (that is , no network was involved.)

Then when network was involved:

1. When testing on a z/OS emulator running in Linux, it has the same performance of the Java counter part;
2. But then when running on a real z/OS, it is 10 times slower than Java (on a z/OS that has 1 CPU), and 13 times slower than Java (on a z/OS that has 5 CPUs)!!!

Who can help me to figure out the reason and solution? Thank you in advance!

[puellanivis]

Are you using WithConcurrentReads(true) and WithConcurrentWrite(true)? These can sometimes be causing poor performance with even the slightest of latency. Many of the tun/tap devices used by emulators are essentially localhost equivalents with unnatural performance characteristics between any two guests and any guest and the host.

The dev-v2 branch (renamed from v2 since this seems to be a special identifier for Go) also improves performance a lot, sometimes coming up on the same performance as the portable openssh version. I have tested the performance over an actual 1 Gbit/s link, so it should have reasonable performance by default, and with less fuss.

[drimmeer]

Thank you Cassondra for the quick response.
Yes, I think I use WithConcurrentReads(true) and WithConcurrentWrite(true), here is the snippet of my code:

	concurrentWriteOption := sftp.UseConcurrentWrites(true)
	concurrentReadOption := sftp.UseConcurrentReads(true)
	maxConcurrentOption := sftp.MaxConcurrentRequestsPerFile(100) // 100 was found to be the best during some testing
	sftpClient, err := sftp.NewClient(sshClient, concurrentReadOption, concurrentWriteOption, maxConcurrentOption)

I'll try the dev-v2 branch then. I assume that it also support z/OS.

[puellanivis]

I’m unsure if it supports z/OS properly, as I don’t have a test platform for it. But I did try to incorporate the changes that were made for GOOS=zos changes in the main branch. If it doesn’t work for some reason, there’s an issue for the branch, where you can drop bug reports.

[drimmeer]

Hi Cassondra,
I tried the new version in branch github.com/pkg/sftp/v2@dev-v2, but could not figure out how to use the new flags WithConcurrentReads(true) and WithConcurrentWrite(true). I could not find any samples as well.
Could you please advice?

[puellanivis]

The v2 API will not require either WithConcurrentRead or WithConcurrentWrite. Concurrency is built in from the start, and safe enough to be not only the default, but the only implementation.

[drimmeer]

Thank you Cassondra,
So is MaxConcurrentRequestsPerFile also a built-in constant in v2? What is the value of MaxConcurrentRequestsPerFile then?

[puellanivis]

I haven’t written any support MaxConcurrentRequestsPerFile there’s currently just a total maximum concurrent requests, which is currently 64, with no knob to control it.

It will be something I could add if it’s demonstrated to be useful to have a per-file option. But often times too much concurrency can end up getting things stepping on each other. Two concurrent file uploads should fairly compete for the available packets, and the total max concurrent requests allows for a lot of optimization in reuse. That is, we don’t have to do a ton of allocs per file. And whatever n max concurrent packets is ideal for some single file transfer should likely saturate whatever socket you’re communicating over anyways. So, I’m hoping that going forward it will make sense to only have the global max inflight.

[puellanivis]

I’ve added a WithMaxInflight client option now into dev-v2. As noted, this isn’t per-file, but is shared among all requests in all goroutines. But if 100 worked well for a single file upload, then two parallel files at 100 each are super unlikely to perform any better.

[drimmeer]

Thank you Cassondra! Not sure if I need to change this option yet.
Good news is that with dev-v2, the performance is significantly improved!
Beforehand, in some situation Go ran 10 times slower than the counter Java code, and now it runs similar to Java, sometimes even better. In other situation, Go is double faster than Java.

In the situation that Go has similar performance as Java, I use a Pipe to be the input (during sending) or output(during receiving). Maybe the pipe usage lined up the data stream, thus the multi-reading(or writing) didn't work?
How to verify this?

[puellanivis]

Yeah, there are some cases where one of the directions is more limited than the other. I’m thinking it might be because it’s picking the other direction that calls either the WriteTo or ReadFrom to the non-sftp receiver, rather than the sftp one.

Unfortunately, I have no hints to you on how to debug where the slow down is, I’m also facing the same issue, and not sure why. 😂