Memory usage of orderers and peers increases without explanation and then stops

[ConstantRohmer]

Description

Setup

I am using HLF v3.0.0-beta and SmartBFT as consensus. 4 orderers, 4 organizations (1 peer each).

What seems wrong

I see a linear increase in the memory usage of my orderers (all of them not only the leader) and peers until a certain point (it reaches values between 400 and 750MB, although I don't have an overflow: neither for my entire system nor for the docker containers). The limit of this increase doesn't seem to be the value of memory used but the time after which it stops increasing, because no matter the value it is at, it practically stops increasing after ~10h (x-axis = 20k)
We can clearly see that on the plot I added.

More explanations on the plot itself and my experimentation :

The plot represents the Evolution of memory usage of orderers and peers nodes for 4 different runs that each lasted 15h. (The x-axis is not very clear, but 4 units corresponds to ~7s which makes the total around 58h = 4x ~15h)
I used multiple channels (5 everytime) for each of my runs, but depending on the run the channels are not updated at the same frequency which perfectly explains the difference between the 2 orange lines on runs 2 and 4, but this is not the problem here.
Each run is actually a repetition (7500x here) of the same actions (some write and reads) and I open a gateway (and close it at the end) for every repetition.

Hypotheses on the problem

According to what was mentioned in this issue, I originally thought that the opening/closing of the gateway connection between each repetition could be the reason I am seeing this increase. However it would only increase the peer's memory usage, but the orderers should not be affected, so it doesn't fit.
In any cases this doesn't explain the increase stopping at x=~20k for every runs I made which are not consistent with a memory leak.

I also thought about the ledger's storage, but beeing stored on disk, I don't see why it would increase the memory usage.

Steps to reproduce

No response

[denyeart]

From my recollection memory use was investigated in the early days of Fabric and the conclusion was that this is a typical pattern for Go applications. Go greedily requests and holds on to memory from the operating system. Even garbage collected memory isn't returned to OS and is reserved for future use. As long as the memory use eventually stabilizes and plateaus, the thought was that this pattern is typical and not an indication of a memory issue on the Fabric side.

Of course, something could have changed in the subsequent years. If somebody could profile the memory use and identify specific problem areas that would be much appreciated.