Indexing with historycache consumin large storage for a small repository

[vinaybs1983]

HI,
Using opengrok version 1.12.6 version , when running indexing with history-cache enabled, the storage for history-cache for a repository is taking more than the storage used for s SRC rrepo. can you please help us regarding this .

Regards,
Vinay BS

[vladak]

What are the approximate sizes you're dealing with ? If a repository has a rich history, the size of the history cache can be significant, esp. in the recent OpenGrok versions that no longer compress the history cache files. Also, it depends whether merge changesets and renamed file handling are enabled and of course how much they are used in the repository in question. The former is (now) by default, the latter is not. See https://github.com/oracle/opengrok/wiki/Indexer-configuration

Incidentally, I had a few conversations recently where people mentioned the size of the data root being a problem for them. One thing is that storage space is mostly cheap these days - one can buy these 12/14 TB drives, "spinning rust", but still. If one cares about performance, then we are talking SSD which will be definitely expensive for larger sizes. Another thing is that my priority for OpenGrok is that the latency of the user interface is paramount and everything else is secondary to that. I often repeat that in this architecture, it's all trade-offs, all the way down, and this is mostly the trade-off I chose. Can we shave off some latency by increasing data root size ? If yes, I usually go for it.

There is currently no way how to turn off history cache without disabling the history completely, #4265 is sort of tracking that feature. But, keep in mind that if implemented, this save the disk space, however will slow down both the user interface (when viewing the history) as well as the indexer. The latter slowdown will be significant (likely orders of magnitude) as it will need to query the history for each file individually.

[vladak]

Another solution might be to use a file-system or storage system that supports transparent compression, e.g. ZFS.