Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DBWorker limited by # of CPUs which is not always a good gauge of how to limit a DB Pool. #463

Open
lwahlmeier opened this issue Mar 16, 2021 · 1 comment
Labels
enhancement New feature or request

Comments

@lwahlmeier
Copy link
Contributor

Currently the DBWorker is limited by the number of cores on the host machine:

	if worker > runtime.NumCPU() {
		worker = runtime.NumCPU()
	}

A DB Connection pool is not usually bound to the number of CPU cores as it a lot of cases it has more to do with network and DB latency.

It would be nice to either remove this limit all together, or implement a way of overriding it.

Let me know if either of these options will work and I will happily put in a PR for it.

@lmangani lmangani added the enhancement New feature or request label Mar 16, 2021
@negbie
Copy link
Member

negbie commented Mar 18, 2021

Hi @lwahlmeier,
hmm that code is quite old and not in use in our production for some time now since we moved to ClickhouseDB which needs a different concurrency pattern to work efficently. So take my words with care but I think you can just remove that condition if you keep in mind that this setting is for the worker count. Every worker will open/maintain

p.db.SetMaxOpenConns(config.Setting.DBWorker * 4)
p.db.SetMaxIdleConns(config.Setting.DBWorker)

So while it can improve performance on machines with high CPU core count I think removing this restriction will reduce the performance for some users who often thinks more is better and set this value to high which will cause contention. Nevertheless I'm ok with it and would kindly ask you for a PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

3 participants