Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Requires large chunk of memory when having many sources #161

Open
martwo opened this issue Jul 21, 2023 · 2 comments
Open

Requires large chunk of memory when having many sources #161

martwo opened this issue Jul 21, 2023 · 2 comments
Labels
enhancement New feature or request

Comments

@martwo
Copy link
Collaborator

martwo commented Jul 21, 2023

This lines brakes when using many sources (400+) due to large memory allocation of 20+GB. Maybe use int32 instead of int64!?

idxs = np.argwhere(mask_dec[:, mask])

@martwo martwo added the enhancement New feature or request label Jul 21, 2023
@martwo
Copy link
Collaborator Author

martwo commented Jul 21, 2023

It is not possible to specify the dtype of the resulting idxs array. Also np.nonzero(a) does not support to specify the dtype of the resulting array :( One might have to write a batched version of np.argwhere and batch chunks of sources.

@martwo
Copy link
Collaborator Author

martwo commented Jul 21, 2023

The issue arises for the pre_event_selection_method of the background generation method, when O(10 million) MC events are selected for all the sources.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant