Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Measurement Builder: perform calculations together / closer to each other, on a single DF pass ideally #98

Open
lsulak opened this issue Oct 25, 2023 · 0 comments
Open

Measurement Builder: perform calculations together / closer to each other, on a single DF pass ideally #98

lsulak opened this issue Oct 25, 2023 · 0 comments
Labels
Agent Issues touching the agent part of the project enhancement New feature or request

Comments

@lsulak
Copy link
Collaborator

lsulak commented Oct 25, 2023 •
edited by benedeki
Loading

Background

Currently, measurements are applied sequentially to a given dataframe. This might be not optimal - besides, this approach doesn't bring any benefit, it only slows things down. A better would be to somehow aggregate the measurements so that they get executed on a single pass through data. This might require implementing slightly different approach to Measure.scala using some kind of MeasurementBuilder class.

Except for Measure class, some other things might be changed here, see #79 (comment) for example.
@lsulak lsulak added enhancement New feature or request Agent Issues touching the agent part of the project labels Oct 25, 2023
@lsulak lsulak mentioned this issue Oct 27, 2023
Merged
@benedeki benedeki mentioned this issue Jan 31, 2024
Merged
@benedeki benedeki moved this from 🆕 To groom to 📋 Backlog in CPS small repos project Mar 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Agent Issues touching the agent part of the project enhancement New feature or request
Projects
CPS small repos project
Status: 📋 Backlog
Milestone
No milestone
Development

When branches are created from issues, their pull requests are automatically linked.

98-measurement-builder-perform-calculations-together-closer-to-each-other-on-a-single-df-pass-ideally AbsaOSS/atum-service
2 participants
@lsulak @ABLL526