Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SSE stream optimization #173

Merged
merged 1 commit into from
Feb 12, 2024
Merged

SSE stream optimization #173

merged 1 commit into from
Feb 12, 2024

Conversation

szkiba
Copy link
Collaborator

@szkiba szkiba commented Feb 12, 2024

On the SSE stream, the snapshot and cumulative events are displayed as JSON objects, where the metric names are the property names. The property values are also JSON objects, in which the property names are the names of the aggregation function (avg, min, max, etc.). This two-level object hierarchy has not caused any problems so far, the amount of data has been acceptable.

Currently, aggregate counting per tag is still supported, but by default it is only allowed for the group tag. It would be necessary to turn on the aggregate calculation for the url and name tags as well (per URL graphs, summary, etc.). With the current SSE stream format, this would significantly increase the amount of data. This would have a direct impact on the k6 memory requirement, so it is a problem.

Based on measurements, it can be concluded that using a two-level array structure instead of the two-level object structure used in snapshot and cumulative events significantly reduces the size of the events and thus also the memory requirement.

For example, for the scripts/demo/demo.js test script, the size of the SSE stream is reduced by 65%. With url and name tags enabled, the new format results in a 10% size increase compared to the original size (without these tags). That is, the new format enables aggregate calculation based on url and name tags, which is a prerequisite for several requested features.

Therefore, instead of the two-level object structure, the use of a two-level array structure must be implemented in the snapshot and cumulative events. The packages/model package can hide SSE stream changes from the rest of the dashboard UI.

On the SSE stream, the snapshot and cumulative events are displayed as JSON objects, where the metric names are the property names. The property values are also JSON objects, in which the property names are the names of the aggregation function (avg, min, max, etc.). This two-level object hierarchy has not caused any problems so far, the amount of data has been acceptable.

Currently, aggregate counting per tag is still supported, but by default it is only allowed for the group tag. It would be necessary to turn on the aggregate calculation for the url and name tags as well (per URL graphs, summary, etc.). With the current SSE stream format, this would significantly increase the amount of data. This would have a direct impact on the k6 memory requirement, so it is a problem.

Based on measurements, it can be concluded that using a two-level array structure instead of the two-level object structure used in snapshot and cumulative events significantly reduces the size of the events and thus also the memory requirement.

For example, for the scripts/demo/demo.js test script, the size of the SSE stream is reduced by 65%. With url and name tags enabled, the new format results in a 10% size increase compared to the original size (without these tags). That is, the new format enables aggregate calculation based on url and name tags, which is a prerequisite for several requested features.

Therefore, instead of the two-level object structure, the use of a two-level array structure must be implemented in the snapshot and cumulative events. The packages/model package can hide SSE stream changes from the rest of the dashboard UI.
@szkiba szkiba linked an issue Feb 12, 2024 that may be closed by this pull request
@szkiba szkiba merged commit 5a730f7 into master Feb 12, 2024
10 checks passed
@szkiba szkiba deleted the feature/sse-stream-optimization branch February 12, 2024 17:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

SSE stream optimization
1 participant