Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Revert "split serving and monitoring, and merging monitoring and evaluation s…" #538

Merged
merged 1 commit into from
Aug 12, 2024

Conversation

axsaucedo
Copy link
Collaborator

Reverts #536

@zhimin-z As discussed let's please review larger changes especially on splits of sections. Reverting as serving and monitoring is closer to serving than evaluation as the latter is only benchmarks. We can open a thread to explore this further.

@axsaucedo axsaucedo merged commit 0f3de31 into master Aug 12, 2024
@zhimin-z
Copy link
Collaborator

zhimin-z commented Aug 12, 2024

"the latter is only benchmarks" this is not true... @axsaucedo
Industry-strength evaluation is a collection of evaluation harnesses for ML & LLM models. Evaluation harness, as far as I understand, are tools that are designed specifically to execute benchmarks, do not necessarily be benchmarks or contain benchmarks.

@zhimin-z
Copy link
Collaborator

zhimin-z commented Aug 12, 2024

As discussed in our last meeting, you agree to split monitoring and serving, right?
I also removed all benchmarks and datasets in this old pr: #511

Thus, basically, there are no pure benchmarks in this list anymore since this proposal was rejected in our last meeting, check here

I follow very closely what you expect and what we discussed in the previous meetings. @axsaucedo

@axsaucedo
Copy link
Collaborator Author

axsaucedo commented Aug 12, 2024

Industry-strength evaluation is a collection of evaluation harnesses for ML & LLM models.

Ok yes, that's a more accurate wording, that's what I mentioned when I was referring to benchmarks, but certainly benchmark framework / evaluation harness is more accurate.

I follow very closely what you expect and what we discussed in the previous meetings.

Much appreciated, it seems it was just a slight miscommunication. I indeed remember mentioning that I have been keen on finding a way to split the serving and monitoring sections, but mentioned it as an example of being quite hard to split as I hadn't seen enough growth from monitoring tools. I can see how that could have come across as a thumbs up, but yeah I meant more as something to explore.

Ok let's dive into this further in the document as discussion, as indeed I would be interested to explore splitting but I'm not yet sure how it could be merged with another section (as opposed to serving), but would be open to discuss

@zhimin-z zhimin-z linked an issue Aug 16, 2024 that may be closed by this pull request
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Explore splitting serving and monitoring
2 participants