Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Retrieval eval incomplete #449

Merged
merged 14 commits into from
Oct 24, 2024
Merged

Retrieval eval incomplete #449

merged 14 commits into from
Oct 24, 2024

Conversation

Louis-Mozart
Copy link
Contributor

Comments from the previous PR have been included. The set sorting ensures that we always sample the set in the same order. Hence, make the results reproducible.

@Demirrr
Copy link
Member

Demirrr commented Oct 15, 2024

The set sorting ensures that we always sample the set in the same order. Hence, make the results reproducible.

A set cannot be sorted.

Edit: Could you please explain your reasoning behind sorting a set to make the code reproducable ? @Louis-Mozart

@Demirrr Demirrr self-requested a review October 15, 2024 10:31
@Demirrr
Copy link
Member

Demirrr commented Oct 17, 2024

@Louis-Mozart is there any update ?

@Louis-Mozart
Copy link
Contributor Author

Hi @Demirrr. Not really. But I think it is possible because, in Python, sets do not guarantee the order of their elements when converted to lists, this means that each time we convert a set to a list, the order may vary (we can experience this by running the current script many times and add exits like so print(object_properties) exit(0) ). My point is that this inconsistency directly affects the random.sample() function because if the input list’s order changes, the sampled elements may differ even when the same random seed is set. Using sorted(), we ensure that the set converted to a list has a consistent order every time, allowing random.sample() to work with the same ordered input, making the output reproducible across different runs.

@Demirrr
Copy link
Member

Demirrr commented Oct 17, 2024

But I think it is possible because

What does it in this sentence refer to ?

in Python, sets do not guarantee the order of their elements when converted to lists

A set does not have an ordering in any programming language. Therefore, any such convertion does not have any guarantee.

No worries, we will fix this issue

@Demirrr
Copy link
Member

Demirrr commented Oct 17, 2024

@LckyLke or @alkidbaci could you take over this PR from @Louis-Mozart ?

In case any of you would have a question, you can directly contact to @Louis-Mozart.

@Louis-Mozart
Copy link
Contributor Author

What does it in this sentence refer to?

To the reproducibility of the code.

A set does not have an ordering in any programming language. Therefore, any such convertion does not have any guarantee.

I am not intending to sort a set as this is not possible. But the idea was just to get a sorted list from a set.

examples/retrieval_eval.py Outdated Show resolved Hide resolved
@LckyLke LckyLke requested review from LckyLke and removed request for Demirrr October 24, 2024 07:44
@github-actions github-actions bot temporarily deployed to pull request October 24, 2024 07:58 Inactive
@LckyLke LckyLke merged commit fec4d1d into develop Oct 24, 2024
4 checks passed
@github-actions github-actions bot temporarily deployed to pull request October 24, 2024 08:29 Inactive
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants