Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add PARENT #14

Open
sebastianGehrmann opened this issue Jan 30, 2021 · 4 comments
Open

Add PARENT #14

sebastianGehrmann opened this issue Jan 30, 2021 · 4 comments
Labels
new-metric-light Adding a lightweight metric

Comments

@sebastianGehrmann
Copy link
Contributor

General version: https://arxiv.org/abs/1906.01081
For ToTTo: https://arxiv.org/abs/2004.14373

@nishantsubramani
Copy link
Contributor

I'm not familiar with this metric or ToTTo, but I found a bit of code we could try and integrate here from the original authors. Looks to be somewhat straight forward for someone who understands the general format of the task. Can just have a wrapper around table_text_eval.py in the compute method for the PARENT metric. I can't give this a shot till next weekend most likely, but can take a look then if no one else can.

https://github.com/google-research/language/tree/master/language/table_text_eval

@sebastianGehrmann sebastianGehrmann added the new-metric-light Adding a lightweight metric label Dec 16, 2021
@sebastianGehrmann
Copy link
Contributor Author

PARENT will require some dataset-specific processing, so it is not super straightforward to integrate (but still super important to add)

@nishantsubramani
Copy link
Contributor

I completely missed this. I agree it is super important to add. Unfortunately, I don't think I have the bandwidth to integrate it. Do you think someone in the metrics working group for v2 would be able to integrate this?

@jordiclive
Copy link

jordiclive commented Mar 23, 2022

@sebastianGehrmann, This https://github.com/KaijuML/parent looks like an easier implementation to integrate. This metric is only for data-to-text so do we have logic for only running a metric on specific datasets?

Also as I understand it we would have to implement preprocessing separately for every data-to-text dataset, happy to try for WebNLG, E2E and DART.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
new-metric-light Adding a lightweight metric
Projects
None yet
Development

No branches or pull requests

3 participants