-
Notifications
You must be signed in to change notification settings - Fork 20
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add PARENT #14
Comments
I'm not familiar with this metric or ToTTo, but I found a bit of code we could try and integrate here from the original authors. Looks to be somewhat straight forward for someone who understands the general format of the task. Can just have a wrapper around table_text_eval.py in the compute method for the PARENT metric. I can't give this a shot till next weekend most likely, but can take a look then if no one else can. https://github.com/google-research/language/tree/master/language/table_text_eval |
PARENT will require some dataset-specific processing, so it is not super straightforward to integrate (but still super important to add) |
I completely missed this. I agree it is super important to add. Unfortunately, I don't think I have the bandwidth to integrate it. Do you think someone in the metrics working group for v2 would be able to integrate this? |
@sebastianGehrmann, This https://github.com/KaijuML/parent looks like an easier implementation to integrate. This metric is only for data-to-text so do we have logic for only running a metric on specific datasets? Also as I understand it we would have to implement preprocessing separately for every data-to-text dataset, happy to try for WebNLG, E2E and DART. |
General version: https://arxiv.org/abs/1906.01081
For ToTTo: https://arxiv.org/abs/2004.14373
The text was updated successfully, but these errors were encountered: