The Coleridge Initiative at NYU has been researching Rich Context to enhance search and discovery of datasets used in scientific research – see the Background Info section for more details. Partnering with experts throughout academia and industry, NYU-CI has worked to leverage the closely adjacent fields of NLP/NLU, knowledge graph, recommender systems, scholarly infrastructure, data mining from scientific literature, dataset discovery, linked data, open vocabularies, metadata management, data governance, and so on. Leaderboards are published here on GitHub to track state-of-the-art (SOTA) progress among the top results.
The first challenge is to identify the datasets used in research publications, initially focused on the problem of entity linking. Research papers will generally mention the datasets they've used, although there are no formal means to describe that metadata in a machine-readable way.
Identifying dataset mentions typically requires:
- extracting text from an open access PDF
- some NLP parsing of the text
- feature engineering (e.g., paying attention to where the text is located in the paper)
- modeling to identify up to 5 datasets per publication
See Evaluating Models for Entity Linking with Datasets for details about how the Top5uptoD leaderboard metric is calculated.
| source | precision | repo | corpus | date | contact |
|---|---|---|---|---|---|
| LARC | 78.36 | link | v0.1.5 | 2019-09-26 | @philipskokoh |
- How To Participate
- Corpus Description
- Downloading Resource Files
- Background Info
- Workflow Stages
- Glossary Terms
Use of open source and open standards are especially important to further the cause for effective, reproducible research. We're hosting this competition to focus on the research challenges of specific machine learning use cases encountered within Rich Context – see the Workflow Stages section.
If you have any questions about the Rich Context leaderboard competition – and especially if you identify any problems in the corpus (e.g., data quality, incorrect metadata, broken links, etc.) – please use the GitHub issues for this repo and pull requests to report, discuss, and resolve them.