Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: Add Parquet format to data page #629

Open
wants to merge 11 commits into
base: main
Choose a base branch
from
Open

feat: Add Parquet format to data page #629

Changes from 6 commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
9696184
Update data page
jeremyarancio Oct 28, 2024
3f51ae9
Update data page
jeremyarancio Oct 28, 2024
35a0805
Pierre feedback
jeremyarancio Oct 29, 2024
5c8fb76
docs: :memo: Update description based on feedbacks
jeremyarancio Oct 29, 2024
49a3bcd
fix: :art: Wrong lang: aa -> en
jeremyarancio Nov 5, 2024
53a4206
fix: :art: Fix <p></p>
jeremyarancio Nov 5, 2024
f2a7d31
Update lang/en/texts/data.html
jeremyarancio Nov 17, 2024
f833a66
Update lang/en/texts/data.html
jeremyarancio Nov 17, 2024
c9b23d1
Merge branch 'main' into parquet-db
teolemon Nov 17, 2024
40ae4b4
Merge branch 'main' into parquet-db
jeremyarancio Nov 17, 2024
cb04b07
Trigger workflow
jeremyarancio Nov 18, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions lang/en/texts/data.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,28 @@ <h3>JSONL data export</h3>

<p>A suitable way to exploit the database is to use DuckDB, an in-process analytical tool designed to process large amount of data in a fraction of seconds. You can read our <a href="https://blog.openfoodfacts.org/en/news/food-transparency-in-the-palm-of-your-hand-explore-the-largest-open-food-database-using-duckdb-%f0%9f%a6%86x%f0%9f%8d%8a">blog post</a> where we walk you through exploring and processing the Open Food Facts database with DuckDB</p>

<h3>Parquet Data Export on Hugging Face</h3>

<p>A simplified version of the JSONL dump is also available in the <a href="https://parquet.apache.org/">Parquet format</a>. During the conversion, we filtered columns that contains duplicated information, are used for internal debugging, or are simply irrelevant for users.</p>

<p>The Parquet format has proved to be handy:<p>

<ul>
<li>Data are organized by column, rather than by row, which saves storage space and speeds up analytics queries, i.e. you can select just the columns you care about, optimizing query performances, even on entry-level computers.</li>
jeremyarancio marked this conversation as resolved.
Show resolved Hide resolved
<li>Highly efficient data compression and decompression, making it good for storing and sharing big data of any kind,</li>
jeremyarancio marked this conversation as resolved.
Show resolved Hide resolved
<li>Supports complex data types and advanced nested data structures.</li>
</ul>

<p>The dataset is available on <a href="https://huggingface.co/datasets/openfoodfacts/product-database">Hugging Face</a>, a collaborative Machine Learning ecosystem where developers and researchers can share models and datasets.</p>

<dl>
<dt>Link</dt>
<dd><a href="https://huggingface.co/datasets/openfoodfacts/product-database/resolve/main/products.parquet">https://huggingface.co/datasets/openfoodfacts/product-database/resolve/main/products.parquet</a>
</dd>
</dl>

</p>Find more information in the <a href="https://wiki.openfoodfacts.org/Reusing_Open_Food_Facts_Data#Parquet_file_hosted_on_Hugging_Face_.28beta.29">Wiki</a>, including guidelines for data reuse and example queries to get started.</p>

<h3>CSV Data Export</h3>
<p>Data for all products, or some of the products, can be downloaded in the CSV format (readable with LibreOffice, Excel and many other spreadsheet software) through the <a href="https://world.openfoodfacts.org/cgi/search.pl">advanced search form</a>.</p>

Expand Down
Loading