Eval: wikimedia/structured-wikipedia for Robin knowledge base and CF RAG pipelines #59

New issue

Open

opened 2026-07-09 23:05:32 -07:00 by pyr0ball · 0 comments

pyr0ball commented

2026-07-09 23:05:32 -07:00

Owner

Source: https://huggingface.co/datasets/wikimedia/structured-wikipedia
License: CC-BY-SA 4.0 (attribution required, derivatives must be same license)
Size: ~44GB Parquet, 10.5M articles (English + French)

Full Wikipedia structured as sections, infoboxes, tables, references, Wikidata links. Parquet format — compatible with DuckDB, Polars, PyArrow.

CF relevance:

Robin: Linux migration companion needs a broad knowledge base for explaining concepts, software alternatives, and procedures to Windows/macOS migrants
General RAG: Chunked Wikipedia sections are a high-quality RAG corpus for Harrier (government info), Ibis (healthcare), and other ★ND products that need factual grounding
DuckDB-compatible format aligns with potential Crossbill data layer

Evaluate:

Chunk and index a Linux/open-source software subset for Robin RAG experiments
CC-BY-SA 4.0: attribution is mandatory — ensure any deployed RAG system surfaces article attribution in responses
CC-BY-SA copyleft: does NOT restrict model training use, only derivative datasets

Note: Start with a filtered subset (Linux, software, government processes) rather than the full 44GB.

**Source:** https://huggingface.co/datasets/wikimedia/structured-wikipedia **License:** CC-BY-SA 4.0 (attribution required, derivatives must be same license) **Size:** ~44GB Parquet, 10.5M articles (English + French) Full Wikipedia structured as sections, infoboxes, tables, references, Wikidata links. Parquet format — compatible with DuckDB, Polars, PyArrow. **CF relevance:** - **Robin:** Linux migration companion needs a broad knowledge base for explaining concepts, software alternatives, and procedures to Windows/macOS migrants - **General RAG:** Chunked Wikipedia sections are a high-quality RAG corpus for Harrier (government info), Ibis (healthcare), and other ★ND products that need factual grounding - DuckDB-compatible format aligns with potential Crossbill data layer **Evaluate:** - Chunk and index a Linux/open-source software subset for Robin RAG experiments - CC-BY-SA 4.0: attribution is mandatory — ensure any deployed RAG system surfaces article attribution in responses - CC-BY-SA copyleft: does NOT restrict model training use, only derivative datasets **Note:** Start with a filtered subset (Linux, software, government processes) rather than the full 44GB.