Every corpus, and how far it has come.
Every corpus moves along the same pipeline. This page says where each one stands.
Choose the field
Only fields where the answer has to come from the original document, and where the licence lets us hold it.
Take the originals
Fetched at the publisher’s own cadence, at the crawl delay it asks for, and followed from then on.
Govern, then store
De-duplicated, parsed into retrievable units — sections, articles, fields — the original kept beside them.
Read it into structure
A model turns each document into fields, summaries and key points — the structure filters work on, and the reason a reader can take one layer at a time. How hard that reading is decides which model gets it.
Index, then serve
search returns cards, read goes brief → detail → full, ask answers with citations. Nothing comes back whole unless it was asked for.
The judgements a later step will trust: disambiguation, holdings, anything a mistake would propagate.
Structured extraction at volume — fields, summaries, translations, the long middle of the work.
What is cheap to check and expensive to skip: keywords, classification, normalising names and units.
| Dataset | Scale | Updates | Stage | Domain | Open the row |
|---|---|---|---|---|---|
| Research6 | |||||
| arXiv full text Every arXiv paper, parsed to the section, synced T+1. | 3M+ papers · complete | T+1 | LiveAnswering through the API. | papers→ | |
| Whole corpus refined by LLM: a TLDR and keywords per paper, a summary per section, section map with token counts. Source arXiv · open access | |||||
| AI researchers Who works on what, where they trained, what they published. | nearly 190k researchers · 170k profiles | daily | LiveAnswering through the API. | talent→ | |
| One LLM-researched profile per person (education, career, representative work, open source); Google Scholar metrics rendered live, staleness flagged. Source Public homepages, Scholar, OpenAlex | |||||
| Objective appraisal Not how often a paper was cited — which senior scholar said what about it, in their own paper. | 5,400+ fellows · 8M+ citation sentences | continuous | LiveAnswering through the API. | appraise→ | |
| Every citation sentence in the fellows’ own papers is extracted ahead of time, so any paper can be matched by title, arXiv id or DOI with no API call. 5.26M references parsed, 5M sentences classified by type and stance — this is what the appraise domain answers from. Source arXiv full text + 15 public fellow and academician rosters | |||||
| NIH funded projects Who got funded for what, with abstracts and amounts. | FY1985 onward · every category | monthly | LiveAnswering through the API. | grants→ | |
| Every ExPORTER category plus patents, in hand; the current fiscal year arrives through the API. In the grants domain with NSF and CORDIS — about 1.3M projects. Source NIH RePORTER · public domain | |||||
| NSF awards Every award the US National Science Foundation has made. | since 1959 · complete | weekly | LiveAnswering through the API. | grants→ | |
| Raw exports in hand. Indexed with NIH as one grants domain; institutions and investigators still to be normalised against the researcher corpus. Source NSF Award Search · public domain | |||||
| EU research projects The third funding line, beside NIH and NSF. | FP5 – Horizon Europe | monthly | LiveAnswering through the API. | grants→ | |
| In the grants domain beside NIH and NSF (106,859 projects, ids prefixed cordis:); amounts are in euros. Institutions still to be normalised against the entity backbone. Source CORDIS · EU open data | |||||
| Law8 | |||||
| World statutes Read to the article, across jurisdictions, in Chinese or the original. | 26 jurisdictions · 1.5M+ articles | quarterly | LiveAnswering through the API. | law→ | |
| Article-level LLM refinement: heading, norm type, subjects, conditions, consequences, key figures, cross-references resolved to article ids; zh/en translation. Source Official gazettes and legislative portals | |||||
| Chinese judgments The judgments out of court documents that run to hundreds of millions — holding and reasoning both there. | 17M+ judgments · criminal and civil | continuous | LiveAnswering through the API. | cases→ | |
| “Court documents” is a wide bag — judgments, rulings, mediations, decisions, notices — and the published ones run into the hundreds of millions. We take the judgments out of it: 2.65M criminal, complete, and 14.8M civil, administrative and enforcement, backfilled to the end. LLM brief per judgment: cause, disputed issues, facts, circumstances, reasoning, outcome, cited articles, amounts. Behind the briefs sit 34.55M original documents — more than the index holds, the surplus still without a brief — split into five sections: parties, claims, facts, the court’s reasoning, the decision. The verbatim text is not served by default: it carries party names, dates of birth and addresses, so the API gives the parsed layer and the text itself needs the cases:fulltext permission. Every response carries coverage. Source Published judgment documents | |||||
| US case law Federal and state opinions, from the printed reporters to last quarter. | 10M+ opinions · federal and state | quarterly | LiveAnswering through the API. | cases→ | |
| 3,438,487 opinions answer through the Cases domain as country=us; extraction has run over 10.07M and the rest follows as the store behind it lands. Source CourtListener, incl. the Caselaw Access Project · public domain | |||||
| UK judgments The licence is confirmed and free; nothing is fetched until it is signed. | England & Wales, Supreme Court, Privy Council | — | Awaiting licenceThe data exists; we will not touch it until the licence is in hand. | — | |
| The National Archives have confirmed our use counts as computational analysis, which needs a transactional licence and carries no fee. The application is written and waiting to be sent. Source Find Case Law, The National Archives | |||||
| US statutes and regulations Federal and state statutes, with the regulations beside them. | 2.4M+ sections · 53 jurisdictions | daily | LiveAnswering through the API. | us_law→ | |
| 4.98M provisions answer through the US law domain, each extracted article by article. The live eCFR replaces the quarterly CFR snapshot; 928k provisions are marked superseded. Compilation licensed CC BY, attributed. Source Open statute compilation (CC BY) + eCFR · official | |||||
| EU law Regulations, directives and decisions — where the law domain was thinnest. | 40k+ documents | monthly | RawOriginals downloaded and kept current; parsing not started. | — | |
| CELEX notices in hand, text only rather than full metadata. Parsing into articles comes next. Source EUR-Lex / Cellar · EU open data | |||||
| UK legislation The open half of UK law — statutes, not judgments. | 18k primary acts · 190k with secondary | weekly | RawOriginals downloaded and kept current; parsing not started. | — | |
| CLML XML keeps structure and amendment notes. Fetched at the crawl delay the site asks for. Open Government Licence v3: attribution required. Source legislation.gov.uk · OGL v3 | |||||
| Federal Register Every federal rule, proposed rule and notice, the day it published. | 1994 onward · every issue | daily | LiveAnswering through the API. | fr→ | |
| 1,008,931 documents answer through the Federal Register domain — action kind, issuing agencies, affected parties, effective and comment-close dates. Source Federal Register / govinfo · public domain | |||||
| Life sciences3 | |||||
| Clinical trials Every registered study: design, arms, eligibility, outcomes. | 600k+ studies | daily | LiveAnswering through the API. | trials→ | |
| In the trials index, all 603,730. 586k refined: conditions, interventions, endpoints, eligibility in structured form. Source ClinicalTrials.gov · public domain | |||||
| US drug labels What a drug is approved to do, and what it must warn about. | ~160k labels | daily | LiveAnswering through the API. | drugs→ | |
| Drugs index: 256,282 label versions keyed by set_id, each readable by section — indications, warnings, adverse reactions. Source DailyMed, NLM · public domain | |||||
| openFDA labels The same labels, plus recalls, shortages, approvals and device filings — already JSON. | 32 endpoints · the full mirror | weekly | LiveAnswering through the API. | drugs→ | |
| Cross-checked against DailyMed and kept as the machine-readable view. Adverse-event files (FAERS, MAUDE) need an explicit pull; label endpoints refresh daily. Source openFDA · public domain | |||||
| Markets7 | |||||
| SEC filings 10-K, 10-Q and 8-K as filed — and now 6-K, DEF 14A, 20-F, CORRESP, SC 13G/A. | 1M+ filings | daily | LiveAnswering through the API. | filings→ | |
| 392,258 8-K and 46.5k 10-K/20-F extracted item by item. The filings domain answers over two layers — 1.21M with vectors, and only the 105k 10-Q quarterlies by keyword alone. Financial figures come from XBRL, not from a model. Source SEC EDGAR · public domain | |||||
| A-share disclosures Annual, interim and quarterly reports of listed Chinese companies. | ~190k reports | nightly | LiveAnswering through the API. | filings→ | |
| Backfill finished after a paging bug was found: 16,133 reports recovered. 63k refined into the same item structure as SEC filings. Source CNINFO, the designated disclosure venue | |||||
| A-share current reports The one-off disclosures between the quarterly reports. | ~95k disclosures | nightly | LiveAnswering through the API. | filings→ | |
| 507k of them sit in the full store behind the filings domain: searchable by words, marked layer=pg, with coverage saying how much of the corpus carries vectors. Source CNINFO, the designated disclosure venue | |||||
| US financial figures (XBRL) The financial figures themselves, each with the filing it came from. | company facts, complete | daily | RawOriginals downloaded and kept current; parsing not started. | — | |
| The authoritative source for US financial numbers — no model is asked to read a financial table. Source SEC · public domain | |||||
| CSRC enforcement Who the securities regulator punished, and for what. | 2,000+ penalties and bans | daily | LiveAnswering through the API. | filings→ | |
| Full decisions extracted — parties, conduct, legal basis, sanction — and in the filings index. Source CSRC · official | |||||
| Exchange inquiry letters What the exchanges asked listed companies, and how they answered. | ~18k letters | daily | LiveAnswering through the API. | filings→ | |
| 10,434 letters extracted with their replies; in the filings index. Shenzhen inquiry and concern letters, Shanghai inquiries, plus disciplinary decisions. Source SZSE and SSE · official | |||||
| Federal spending Every federal award and contract — money as it flows, not project abstracts. | FY2021 onward · awards and contracts | weekly | RawOriginals downloaded and kept current; parsing not started. | — | |
| Feeds the entity backbone (organisation → federal money); it complements NIH and NSF rather than replacing their abstracts. Source USAspending.gov · public domain | |||||
| Security2 | |||||
| Vulnerabilities Every disclosed vulnerability, with what is known to be exploited. | NVD + CVE V5 + KEV + EPSS | daily | LiveAnswering through the API. | cve→ | |
| 384k records answer through the vulnerabilities domain, with CISA exploited-in-the-wild flags and EPSS probabilities to rank by. Showing NVD data carries a not-endorsed notice. Source NVD · CVE Program · CISA KEV · FIRST EPSS | |||||
| Package advisories The same vulnerabilities seen from the package side. | all ecosystems | daily | RawOriginals downloaded and kept current; parsing not started. | — | |
| Raw advisories in hand; they join the vulnerability records by CVE id. Source OSV.dev · open | |||||
| Standards1 | |||||
| Chinese national standards Which GB standard applies, and whether its text is public. | nearly 80k standards · catalogue only | weekly | LiveAnswering through the API. | standards→ | |
| Catalogue metadata only — number, title, status (72,083 recommended, 6,201 mandatory, 1,027 guiding), adoption, where the text lives. All 79,311 are in the standards index; the standards themselves are not ours to republish. Source SAMR national standards platform | |||||
| Reference1 | |||||
| Entity backbone Three vocabularies so names join instead of merely matching. | ROR · GLEIF · MeSH | daily | LiveAnswering through the API. | entities→ | |
| 4.58M organisations and 5.52M aliases answer through /v1/entities, with 5.30M links over 2.66M documents back to the organisation that filed, ran, received or labelled — 1.09M on an identifier, 1.13M on an unambiguous name, 3.08M adjudicated by a model and marked as such. Source ROR CC0 · GLEIF CC0 · MeSH public domain | |||||
| Yours1 | |||||
| Bring your own data Hand us a corpus and get it back as a domain. | your corpus | — | OpenNot ours — yours. Talk to us and it becomes a domain. | Talk to us→ | |
| The same pipeline behind every dataset here — parse, section, refine, cross-link — run on your documents, under your keys. Source Public contribution or private engagement | |||||
Inventory updated 2026-10-10