SourcesPlease public API — protos and client examples

A read-only API over what SourcesPlease publishes: sources it has archived, the factual claims extracted from them, the exact passages each claim is anchored to, and what people have concluded about whether those claims are true.

Five RPCs, two services, one protobuf package — sourcesplease_demo.v1.

Service	RPC	Answers
PublicSourceService	LookupSource(url)	what is published for a URL, one entry per archived snapshot
PublicSourceService	SearchSources(query)	ranked sources matching your search terms
PublicClaimService	ListSourceClaims(sourceId)	the claims published for one source, with their quote anchors
PublicClaimService	GetClaim(id)	one claim, with every passage it is anchored to
PublicClaimService	SearchClaims(query)	ranked claims matching your search terms

Read the proto comments. They are the specification, not decoration: they say which fields are absent rather than zero, why a claim's text is not a statement of fact, and which three attribution verbs invert the meaning of the sentence they introduce. Most ways of misreading this API are documented in the field that causes them.

---

Start here: one POST, no code generation

In a hurry, and working in Python or TypeScript? docs/python.md and docs/typescript.md each get you from a credential to a real result in one file with nothing installed.

The API speaks Connect. A unary Connect call is a plain JSON POST, so you can use the whole API with an HTTP client and nothing else — no protobuf toolchain, no generated stubs.

curl -X POST \
  -H "Authorization: Bearer $TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"url": "https://www.example.com/some-article"}' \
  https://public.sourcesplease.net/sourcesplease_demo.v1.PublicSourceService/LookupSource

The URL is always https://public.sourcesplease.net/sourcesplease_demo.v1.<Service>/<Method>.

A URL we hold nothing for answers:

{ "state": "STATE_NOT_HELD" }

and a claim id that was never issued answers HTTP 404 with

{ "code": "not_found", "message": "no such claim" }

Both examples in this repository are built on exactly that, in about 200 lines each, with no dependencies at all:

• python/ — standard library only; python3 example.py <url>
• typescript/ — Node built-ins only; node src/example.ts <url>

Generate typed stubs when you want them (below), but you do not need them to start.

---

Searching

Both search RPCs take a query parsed as a web search: bare words are ANDed and "quoted phrases" match adjacently. Hyphens inside a word are part of the word, so covid-19 searches for covid-19.

curl -X POST \
  -H "Authorization: Bearer $TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"query": "island lighthouse", "pageSize": 5}' \
  https://public.sourcesplease.net/sourcesplease_demo.v1.PublicSourceService/SearchSources

Three things about the results that are easy to get wrong:

1. SearchSources's score is ordinal within one response. It is recomputed for every query, so it is not comparable across responses and is not a quality measure. Use it to order results, never to threshold them. SearchClaims results carry no score at all; they arrive in rank order and that order is the whole of the ranking signal.
2. Ranked pages are not stable. If the corpus changes while you page, a result can move between pages or appear twice. Deep paging through a ranked set is best-effort by construction, not an oversight. For a stable walk instead, page ListSourceClaims for a source you already have.
3. Both search results are deliberately smaller than the single-record call they correspond to. SearchSources omits publishedClaimCount and snapshotOrdinal, because computing them for a whole page of results costs roughly 500x the rest of the query; call LookupSource with the url for the full record. SearchClaims is the same story: a result carries only id, text, asserter, asserterKind and createdAt, and its citations, attestations, support and attribution are always empty. That matters most for citations — every published claim has at least one, so an empty list there never means "no evidence", only "not in this response". Call GetClaim with the id for the complete claim.

A pageToken is bound to the query that produced it — sending it with different terms is an error, not a silently unrelated page.

---

Authentication

An OAuth2 client_credentials grant against

https://auth.sourcesplease.net/realms/sourcesplease/protocol/openid-connect/token

returns an access_token you send as Authorization: Bearer <token>.

TOKEN=$(curl -s -X POST \
  -d grant_type=client_credentials \
  -d "client_id=$SOURCESPLEASE_CLIENT_ID" \
  -d "client_secret=$SOURCESPLEASE_CLIENT_SECRET" \
  https://auth.sourcesplease.net/realms/sourcesplease/protocol/openid-connect/token \
  | python3 -c 'import json,sys; print(json.load(sys.stdin)["access_token"])')

Tokens are short-lived — about five minutes — and that is deliberate, not a limitation to engineer around. The API validates your token offline against the issuer's published signing keys and never calls back to ask whether it is still valid, which means the expiry time is the only mechanism by which a credential is ever withdrawn. A longer token would be a longer window in which a revoked credential still works.

So: refetch on expiry. Read expires_in from the token response rather than hard-coding a lifetime, refresh a little before the deadline, and retry once on a 401. Both examples do this in about fifteen lines; caching a token past its expiry does not extend your access, it just turns your next call into a 401.

Keep credentials in the environment (SOURCESPLEASE_CLIENT_ID, SOURCESPLEASE_CLIENT_SECRET) or in your own secret store. Nothing in this repository contains a credential, and nothing should.

If you can sign a JWT assertion, ask us for a client-jwt credential instead of a shared secret: we then hold only your public key, and you can rotate keys without coordinating a cutover with us.

---

Reading the JSON without getting it wrong

Five things about this wire format that have surprised people:

1. Empty and zero fields are omitted entirely. An empty page of claims is {}, not {"claims": []}; a state of STATE_UNSPECIFIED is absent, not spelled out. Default everything.
2. Field names are lowerCamelCase — sourceId, nextPageToken, publishedClaimCount. The underscored proto names are accepted on input as well, but the responses use lowerCamelCase.
3. Enums are strings, by name: "STATE_NOT_HELD", "SPEECH_ACT_DENIED". Treat an unrecognised value as unknown rather than as an error — new values may be added.
4. An unknown request field is rejected, and today it is reported as {"code": "unknown"} with HTTP 500. Do not retry that: fix the field name. The error message lists the ones the message accepts.
5. An absent field is not a zero. Several fields are documented as meaning something specific by their absence — a missing publication date is not "no publication date", a missing precision measurement is not "never right". The proto comments say which.

---

What the data is, and is not

This matters more than the wire format, because it is where an integration goes wrong in ways no test catches.

• A claim's text is not a fact we vouch for. It is a sentence a machine extracted from a source. The asserter on a claim is usually that extractor, not a person and not the publisher.
• Three attribution verbs invert the sentence. A claim's text is always stored affirmatively, so when attribution says a speaker SPEECH_ACT_DENIED it, their position is the opposite of the text. SPEECH_ACT_DISAGREED and SPEECH_ACT_RETRACTED do the same. SPEECH_ACT_UNSPECIFIED means the verb could not be interpreted — unknown polarity, not "none of those three". Both examples implement this; copy it.
• attestations is a floor, never a consensus. It lists people who published a judgment. It does not say nobody else looked, and it is not a tally: two entries sharing a display name are more likely two people than one person counted twice, so do not deduplicate them.
• support answers a different question from attestations — whether the cited passage supports the claim, not whether the claim is true. A passage can fail to support a claim that is nonetheless true.
• snapshotOrdinal is a rank, not an identifier. It is recomputed per response, so today's snapshot 1 may be tomorrow's 2. Store the id.

---

Setup

You do not need this to call the API. Every RPC is a plain JSON POST, and both examples run with no build step. Run setup when you want generated, typed stubs.

./setup.sh

It finds buf — offering to install one pinned, checksum-verified copy into ./bin if you do not have it — fetches the single proto dependency that no toolchain ships, checks the tree resolves, and writes generated code into gen/. Re-running is safe: each step is skipped once it is done, and nothing is installed outside this directory or needs administrator rights.

flag	effect
--yes	answer prompts automatically. Required when running non-interactively, so that nothing is ever installed without an explicit choice
--skip-codegen	resolve and verify the protos, but do not generate
--help	list the flags

Codegen reaches the network, because buf.gen.yaml uses remote plugins. Point it at local plugins if that does not suit you.

Versions, and the one file setup fetches

The protos live under proto/ at their canonical import path, so generated code built here matches ours byte for byte. They use protobuf editions (edition = "2023"), which older toolchains reject outright — which is why setup pins a known-good buf rather than trusting whatever is on your PATH.

Toolchain	Version
buf	1.72.0
protobuf / protoc	35.0
@bufbuild/protobuf, @bufbuild/protoc-gen-es	2.12.x

source_service.proto imports google/type/date.proto — an Apache-2.0 googleapis file rather than a protobuf well-known type, so nothing in a toolchain supplies it. That is the one file setup.sh fetches, and the script names the alternatives if it cannot reach it. buf.yaml already excludes proto/google from linting, because googleapis' own package naming does not satisfy the rules applied here.

---

Support and stability

sourcesplease_demo.v1 is a demo surface: narrow on purpose, and still settling. We check every change against the last published state of this repository for wire and JSON compatibility, so an existing field will not change meaning under you — but expect new fields, and expect additional RPCs.

Tell us what you need that is not here. The set of five RPCs is a starting point, not a considered ceiling.