How it works
Decisions instead of text
Section titled “Decisions instead of text”Most AI tools generate text. For reviewing documentation, text is hard to act on: it has to be read, it can be wrong in convincing ways, and a build cannot fail on it reliably.
Compass asks typed questions instead: pick one of these options, rate this on a scale, or answer yes or no. Models built for this, like TypeSafe’s Jev, answer with calibrated probabilities and a confidence score in well under a second. The answer is always one of the options you offered, and the confidence tells Compass whether to report a finding or stay quiet.
Providers and rules
Section titled “Providers and rules”Compass separates what to ask from who answers:
- Rules decide which questions to ask about a page and how to interpret the answers, like the
diataxisrule. - Providers answer the questions, like
typesafe().
Typed questions are not specific to one vendor. As more models of this kind appear, a new provider can answer the same rules. Rules can also come from outside Compass, for example to review the frontmatter of other Starlight plugins, as shown in Write a custom rule.
When Compass runs
Section titled “When Compass runs”A build should stay fast, deterministic and offline. Compass only calls a provider in two situations:
- In the dev toolbar, when the provider is configured, e.g. through a
TYPESAFE_API_KEYin a.envfile. Each page you open is reviewed once, and again whenever its content changes. This is where feedback is most useful: while you write. - In audits, when
astro buildruns withSTARLIGHT_COMPASS_AUDIT=1, typically in CI.
Regular builds and deployments never call a provider and need no API key.
One request per page
Section titled “One request per page”The questions of every rule are sent together in a single request per page, and responses are cached in Astro’s cacheDir by a hash of the request. A page is only sent again when its content, the rules or the model change.
The cost stays small: at the time of writing, Jev charges $0.042 per million input tokens, so reviewing a 3,000-token page costs around a hundredth of a cent.
What is sent
Section titled “What is sent”For each page, Compass sends the title, the description, and the Markdown or MDX source without frontmatter to the provider. Check your provider’s terms, e.g. TypeSafe’s legal documents, before reviewing private documentation.