How Compass works

Decisions instead of text

Most AI tools generate text. For reviewing documentation, text is hard to act on: it has to be read, it can be wrong in convincing ways, and a build cannot fail on it reliably.

Compass asks typed questions instead: pick one of these options, rate this on a scale, or answer yes or no. Models built for this, like TypeSafe’s Jev, answer with calibrated probabilities and a confidence score in well under a second. The answer is always one of the options you offered, and the confidence tells Compass whether to report a finding or stay quiet.

Providers and rules

Compass separates what to ask from who answers:

• Rules decide which questions to ask about a page and how to interpret the answers, like the diataxis rule.
• Providers answer the questions.

A provider is named after the API it speaks, not after the model behind it. typesafe() speaks the API of TypeSafe’s models, which are built for typed answers. openaiCompatible() speaks the chat completions API that OpenAI, OpenRouter, DeepSeek, Ollama and many others implement, so it covers all of them with a baseUrl and a model. When another vendor offers models that answer typed questions, it gets its own provider, or you write one. Rules never change, because they only describe questions.

Rules can also come from outside Compass, for example to review the frontmatter of your own site, as shown in Write a custom rule.

When Compass runs

A build should stay fast, deterministic and offline. Compass only calls a provider in these situations:

• In the dev toolbar, when the provider is configured, e.g. through a TYPESAFE_API_KEY in a .env file. Each page you open is reviewed once, and again whenever its content changes. This is where feedback is most useful: while you write.
• In audits, when astro build runs with STARLIGHT_COMPASS_AUDIT=1, typically in CI.
• When readers use ask the docs or send feedback, from an endpoint on your server.

Regular builds and deployments never call a provider and need no API key.

One request per page

The questions of every rule are sent together in a single request per page, and responses are cached in Astro’s cacheDir by a hash of the request. A page is only sent again when its content, the rules or the model change.

The cost stays small: at the time of writing, Jev charges $0.042 per million input tokens, so reviewing a 3,000-token page costs around a hundredth of a cent.

What is sent

For each page, Compass sends the title, the description, and the Markdown or MDX source without frontmatter to the provider. Some rules add context to their questions: sidebar() sends the labels of the sidebar group, and duplicates() sends the title and the start of the most similar pages.

The ask the docs and feedback features send what readers type, together with the title and start of the pages found by their search, to the provider from your own endpoint. Check your provider’s terms, e.g. TypeSafe’s legal documents, before reviewing private documentation.