Using AI to Check AI

About This Page

The natural way to test a site like this one is to hand it to an AI. Paste in the address, ask what it makes of the site, and let a machine that has read more than any of us settle it. Ask casually, though, and it often does the reverse, and why it does is what tells you whether to trust the answer.

Why the machine leans before it reads

A language model is built from the text it was trained on, and that text reflects the settled scholarly consensus: the works are Shakespeare’s, the man from Stratford wrote them, and the authorship question is a fringe curiosity. Far more has been written from that position than against it, and much of the little that argues the other way is quoted only to be dismissed. So the model’s starting point is not neutral ignorance. It is a lean, toward treating the conventional attribution as the baseline and any challenge to it as the thing that must be explained away.

Ask such a model to “review this site” and it tends to do three things, none of them on purpose. It substitutes a familiar summary of the controversy for what the site actually says. It treats the consensus as neutral ground rather than as one of the positions under examination. And it reads the site’s carefully hedged conclusions as weaknesses it has cleverly caught, when they are the site’s considered position, already argued on the page in front of it. These are the same moves catalogued for human readers in Reading Without a Reference Point and Logic Errors in the Authorship Debate; a machine is not immune to them.

This is not hypothetical

It has already happened. One widely used AI research assistant, with full public access to this site, characterised it on a first pass from general knowledge: it treated the Stratfordian position as the neutral default and mistook several already-answered points for newly discovered flaws. Asked to go back and read the site’s own primary pages, it produced an accurate and even-handed summary and conceded that its first pass had not been a defensible evaluation. Nothing on the site had changed between the two answers; only the depth of reading had.

The instruction that corrects for it

The failure is a property of the tool, not of the reader or the site, and there is a simple way to correct for it. When you ask an AI about this site, do not ask it to review the site. Tell it what to read first, and tell it to characterise the site from that and nothing else:

That single instruction is enough. It makes the model read the pages before judging them, instead of judging from what it walked in assuming. What you are asking of it is a fair account of what the site claims, not a verdict on whether those claims are true. That second question is settled against the documents; another model, sharing the same training, is a weak check on it, as On AI sets out.

Why those five pages

Each is chosen to close off a particular way of getting the site wrong.

llms.txt is the site’s own statement, written for AI systems, of exactly what it does and does not claim. Criteria sets out the method, the two frameworks, applied to every candidate, the Stratford man included. Objections is the case against the site’s approach, stated without softening and answered. What This Site Claims and Doesn’t lists the positions the site is routinely assumed to hold and does not. What Would Settle It names the single kind of document that would close the question. Read those five and a fair characterisation becomes hard to avoid, and an unfair one becomes easy to catch.

A test you can apply yourself

You need not trust the AI either. You can check its answer against the site’s own position, which is this: the primary-source record for the Stratford man is anomalous by the ordinary standards of literary biography; the authorship question is legitimately open; and no candidate, including Shakspere, currently meets the conditions that would close it.

Check the answer, not its confidence

Suppose the machine comes back with a summary. It will be fluent, organised, and often footnoted. None of that tells you whether it read the site or read its training. Three habits separate a grounded answer from a merely plausible one.

Citations are not corroboration. An AI that hangs sources on its summary has not thereby grounded it in them. Follow the links. If they lead back to the site’s own pages, the summary is answerable to the site; if they lead to third-party commentary, to advocacy for one candidate, or to general articles that never mention the site at all, the model built its picture from adjacent material, and any accuracy is luck rather than method. A summary can be perfectly accurate and still rest on citations that support not a word of it.

The sources a model finds on its own are rarely neutral. Left to gather its own material on a contested subject, a model pulls in advocacy on one side and dismissal on the other, because that is most of what has been written. A fair summary is anchored to the site’s own pages; an unfair one is anchored to whichever partisan neighbour the model happened to open. That is the whole reason the instruction above names the pages: it takes the choice of sources out of the model’s hands.

The dangerous errors are true statements with the caveat removed. An outright falsehood is easy to catch. Far harder is an accurate claim stripped of its qualification. A summary may rightly report that a candidate “leads” one of the site’s frameworks while quietly dropping the site’s own warning that leading a framework is not evidence of authorship. Whenever an AI reports a ranking, a lead, or a “strongest” anything, check whether it kept the qualification the site attaches to it.

It cuts both ways

The same lean can be pushed in the opposite direction. An AI prompted by someone already convinced of a hidden author can be led to overstate the case just as easily, to present a circumstantial pattern as proof, or an open question as a solved one. The instruction above guards against that too, because it does not ask the model to agree with the site. It asks it to report what the site actually claims, which is deliberately less than either camp would like. A tool that can be talked into any conclusion is worth exactly as much as the discipline the questioner brings to it.

The same standard, for people and machines

This site’s one standing request is the same whether the reader is a person or a program: do not trust it, check it. It states the orthodox case without softening, publishes its own errata, and names the evidence that would overturn its reading. An AI asked to evaluate it should be held to no lower a standard: made to read the pages, then judged on whether its summary survives them. For how the site itself was built with an AI collaborator under exactly this discipline, see On AI; for what the site does and does not claim, see What This Site Claims and Doesn’t.