Sources

How to read this table

Each row is one constant or one claim used somewhere on the site. The grade is about the strength of the evidence behind it, not about how important it is or how confident we are in the answer.

The key is printed rather than summarised so you can disagree with a grade. If a row is graded wrongly, the grade is the thing to argue with, and it is meant to be arguable.

The qualifiers in that key have themselves been audited, and the audit is worth describing because its own first answer was wrong. "Read directly" is a claim rather than a description, so each one was checked against what it would take to establish. One source came back looking unread, on the grounds that its pages appeared to refuse automated requests, and it was regraded downward on that basis.

Then somebody checked the regrade. The page had not refused anything: two of its addresses were simply wrong and returned not-found, while the third returned the document and had done all along. So the source was read directly after all, it is graded primary, and the level it had been moved to has since been retired for want of anything to put in it.

That is the more useful kind of finding, because it is about the grading rather than about a number, and because the audit caught its own error rather than only somebody else's. A scale whose own labels have not been checked grades everything confidently and tells you nothing. And a downgrade is a claim like any other: refused and not-found look identical from a distance and mean opposite things, so grading a source down for the wrong reason is still grading it wrong.

The audit then went one level further down, and found the definition itself had a hole. "Read directly" was defined to exclude a search result, an abstract and an indexed excerpt. It did not exclude a page fetched by a tool that has a model read it and report what it says, which is a fourth thing, is none of those, and feels exactly like fetching. Every source claiming a direct read has since been re-checked against the raw bytes the publisher serves.

Six of them could not say, from their own notes, how they had been read. That is unaudited rather than wrong, and the distinction decided what to do about it: all six were resolved by GOING BACK AND READING THEM AGAIN rather than by relabelling them to something weaker. Relabelling would have been quicker, would have looked more cautious, and would have left six figures on this site whose provenance nobody had actually established.

One thing the grades deliberately do not tell you, and it is worth saying because it is the obvious next question. A grade describes where a number came from. It says nothing about how much any answer depends on it.

Those are separate axes and a low grade on a figure that barely moves a result is a smaller problem than a judgement that dominates one. This site found that out the hard way on the pool evaporation baseline, where the input with no source at all turned out to move the answer least of the three that were measured, and a judgement about how sheltered a garden is moved it nearly four times as far. The page had been pointing readers at the wrong assumption for exactly this reason.

So read a grade as provenance and nothing else. Where a figure has enough influence to be worth testing, the page that uses it says so and shows how far the answer moves across the range, which is a question this table cannot answer and does not try to.

What the distribution looks like

Publishing the counts matters as much as publishing the rows, because the shape of the evidence is the honest summary of what this site knows.

Those six add up: 3 plus 11 plus 2 plus 4 plus 4 plus 16 is 40. The sum is printed rather than left implied so that a drift breaks the addition in public. If the rows above ever stop reaching the total, one of them is stale and you can see it without counting anything yourself.

These counts describe the table below as at the review date at the top of this page, and the table is the authority. Where a number in this paragraph and a row in that table ever disagree, the table is right and this paragraph is stale. It has happened once already: the count of primary sources sat two behind the table for several days while more sources were read, which understated the evidence rather than overstating it, and nothing on the site noticed.

The biggest single group is E, at eleven, and that is the headline rather than something to bury. Between a quarter and a third of what this site ships is a judgement rather than a measurement.

Which is exactly why every one of those eleven renders as an editable input with a stated default rather than as a hidden constant. Here is what we do not know, and here is the field where you can put your own figure in and watch the answer move. A site that quietly hard-coded those eleven would look more authoritative and be worth less.

Every source, worst evidence first

The three grade F rows are not restated here, because they need an argument rather than a row. what we found when we tried to trace the figures everyone else uses sets out what those pages contain and what they do not contain.

What happened when we measured which of these actually matter

The grades above describe where each number came from. They say nothing about how much any answer depends on it, and this site found out the hard way that those two orderings can be almost opposite.

Every adjustable assumption was swept across its whole input range to see how far the answer moved. The result was not what anybody expected.

Read the first column against the third. The two assumptions with the strongest evidence behind them turned out to be the two that move an answer most, and the three with no evidence at all were not moving anything, because two were never wired up and the third offered a choice between identical options.

That is a rule obeyed and its purpose defeated. Our rule is that a figure which cannot be sourced is exposed as an editable input rather than hidden as a constant, so that a reader can substitute their own judgement. This site did that faithfully for the figures it could source, and left the ones it could not as controls that did nothing. The disclosure budget was being spent almost entirely on the numbers that least needed disclosing.

What changed as a result: the garden and pool allowances are now wired into the benchmark calculation, with their defaults still at zero so nothing is quietly inflated. The property type adjustment was removed altogether rather than wired, because making it work would have meant inventing a number for each property type, and no source for any of them exists.

One caution on the measurement itself, because it has a blind spot worth knowing. It reports how far a figure moves, so it reports nothing for the two constants that move a threshold rather than scale a result: the gauge resolution below which no rate of loss is reported, and the flow at which a pool pump is treated as running. Both matter a great deal and both score zero on this test. It was also one sweep, in one part of the code, so it is a finding rather than an audit.

What is not in the table

Three kinds of gap are tracked separately, because they are absences rather than weak evidence, and grading an absence would be a category error.

• Supplier rates. Every one is currently unverified and renders an on-page notice rather than a figure. We do not invent tariffs, so none has been invented.
• The sewerage charging basis for most suppliers, which is recorded as unknown rather than defaulted. Both of the plausible defaults are real values for real companies, so picking one would silently take a side.
• A handful of source URLs that could not be verified from the machine doing the checking. The records say they are unchecked and do not say why, which is deliberate: a refusal and a wrong address look identical from the outside and mean opposite things, and this site has already once graded a source down by guessing which it was. Unchecked is not evidence a page is missing.

A figure that is missing is visible on the page as missing. That is a deliberate trade: an obvious gap is more useful than a plausible number, and far easier to fix.

Common questions