Trust & Security

We run our scanner on our own codebase across five core profiles and publish the result whether or not it flatters us: 93 pass, 0 block, 9 warn across 102 checks — score 96, grade A. On 2 August this page read 84, grade B, across 91 checks. The code did not get worse; we added 20 more checks that inspect code and configuration rather than documents, and seven of them found controls we describe in writing and have not built. Read the reason before the grade. Only one of those seven was a control we lacked: data subject rights, which we built on 4 August, the day after the check found it missing. Four of the other six were faults in our own checks — they were reporting gaps in controls we already had. Our network segmentation check could not match the ordinary Docker Compose form. Our model card check ignored the folder model cards live in. Our access control check did not recognise the authorisation model of the web framework we are written in. And our cookie consent check demanded a consent banner from a site that sets no tracking cookie. Every one of those faults was reaching customers too, not just us. The grade moved because the measurement got more accurate, not because the code got safer — the same reason it moved down two days ago, running the other way. This is the second time deepening our own measurement has cost us a grade, and the trade is the right one: a score that only moves when the code improves is worth more than one that flatters the scanner. On 18 August it moved again, 82 to 76, and this time we did it to ourselves on purpose. We built a corpus of deliberately planted credentials and NRIC values — the samples our detection checks must fire on — and our own scanner duly found them. Five of the eleven warnings are that test data. We could have excluded the directory and kept the higher number; we downgrade it to a warning instead, because a planted NRIC is still an NRIC sitting in the repository, and a scanner that cannot see its own fixtures cannot be trusted to see yours. Later the same day it went back to 82, and that time the code really did get better. Two of the warnings were controls we genuinely did not have. We log every AI inference this product makes — the prompt digest, the answer, and the decision it drove — into a hash-chained log, because NAIO 2026 requires AI output to be reviewable afterwards and we were asking customers for that while keeping nothing ourselves. And our application logs now ship to a collector that runs beside the app with no credentials and no published port, so an attacker who reaches the API cannot erase the record of reaching it; the capability had existed since 2 August and was switched off, because there was nothing on the other end. Two more were faults in our own checks, both of which had been reaching customers: our AI-logging check knew the name model_output_log and not ai_output_log, and our HTTPS check could not tell a router at 10.0.0.1 from the open internet. It also found one real thing we would rather it had not: our timestamp authority spoke plain HTTP, which meant the digest of a customer's assessment crossed the network in clear. We moved to a TSA that offers TLS. The nine warnings that remain are all low confidence, and eight of them are our own test fixtures and rule definitions — the scanner seeing the planted evidence it is supposed to see. On 19 August we changed a scoring rule, and it moved our own grade from B to A. You should read that sentence sceptically, so here is everything you need to. The findings did not change: the same 93 passes, the same nine warnings, the same nine files. What changed is that a low-confidence warning now costs one point instead of two. Every detection check already weighs its evidence and names the reason — a credential inside a rule definition is reported as a rule definition, not as a leak — and the score was ignoring all of it, charging a finding we had ourselves labelled 'this is your own test fixture' at the same rate as a real one. The report and the number were arguing, and only the report was right. Half rather than nothing, because a planted NRIC is still an NRIC sitting in a repository. A BLOCK is not discounted at all, at any confidence: if we are not sure enough to charge full price it should have been weighed down to a warning first, and deployment must stop, probably is not a finding. Under the previous weighting this same scan was 82, grade B; both numbers are printed here on purpose. The change applies to every customer's report identically, and it is the reason a codebase with a serious test suite is no longer punished for having one. It went unnoticed for eighteen months because there was no test on the scoring function at all — there is now, and it asserts this page's figure, so if the two ever drift apart a test fails rather than a claim quietly goes stale. Then we asked the obvious follow-up question, and the answer was worse than the original problem. If we score ourselves an A, can anything else? We scanned eighteen real codebases to find out — our own nine products, and nine of the most widely deployed Python packages there are, including fastapi, requests, sqlalchemy and pydantic. Seventeen of the eighteen scored exactly zero and grade F. The only exception was AuditCore. Five BLOCK findings reached zero under the old arithmetic and every real codebase has more than five, so the scale had no resolution at all — it could not tell eight failures from eighteen. That is not a grading scale, it is a test of whether a project is AuditCore, and no adjustment of the band boundaries could have fixed it because there was nothing between 0 and 91 to band. The score is now the proportion of checks passed, which spans 27% to 96% across that set, with severity setting a ceiling the proportion cannot lift: any BLOCK caps the grade at C, five or more at D, because passing 95% of checks while missing a statutory requirement is not an A. Our own figure moved from 91 to 96 and stayed grade A; the codebases that used to be indistinguishable at zero now land between F and C, in an order you can argue with. The full data, the method, and four limits we could not resolve — including that we are the only project in the set that had configured the scanner, and so the only one whose findings were weighed down — are published in docs/CALIBRATION_20260819.md.