The Sunday Test: Methodology

What a Sunday Test snapshot is

A Sunday Test snapshot is a catalog of what one Australian church publishes on its own website and public channels, mapped against the five questions of The Sunday Test. It is not a ranking, a rating, a score, or a review. Every claim on a snapshot page links back to the specific source page it was drawn from. Where a signal has no public source, that is stated plainly.

Sources we read

• The church's own website. We follow internal links up to depth 2, capped at 20 pages per church, prioritising pages with keywords covering the five questions.
• Sermon podcast RSS, if the church publishes one. The last 12 episodes' titles, dates, and short descriptions. No transcripts, no audio, no attribution to a preacher by name.
• YouTube livestream metadata, if the church has a channel. The last 4 livestream titles, dates, and durations from the channel's uploads playlist. No transcripts, no video content.

Sources we exclude

• Any page whose primary purpose is a leadership or staff biography. Pastor, elder, staff, and member names are stripped from the extraction at ingest.
• News commentary, safeguarding claims, or doctrinal-correctness assessments. Version 1 of the framework does not evaluate these.
• Anything behind a paywall, sign-in, or private-group boundary.

Extraction prompt

A single uniform prompt runs against every church in a cohort. It is version-controlled. Any change to the prompt bumps the extraction version and triggers re-extraction so cross-cohort comparability is preserved. The prompt is available on request from sundaytest@auschurches.com.au.

The prompt instructs the extraction model to phrase every claim in neutral present tense about the source, not the church. It caps direct quotes at 30 words per source page. It requires every claim to cite a specific source URL. It forbids rating, scoring, ranking, comparison to other churches, or characterisation of the church's overall quality.

Refresh cadence

Snapshots refresh at least once every 12 months, or on request when a church notifies us that material on its own site has changed significantly. Refresh runs the same prompt against the updated source material and produces a new pending snapshot for moderation.

Automated verification

Before a snapshot reaches moderation it passes an automated verification chain. Two blocking checks run first, on the source material rather than the output: whether enough usable pages were retrieved, and whether what was retrieved is real content rather than a JavaScript wall, a sign-in wall, or a parked domain. Failing either aborts the run, so no snapshot gets built from material we couldn't actually read.

After extraction, five blocking checks run on the output. Every claim's cited URL must be one of the pages actually fetched. Every claim is then re-read by a second, different model with fresh context and no sight of the original extraction, and asked whether the source supports the claim. Two further passes look for rating language and for named individuals. A failure on any of these holds the snapshot on a separate review surface instead of publishing it.

Separately, a set of hand-signed reference snapshots is replayed against the current prompt and model, and the output compared to the signed reference. Drift beyond a per-church tolerance is recorded and reviewed. This check is advisory rather than blocking: run-to-run variance on identical source material is high enough that a single low score is not reliable evidence of a regression, so it is read as a trend rather than used to hold an individual snapshot.

Verification results, current counts, and an explicit list of what this chain does not establish are published at /for-researchers.

Moderation policy

No snapshot publishes while any of the blocking checks above is flagged (the regression check is advisory and does not hold a snapshot on its own). Flagged snapshots go to a separate review surface, not to the publishing queue, and a human resolves each one. The editor is Vinay Koshy, and rejection requires a written reason. Rejected snapshots never publish. A snapshot can be re-extracted and re-submitted at any time.

Snapshots that pass every verification check cleanly may publish without individual editorial review. A random 20% of clean snapshots are still routed to the editor as a spot check, and every published snapshot records whether a human or the automated gate approved it.

Correction process

If you represent a catalogued church and want to correct, clarify, or reply to a snapshot, email sundaytest@auschurches.com.au. We respond within 7 days. Corrections publish inline on the per-church page, dated. If you want to verify the snapshot as accurate, we note the verification date at the top of the page.

Full takedown requests are honoured. If a church asks for its snapshot to be removed, the page comes down within 7 days of the request and the underlying record is marked withdrawn.

What snapshots are not

• Not a rating, score, ranking, or badge.
• Not a comparison to other churches.
• Not a doctrinal assessment.
• Not a safeguarding or governance audit.
• Not an endorsement, adoption, or vouching for the church's own content.
• Not a substitute for visiting a church.

Why the framework looks this way

The framework is designed so a snapshot page is trivially defensible as a factual catalog of what a church has itself published. That shape does the legal work: no named individuals, no long quotations, no rating language, transparent methodology, a real right of reply, and a clearly labelled non-endorsement stance. The full argument sits in the manifesto.