AI Demonstration Testing Record

This record documents the saved demonstrations used to prepare the November 4, 2026 presentation: three plain and three structured responses from each of five accessible systems, plus one separately identified source-bound document task and related preparation examinations. It identifies the evidence considered, methods, observed differences and limitations. It does not estimate population error rates or rank vendors.

The test execution and source review were conducted through Codex under Sam Rogers' instructions. Browser products were operated in the shared Comet tabs; Qwen was invoked locally through Ollama. This memorandum was assembled after execution from retained records. It is an observational demonstration record, not a preregistered study.

The materials consist of the 30 original paired-response JSON records and the additional document-task record; Qwen stdout, stderr and metrics; selected browser/settings captures; the court documents and rule listed below; the separately scheduled Legal Graph references and supplied illustrations in §11; the claims register; and the independent deck evaluation. The original run hashes and reader-transcript hashes are in the exhibit schedule and structured inventory.

Each accessible product received the plain prompt three times and the structured prompt three times, with a fresh-chat start intended for each. Every actual response was retained; no response was regenerated to obtain a desired result. All 30 records have identical prompt bytes within each prompt condition. A composer fill that did not submit was checked before submission, not counted as a model run.

Plain prompt:

Structured prompt:

Recorded settings and search conditions are shown below. Browser timestamps mark response capture, not submission or generation duration. Qwen timestamps mark invocation start; its elapsed time includes thinking. UTC is retained in the inventory and Mountain Standard Time is shown in the reader exhibits.

The source review examined the named cases and central demonstration propositions. Existence, source support, quotation fidelity and pinpoint accuracy were considered separately. No exhaustive legal clearance of all general rule assertions or predictions in every response is represented. A statement that no verified case was available to a system is not evidence that no case exists.

Claude A1 and A2 located Garner but characterized unchecked signing as recklessness under RPC 3.3. Garner footnote 4 declines the requisite intent finding while the opinion imposes appellate Rule 40 sanctions. Claude P1 states that distinction correctly. Bases: Ex. CJ-CL-A1, L020; Ex. CJ-CL-P1, L042; CJ-SRC-01, ¶10 n.4 and ¶16.

Structured prompting did not assure verbatim quotation or correct attribution. Gemini P1 puts a paraphrase of Garner paragraph 8 in quotation marks; P2 and P3 misattribute paragraph 7's quoted proposition to Mata rather than Wadsworth. Basis: Exs. CJ-GM-P1 through P3 and CJ-SRC-01, ¶¶7–8. Claude P2 recognizes the intent distinction but attaches footnote 4 to the wrong paragraph.

ChatGPT's structured runs declined verified Utah case quotations under their stated search limitation. Local Qwen also did not identify Garner. Qwen P3 nevertheless supplies a false quotation attributed to Rule 8.4(b), whose published text concerns criminal acts reflecting adversely on fitness. Bases: Exs. CJ-CG-P1 through P3; CJ-QW-P3; CJ-SRC-05. Abstention on one part of an answer does not verify its other assertions.

Copilot's central Garner quotations were checked against the opinion. A2 and A3 reused the same conversation URL despite an observed empty New chat view, so their independence is uncertain. No success percentage is assigned. Each run's narrow observation is in the schedule; the absence of a listed error does not constitute approval of the complete answer.

The structured condition changed jurisdiction, dates, quotation/pinpoint instructions and permission to abstain together. It cannot isolate the causal effect of one instruction. Three repetitions at one seed do not establish performance across legal work; no statistical inference or cross-vendor ranking is supported.

Search availability, account context and model modes differed. ChatGPT changed from Instant at A1 to Medium for the remaining runs. Copilot used Auto with unidentified backend model. Account memory and personalization were not disabled. Fresh chats therefore do not prove absence of account-level influence. Browser sampling parameters and exact backend versions were not exposed or fixed across all systems.

Perplexity produced zero responses in this paired test because the controller did not expose the requested tab and one navigation was denied. The earlier Perplexity contradiction review reported by Sam is separately identified as CJ-REV-02; it is not a sixth system in the paired sample.

Source checking and independent current-treatment review remain distinct. Reporter page pinpoints were not independently verified. General rule assertions, completeness of unpublished-case retrieval and predictions about future outcomes were not exhaustively evaluated. Browser PNGs may show partial visible excerpts; the retained response text is the complete captured text.

The reader exhibits reproduce each captured browser response field or the separately extracted final Qwen answer. Qwen thinking and private conversation URLs are omitted from this reader edition; untouched original JSON/stdout/stderr remain retained locally with hashes. Logical line numbers are supplied for citation and are not court-page or browser-screen numbers.

The observations support checking material propositions against their sources and distinguishing source discovery from source interpretation. They do not establish that evidence instructions always improve correctness, that attached-source analysis cannot fail, or that any product is approved for client information. No human verification-time benefit was measured.

Citation form: AI Demonstration Testing Record, CJ-TR-20261104 (Nov. 4, 2026), ¶9, Ex. CJ-CL-A1, L020. Each linked exhibit contains the exact prompt, full reader transcript, timestamp, observed model label, provenance and hashes. The local original remains authoritative for omitted metadata.

Preparation examinations are separate from the 30 paired model responses and additional document task. Fresh command receipts and retrospective session observations are labeled separately. No undocumented query exports or review details are reconstructed.

After approval to restore a short useful-work example, one source-bound chronology trial was declared before submission. Claude Opus 5.5 Medium received all seven pages of the public Garner opinion as extracted text with added PDF-page labels. The task requested only petition filing, OSC hearing and opinion filing, with absent dates marked Not stated. Basis: Ex. CJ-DOC-01; source CJ-SRC-01; examination CJ-QA-07. This additional response is separate from the original 30 paired runs.

The saved table states Not stated for the petition filing date (p.2 ¶2), April 22, 2025 for the OSC hearing (p.1 ¶1; p.3 ¶4), and May 22, 2025 for the opinion filing (p.1 caption). Each row and the hearing quotation were checked against the opinion. The main slide retypesets those three original table rows for readability and quotes the checked source sentence. This single example does not establish speed savings, comparative superiority or complete docket-history coverage.

The retained observations support four practical conclusions: (a) finding a real source and reaching the broadly right answer does not establish correct supporting analysis (Claude A1/A2); (b) quotation and pinpoint requests aid inspection but do not verify themselves (Gemini P1–P3 and Claude P2); (c) an abstention on cases does not validate surrounding rule assertions (Qwen P3); and (d) differing search, mode and account conditions prevent a controlled vendor ranking. Bases and boundaries are in ¶¶9–18 and their cited exhibits. These conclusions are drawn from this sample, not population estimates.

The additional chronology example shows a bounded document-organization task producing a draft whose requested fields can be checked directly. An unknown date was preserved as unknown. Practical implication: separate discovery from source interpretation and organization, verify material claims against primary passages, and keep source limits visible. Agreement between models does not itself establish independent corroboration; multiple summaries of one opinion remain one underlying authority. No verification-time benefit was measured; providing a source does not remove the need to check the output.

H1, not comparatively tested: for matched legal-analysis tasks under fixed system conditions, evidence-oriented instructions and source-supplied analysis will reduce unsupported material propositions and human verification burden relative to an ordinary request, while preserving material qualifications. The chronology trial is one feasibility example, not evidence that this comparative hypothesis holds.

Candidate seed for the next test, not yet run:

Proposed conditions: A, the ordinary realistic assignment; B, the same assignment with primary-source links, exact supporting quotations/pinpoints and explicit gaps; C, the corresponding analysis against the supplied opinion only, with outside claims excluded. Hold model, mode, account context and tool availability fixed within each comparison and counterbalance order where feasible. Score common source-interpretation propositions across conditions; score discovery coverage separately for A/B, because C is not a test of legal-research completeness.

Choose four to six representative tasks before inspecting outputs and define source-based answer keys with qualified legal review where judgment is required. Include an important exception, extraction, authority comparison, a time-sensitive question and a question absent from supplied material. Record correctness of each material claim, source support, quotation and locator fidelity, omitted qualifications, justified abstention, human verification time and substantial corrections. Preserve every output. Failure to reduce errors or verification burden, or increased omissions, would count against H1. This appendix proposes future work; no comparative new batch has been conducted.

The Legal Graph addendum separates four evidentiary roles: court documents support legal propositions; retained CJ-TR exhibits support observations about model outputs; graph records organize source-linked relationships; and supplied screenshots illustrate the publisher’s architecture. Disclosure: Sam Rogers builds and maintains these resources through PAICE.work PBC. They are not independent corroboration of one another.

The published Mata proceeding and determination (CJ-LG-01/02, AIEL-2023-002) connect to the court order and to the Human Oversight category. Both preserve a verified field of 2026-09-01 and admission_status legacy-unreviewed. The graph-linked court PDF (CJ-LG-10) differs in file bytes from the original retained CJ-SRC-02 PDF, but all 43 pages of pdftotext -layout output match exactly. The legal example rests on the court opinion’s reasoning at pp.1–2 and sanctions at p.34, not on the graph’s editorial disposition label. No assertion is made that AI use alone caused sanctions or that the citation misconduct caused dismissal of the underlying action.

The Human Oversight page and category export (CJ-LG-03/04) provide an editorial framework for meaningful review, reviewer competence and intervention. The category export also retains legacy-unreviewed and null source-review fields. A topic anchor does not establish that a statute indexed in that category applied in Mata or Garner. No Garner name match was found in the retrieved published native corpus (CJ-LG-08, generated_at 2026-09-22); this is a bounded corpus observation, not a claim that no case exists. Garner remains cited directly to CJ-SRC-01.

PubLedge’s Utah coverage review (CJ-LG-05, #coverage-review) records an October 14, 2026 inventory comparison and identifies known records not indexed. It expressly distinguishes inventory comparison from source-document review and clause-level mapping. This passage illustrates transparent coverage limits; it does not certify completeness or the present legal status of each agreement. Obligation-First’s public overview (CJ-LG-06) supplies the schema explanation distinguishing a proceeding, allegation and determination, not a substantive rule of law.

Two images were supplied by Sam Rogers on November 4, 2026 (CJ-LG-IMG-01/02); their capture times are unknown. The landscape connection overview, image 2, is included unchanged in the optional deck appendix and linked to the exact publisher page (CJ-LG-12). The tall project-card image, image 1, is retained as an alternative in the source schedule. These are explanatory illustrations, not independent proof of source authenticity, legal accuracy or operational functionality.

Additional research comprised bounded public URL retrievals, published-field inspection, graph-link resolution and a comparison of the Mata PDF text. HTTP statuses, exact retrieval times, retained bytes and hashes are listed in the linked source schedule and manifest. The Justia Garner mirror returned 403 to direct HTTP; the court-authored text was accessible through the web research tool and the existing CJ-SRC-01 PDF remains retained. No blocked direct request was retried. Source consistency, current treatment, source admission and human verification dates remain separate. No exhaustive citator review, paid research call, model rerun or graph-owner source edit occurred. Hashes identify retained bytes; they do not authenticate legal authority.

At Sam’s request, a public-source current-treatment review was completed on November 4, 2026 for the cases relied on in the presentation, speaker notes and memorandum, plus contextual cases in the saved answers. Exact findings, dates, locators, source snapshots, selected executed queries and access limits are in the linked eight-case review. Later judicial citation is distinguished from subsequent appellate history, and a case’s own reversal of a lower judgment is distinguished from reversal of that case.

No adverse disposition was located that changes the narrow historical propositions used from Garner, Mata, Beus Gilbert, Lacey, Prisbrey or Wadsworth. This is a qualified search result, not proof that no later docket event exists. Garner and Mata remain historical examples; Rule 40 and Garner ¶10 n.4 remain distinct from a finding of RPC 3.3 intent. Prisbrey remains excluded as evidence of AI-caused sanctions because its opinion does not find AI use.

The review adds material procedural context. Mata ECF 55 separately dismisses the underlying claim as time-barred; that merits dismissal must not be attributed to the fabricated citations. Beus Gilbert ECF 386, September 14, 2026, addresses arbitration merits, not vacatur of ECF 385 sanctions. Lacey ECF 119 is a special-master order; after settlement notice, the action was dismissed without prejudice on July 24, 2025 (ECF 146) and with prejudice on September 18, 2025 (ECF 148). The reviewed dismissal does not address ECF 119; neither vacatur nor continued enforceability is inferred from silence.

Positive later treatment was inspected in court-authored documents: Whiting’s March 13, 2026 sanctions opinion p.6 cites Garner, Mata and Wadsworth; the TOV Realty / Kosel Equity order, 355 Conn. 902, 905, quotes Garner; and Watson v. Prestwich, 2026 UT App 122, ¶27 n.8, cites Prisbrey’s briefing standards. The TOV order is dated July 31, 2026 while mirror release metadata says August 4; those dates are not interchangeable. These citations do not establish affirmance or exhaustive current treatment.

Coverage limits are express. The official Utah disposition index linked reports only through March–April 2026 and the granted-order index showed entries through June 11, 2026. The reviewed May 2025–April 2026 reports contain no Garner/Kadince/Prisbrey name or relevant docket-number match, but cannot exclude later activity. CourtListener anonymous search required authentication; several direct mirror requests failed. Dated docket excerpts and negative public searches do not establish a complete PACER or commercial citator report. No paid retrieval, authentication change or source-admission renewal occurred.

Varghese and Royer remain labeled fabricated examples supported by the actual court records. Bingel is an unverified name in Qwen P3 and is not adopted as legal authority; a failed search is not a finding that it was fabricated. Original model JSON and transcript bytes are unchanged. The presentation is ready to use these sources as narrow historical examples with the recorded limits, not as unqualified certification of current good law.

Sam directed that every slide, including the appendix, be functionally complete before design handoff. The two original unrun Demo C placeholder slides were removed; the checked Garner chronology remains in the main sequence. The historical Mata source exercise is retained with an explicit label and a return-to-deck link. Stale preparation instructions were reconciled, source links point to retained documents where appropriate, and the final full-slide audit and integrity receipts are linked below. The future H1 study remains explicitly proposed, which is a completed hypothesis appendix rather than an unfinished demo.