Presented November 4, 2026. A record of that day, not updated since. Not legal advice.
AI-generated illustration
Quick show of hands
Who's in the room
Used an AI tool this week?
Used one for client work?
Ever got an answer you didn't trust, or later found was wrong?
AI-generated illustration
One more hand
What worries you more about AI at work?
AI-generated illustration
That it replaces what we do
That it's confidently wrong in something we sign
Who's talking
Sam Rogers
Founder, Snap Synapse. I help people ask AI better questions, judge what comes back, and keep their own judgment in charge.
PAICE.work PBC: the Legal Graph connects AI laws, obligations and incidents. So I read what AI gets wrong about law, and what courts do about it.
Past expert witness work. Current research: AI collaboration, what I call Aggregated Intelligence.
Not a lawyer. Nothing today is legal advice.
Why I'm here
No pitch today.
A Salt Lake City law office made this happen, and I'm grateful.
Nothing for sale here, no sign-up sheet, and the same four checks apply to every tool on these slides.
What I get out of it: your questions, in your words. They shape what I teach next.
By the end: one thing to try, and one check to run.
The card is for the question you'd rather not ask out loud, or the one thing about AI at work that worries you. No names, no client details. I'll collect them before the questions.
What to question
Wrong answers look exactly like right ones.
AI-generated illustration
Demo
A narrow Utah question
A general chatbot
Asked of a general chatbot, the way most people use one.
Testing record: CJ-TR-20261104, ¶¶4–8; Exs. CJ-CL-A1–A3. All 30 runs are indexed in the appendix.
Saved run |
What came back, and what we found
The answer, as given“Why the defense fails: Rule 40 is the appellate counterpart to Civil Rule 11. Your signature certifies the filing, so who drafted it is irrelevant. The court also cited Utah RPC 3.3, which bars a lawyer from knowingly or recklessly making a false statement of fact or law to a tribunal. Signing unchecked citations is the reckless part. And under RPC 5.3, a lawyer supervising nonlawyer staff is responsible for their work […]”
Garner, page 5 footnote 4, rendered from the opinion. The court expressly declines to find the intent Rule 3.3 requires. Real case; misread source.
ABeus Gilbert PLLC v. Brigham Young UniversityNo. 2:12-cv-00970-TS, ECF 385 (D. Utah Sept. 9, 2026)
BVarghese v. China Southern Airlines Co., Ltd.925 F.3d 1339 (11th Cir. 2019)
One of these is real. Which one?
What to try
Same tool. Ask differently.
AI-generated illustration
Demo | worth a photo
Ask for evidence, not just an answer
1 Jurisdiction2 Dates3 Quote and pinpoint4 Say so if not found
A general chatbot
CJ-TR-20261104, ¶¶6–8, 13; Exs. CJ-CL-P1–P3. Evidence instructions were tested as one bundle.
Saved run |
What came back this time
Asked for quotes and pinpoints“No Rule 3.3 finding. In n.4 (attached to ¶10), the court found that counsel made a false statement of law, but “we find that their conduct fell short of the level of intent required by the rule.” Rule 3.3 requires that the false statement be made knowingly or recklessly.”
The quoted passage, checked in the opinion. Same footnote as the last run; this time the answer quotes it and gets the distinction right.
Public opinion supplied. Three referenced rows returned. Every row checked.
Event
Date
Source in the opinion
Petition for interlocutory appeal filed
Not stated
PDF p. 2, ¶2
Order-to-show-cause hearing
April 22, 2025
PDF p. 1, ¶1; PDF p. 3, ¶4
This opinion filed
May 22, 2025
PDF p. 1, caption (no paragraph number)
“On April 22, 2025, Petitioner’s counsel appeared at the OSC hearing along with Petitioner.”
Open Garner, PDF p. 3, ¶4. The petition's filing date stays “Not stated.”
Saved Claude table, retypeset without changing its cells. Opus 5.5 Medium, November 4, 2026. Ex. CJ-DOC-01; CJ-TR-20261104, ¶¶21–24; source opinion (see testing record materials).
What to give it, what to keep out
Before anything goes in
Give it
The document, if it's cleared to share
The jurisdiction
The date or time window
The task, and the form you want back: quotes and pinpoints
Keep out
Client information, until the firm has:
approved the tool and the workflow
checked data handling and security
any client consent required
A local model needs the same checks
Running it locally changes who could see the data. It doesn't replace the firm's checks, or yours.
Where the data settings live
Consumer account or firm account?
Claude Max
Settings → Privacy → Help improve our AI models “Allow the use of your chats and coding sessions to train and improve Anthropic AI models.”
Off in this account
ChatGPT Pro
Settings → Data controls → Improve the model for everyone
Off in this account
These are training controls. Retention, access and the firm's approval are separate checks. Setting names as captured November 4, 2026; vendors rename them.
Worked example: Mata, ECF 54, p. 19, findings 45-46 (excerpt). A 2023 federal order doesn't establish current model performance or controlling Utah law.
Check 4 of 4
Own it.
Would you sign it? If not, it's a lead, not an answer.
Everything that I do, I gotta own. I am accountable for that.
Michael SimonAttorney and owner, Law Plus Data LLC Chief Strategy Officer, Bells Up AI October 2026
Worked example: Mata, ECF 54, p. 34, conclusion (b)-(d) (excerpt). After a finding of bad faith: letters to the client and each judge falsely named, and a $5,000 penalty, joint and several.
Asked for an answer“[…] The court also cited Utah RPC 3.3, which bars a lawyer from knowingly or recklessly making a false statement of fact or law to a tribunal. Signing unchecked citations is the reckless part. And under RPC 5.3, a lawyer supervising nonlawyer staff is responsible for their work, so “my clerk did it without my knowledge” is less a defense than an admission of a supervision failure. […]”
Asked for evidence“No Rule 3.3 finding. In n.4 (attached to ¶10), the court found that counsel made a false statement of law, but “we find that their conduct fell short of the level of intent required by the rule.” Rule 3.3 requires that the false statement be made knowingly or recklessly.”
Invitation copy and the promises this deck is checked against.
Sam confirmed the description before it went out.
Oct 18
Sam → Claude
First run of show. Sam picked the demos.
Sam's choices, recorded in the vault note.
Nov 3
Claude → deck
One offline page, 27 slides, plus a claims register: every checkable claim with a status and a source.
Script syntax, unique ids, every local link resolves.
Nov 3
Claude → Codex
The deck, by SHA-256, the run of show and the kit. Codex wrote its verdict before reading the author's own.
Five ranked findings, each with a line number. Only the findings file was written.
Nov 3
Codex → Claude, at Sam's go
Corrections applied; declined items named with the reason.
Each change logged against a register row.
Nov 3
Sam → Perplexity → Claude
Five searches written to find contradictions. Claude opened every cited source.
No row changed without Sam's decision. Differences recorded, not fixed silently.
Nov 4
Codex → test record
30 paired runs in five systems, one checked chronology, memorandum CJ-TR-20261104 with exhibits and hashes, eleven examinations.
Preflight exit 0; every slide checked at native size; a written handoff for the next model.
Nov 4
Sam → Claude Fable 5.1 → Claude Sonnet 5.5
32 numbered slide notes; design pass v1.4; then a read-only audit by a second model.
Preflight exit 0; quotes verbatim against the exhibits; every link resolves; CJ-QA-12.
Nov 4
Claude → Sam
Timed rehearsal, printing, HDMI, the talk itself.
Still Sam's. No model can rehearse for him.
Records: run of show and claims register notes, CJ-TR-20261104, CJ-QA-12, `review/codex-evaluation-2026-11-03.md`. Versions given where the record names them; otherwise the product name stands.
Appendix | How this deck was made · 2 of 3
What the checking caught
Between models
Codex: the first keep-out rule read as permission. Under Rule 1.6, a license or owned hardware isn't an exception.
Codex: the slide meant to show a $5,000 sanction had cropped it out of view.
Codex: a note called a model "released today"; its own posts were dated October 2026.
Claude: a prep-kit preview labeled a newspaper the "primary source" for a court case. Cut.
Claude: Codex flagged "courts elsewhere cite Garner" as unverified; opening both later opinions confirmed it, so it stayed.
A script: Claude's own chat summary of the register miscounted three status totals.
On the day
Codex: two of three plain Claude answers misread Garner's footnote 4. Real case, wrong intent finding. That became the demo.
Sam: the saved-run crops were 63 to 148 pixels tall, unreadable on a projector. Retypeset verbatim, exhibit cited, crops kept and linked.
Sam: the real case on the real-or-fake card was the one the previous slide had just named. Swapped for Beus Gilbert.
Claude Sonnet 5.5, auditing Fable 5.1: every retypeset quote matched its exhibit; one AI illustration had no label. Labeled.
Third pass, Perplexity, run by Sam: five searches looking for contradictions found none. One new Utah sanction (Beus Gilbert v. BYU, D. Utah, September 2026), confirmed in the court's own docket. Of 107 sources returned, at least 13 were unrelated keyword matches. A citation in a list isn't support.
The register distinguishes primary sources, issuers' pages, my own facts, demo-dependent claims and open questions. Test record and other examinations: CJ-TR-20261104, §§1–7.
Appendix | How this deck was made · 3 of 3
Every handoff carries an artifact and a check
Drawn from the records on the previous two slides. Editable source: `review/deck-workflow-2026-11-04.mmd` (Mermaid). Dark boxes outline the builder, grey the checker.
The real-or-fake card: Varghese v. China Southern Airlines Co., 925 F.3d 1339 (11th Cir. 2019) is one of the nonexistent decisions in Mata (ECF 54, p. 7). It does not exist.
T opens this index. Arrow keys tour the appendix; Escape returns to the main slide.
Appendix | Testing record · CJ-TR-20261104
Instructions, materials and method
Three plain and three structured responses per accessible system. Every actual answer retained; none regenerated to obtain a desired result.
Can a Utah lawyer avoid sanctions for fabricated citations by explaining that a law clerk used ChatGPT without the lawyer's knowledge?
Structured condition added Utah state courts, 2023 to November 4, 2026, full citations, quoted support with pinpoints, and permission to report no verified case.
Materials: original records, selected captures, court opinions, rule text and the claims register. The complete prompts appear in every exhibit.
CJ-TR-20261104, ¶¶3–8. Findings are proposition-specific, not complete legal clearance.
Appendix | Testing record · CJ-TR-20261104
Conditions of examination
System
Observed mode
Search / material limitation
Claude
Opus 5.5 Medium
Visible web search
ChatGPT
A1 Instant; remaining runs Medium
All responses said web search unavailable
Gemini
Flash High
Visible web search
Copilot
Auto; backend model unidentified
Visible search/source results
Qwen
qwen3.8:latest; 27.3B Q4_K_M; medium thinking
No web search tools
Perplexity: zero paired-test runs, access blocked. Account memory was not disabled. Copilot A2/A3 shared a conversation URL.
CJ-TR-20261104, ¶¶7, 13–17. Browser timestamps are captures; Qwen timestamps are invocation starts. Exhibit times are UTC and MST.
Intent distinction and footnote locator recognized.
CJ-TR-20261104, §6. Click an exhibit for the exact prompt, complete transcript, line references and hashes. A narrow observation is not approval of every assertion.
Appendix | Testing record · CJ-TR-20261104
ChatGPT: complete run schedule
A1 Instant; remaining runs Medium · All responses said web search unavailable
CJ-TR-20261104, §6. Click an exhibit for the exact prompt, complete transcript, line references and hashes. A narrow observation is not approval of every assertion.
Same attribution error; paragraph 10 text stitched.
CJ-TR-20261104, §6. Click an exhibit for the exact prompt, complete transcript, line references and hashes. A narrow observation is not approval of every assertion.
Appendix | Testing record · CJ-TR-20261104
Copilot: complete run schedule
Auto; backend model unidentified · Visible search/source results
Central Garner quotations checked against opinion.
CJ-TR-20261104, §6. Click an exhibit for the exact prompt, complete transcript, line references and hashes. A narrow observation is not approval of every assertion.
Appendix | Testing record · CJ-TR-20261104
Qwen: complete run schedule
qwen3.8:latest; 27.3B Q4_K_M; medium thinking · No web search tools
No verified Utah case; false Rule 8.4(b) quotation.
CJ-TR-20261104, §6. Click an exhibit for the exact prompt, complete transcript, line references and hashes. A narrow observation is not approval of every assertion.
Appendix | Testing record · CJ-TR-20261104
Limits of the observations
One seed, three repetitions per condition. No population accuracy estimate or vendor ranking.
The structured prompt changed several instructions together. It does not isolate which instruction helped.
Search, mode and account context differed. Fresh-chat starts do not establish independent account context.
Core propositions were checked; every surrounding assertion was not exhaustively graded.
No human verification-time benefit was measured. An abstention does not verify the rest of an answer.
CJ-TR-20261104, ¶¶13–18. Observations and interpretations remain distinct from the verbatim exhibits.
H1, not comparatively tested: evidence instructions and supplied-source analysis will reduce unsupported material claims and human verification burden while preserving important qualifications.
A: ordinary realistic research assignment.
B: the same assignment, with primary-source support, exact quotes, pinpoints and explicit gaps.
C: corresponding analysis using the supplied opinion only.
Keep system conditions fixed. Compare common interpretation claims; assess discovery coverage separately for A/B. One successful chronology extraction does not establish H1.
Hypothetical: In a Utah state appeal, a lawyer signs a petition containing invented case citations. An unlicensed clerk used ChatGPT without the lawyer's knowledge. A second lawyer is counsel of record but neither drafted nor signed the petition. Prepare a 250-word research note for counsel responding to the court's order to show cause. Identify the most relevant Utah state decisions, arguments that could limit each lawyer's exposure, adverse authority, and remedies the cases actually imposed.
Choose four to six tasks and source-based answer keys before running. Retain every output.
Measure material-claim correctness, source support, quote/pinpoint fidelity, omitted qualifications, justified abstention, human checking time and substantial corrections. More errors, omissions or checking burden count against H1.
Dated inventory comparison and explicit missing records.
Oct 14, 2026 comparison; not renewed legal review.
Sam Rogers maintains these resources through PAICE.work PBC. The graph organizes evidence; it is not independent corroboration. Garner stays directly cited to its opinion.
Sam-supplied illustration, Nov 4, 2026; capture time unknown. CJ-LG-IMG-02 · Evidence map and limits. Each project retains responsibility for its sources and judgments.Appendix | Public-source review · November 4, 2026
Case-treatment review
Case
History or later treatment found
Use in this deck
Garner
Whiting and TOV/Kosel cite it in 2026. Rule 40 / RPC 3.3 distinction preserved.
Historical Utah example.
Mata
Later cited by Whiting and Beus Gilbert. ECF 55 separately dismisses the claim as untimely.
2023 record; bad-faith qualification.
Beus Gilbert
Sept 14 ECF 386 concerns the arbitration merits, not vacatur of Sept 9 sanctions.
Dated sanctions order.
Lacey
Special-master order; later settlement and Sept 18, 2025 dismissal with prejudice.
Context only; reviewed dismissal does not address sanctions.
Prisbrey
Watson cites its briefing standards in August 2026. No AI-use finding in Prisbrey.
Excluded as AI-sanctions evidence.
Wadsworth
Whiting cites it; Garner ¶7 is the source of the attribution check.
Context and quotation attribution.
No adverse disposition located that changes the narrow propositions used. Public docket and certiorari-index coverage is incomplete; no commercial citator clearance is claimed.
Michael SimonAttorney and owner, Law Plus Data LLC
Checked like we'd check anything else
