ChatGPT vs LorePanic: a six-question head-to-head on D&D 5e rules

The first time I showed LorePanic to my wife, she looked at me weird and said, "But Georges, you wasted your time! You could have just put the PDF in ChatGPT and gotten the same thing."

I was annoyed about it for a while, but I had to admit she had a point. ChatGPT will absolutely take a 350-page rulebook and answer questions out of it. So why had I spent so long building a search tool around the same documents?

My day job is software engineering, and I spend most of it inside Claude Code. I know what agentic AI does when you give it specialized tools and a set of text documents: it gets faster, its citations get more honest, and it handles the case where what you need is in one of fifteen documents instead of one. I wanted that experience for running D&D, where the document set sprawls across rulebooks, adventures, homebrew zines, and campaign notes, and where you need an answer very quickly, in the middle of a session. That instinct is why LorePanic exists.

But the instinct could still be wrong, and my wife had earned a real answer. So we ran the test: I threw six tricky 2024 D&D rules questions at ChatGPT with the SRD 5.2.1 PDF attached, and the same six at LorePanic's SRD search. Same source PDF, both graded against the 2024 wording. LorePanic got three right that ChatGPT missed, two were level, and for one we realized that the question itself was ambiguous, thanks to the answers. The bigger surprise, though, wasn't on the answer side. It was what happens when you click the source link.

The setup

The six questions were all the kind that show up while you're in panic mode in the middle of a session. Each one was chosen to land on a place where the 2024 rules differ from the 2014 ones in subtle ways, or where the answer hinges on a careful read of one spell's exact text.

1. If I turn an erinyes into a lamb, can I talk to it with Speak with Animals?
2. Can a Wild Shaped druid concentrate on a spell they cast before transforming?
3. If a warlock's Pact-of-the-Chain familiar delivers a touch spell, who provokes opportunity attacks?
4. Does Counterspell work on a spell cast as a reaction, like Shield or Counterspell itself?
5. Can a Paladin Divine Smite with an unarmed strike, a thrown weapon, or an improvised weapon?
6. If I'm grappled, can I cast Misty Step? Dimension Door? Thunder Step?

Where they tied

Two questions came out level. On Wild Shape concentration, both engines cited the SRD line that shape-shifting doesn't break concentration on already-cast spells, and both flagged the level-18 Beast Spells exception for casting new ones. On grappled plus teleport, both correctly walked through it: grappled sets speed to 0, teleport doesn't use speed, and once you're out of the grappler's reach the grapple ends. Identical answers, identical reasoning.

Where ChatGPT slipped

The three losses had the same shape. The 2024 rules changed in small wording-level ways that only show up if you read the new spell text. ChatGPT, even with the 2024 PDF attached, kept giving the 2014 answer. We can't prove why, but our best guess is that the model has read tens of thousands of forum posts and rules-blog archives from the 2014 era, and that's a strong prior to argue against from a single PDF attachment.

Polymorph and Speak with Animals

Turn an erinyes into a lamb, can you Speak with Animals to it? Depends on whether the transformation makes the target a Beast in stat-block terms. The 2024 Polymorph spell says the target retains its creature type, so the erinyes-as-lamb is still a Fiend, and Speak with Animals fails. The 2024 True Polymorph replaces the type, so it works (though the lamb will probably swear at you in lamb).

ChatGPT hedged with a generic conditional: "if the effect changes the type, yes; if it retains the type, no," without naming which 2024 spells fall on which side. Useful as theory, useless when a player has just cast Polymorph and you need a ruling. LorePanic named the actual spells and ruled correctly on each.

Divine Smite with thrown weapons

The 2024 Divine Smite text says it triggers on a hit with "a Melee weapon or an Unarmed Strike." The 2014 version got widely read as "a melee attack." Those are not the same thing. In 2024, "Melee weapon" is a weapon category, not a description of how the attack was made. A javelin is a Melee weapon with the Thrown property. Throwing it is a ranged attack made with a melee weapon. Per 2024 wording, that still triggers Smite.

ChatGPT ruled that thrown attacks don't qualify. That's the 2014 ruling, given confidently against the 2024 question, with a 2024 PDF sitting in the attachment slot. LorePanic ruled correctly: a thrown javelin or dagger still triggers Smite, because the weapon is a Melee weapon.

Counterspell on a reaction spell

Both engines got the headline right: yes, you can Counterspell a Counterspell. Both cited the SRD line about the reaction being wasted on a failed save. ChatGPT added a fair caveat about reactions only refreshing at the start of your next turn.

The 2024 wrinkle that actually decides what happens at the table is the One Spell with a Spell Slot per Turn rule. If you cast Fireball and it gets countered, you can't Counterspell their Counterspell on the same turn, because that would mean burning a second spell slot in the same turn. A teammate has to step in. If what got countered was a cantrip, the rule doesn't apply and you can fire back. ChatGPT didn't mention it. LorePanic did, with the cantrip-vs-leveled-spell split spelled out.

Where ChatGPT had a fair point

The familiar / opportunity attacks question came out as a draw, and the reason was on us: the question was ambiguous. "Who provokes, the familiar or the warlock?" can mean either "does the act of casting trigger an OA?" or "during the familiar's round-trip in and out of melee, who's exposed?" LorePanic answered the first reading. Nobody, in 2024, casting in melee isn't an OA trigger, only movement is. ChatGPT answered the second reading. The familiar, because it's the one moving in and out of reach. Both answers are correct under their respective interpretations. The prompt was the weak link, not either model.

The score isn't really the point

Three wins out of six is too small a sample to make claims off of, and we're not going to pretend otherwise. The thing that actually struck us during the test wasn't on the answer side. It was the citation step. When a model gives you a rules answer mid-session, you can't act on it until you've checked the source. How long that check takes determines whether the tool is usable at a real table.

In ChatGPT it goes like this. You ask the question, the model writes a confident paragraph, you click the citation because you don't trust an LLM blind, and this is what opens.

A 350-page PDF, opened to page 1, with no jump and no highlight. You're a GM, the wizard has already cast the spell, and three players are watching you decide. There is no version of this where you ctrl-F through the SRD to verify a paragraph the model wrote in three seconds. So you trust the answer, move on, and find out next session that you ruled it wrong.

In LorePanic, the same click does this instead.

One click and you're scrolled to the line, with the sentence highlighted. The answer is a paragraph; the proof is two lines below it; the rest of the SRD is right there when you want it. The verification step that takes two minutes in ChatGPT takes about four seconds here. That's the difference between a tool you check and a tool you skip checking, and we built LorePanic to be the first kind.

Why we built it this way

Two things drove the result. The first is that retrieval beats general training on rules that just changed. ChatGPT's misses in this test were almost all on 2024-specific wording: Polymorph keeping creature type, Divine Smite using "Melee weapon" instead of "melee attack," the one-spell-slot-per-turn rule. There are tens of thousands of forum posts and rules-blog pages from the 2014 era in the training set, and even with a 2024 PDF attached, that's a hard prior to overcome. Our retrieval pulls the actual 2024 paragraph and gives it to the model as the input it has to defer to. That keeps the answers honest about the wording that just changed.

Not every 2024 change is dramatic, though. We compared one rule against its 2014 text line by line in our guide to cover rules in D&D 5e: the mechanics barely moved, but one related change quietly matters.

The second is that the citation experience is half the product. A rules answer you can verify in four seconds is a different thing from a rules answer you can verify in two minutes. The first one you check, the second one you don't. The SRD viewer is built around making verification fast enough that you actually do it. The model accuracy will keep getting better on both sides, a year from now both engines will get the Smite question right, but the part that stays useful is whether you can trust the tool at game speed.

We tested this on the SRD because it's where a GM lives during a session, and because ChatGPT could plausibly compete on it once it had the PDF. The same retrieval and the same source-jump runs against any rulebook or adventure you upload, including the homebrew zine your friend wrote in Word in 2019. That's harder to demo in a head-to-head, because ChatGPT doesn't have your homebrew. It's also the reason most of our paying users are paying us.

Sources

Both sides of the test are public if you want to read the full answers and judge for yourself.

ChatGPT conversation (all six questions, with SRD 5.2.1 attached):

• chatgpt.com/share/69f9f476…

LorePanic answers, one per question:

• Erinyes turned into a lamb & Speak with Animals
• Wild Shaped druid concentrating on a spell
• Pact of the Chain familiar delivering a touch spell
• Counterspell on a reaction spell (first run)  ·  second run with the spell-slot rule
• Divine Smite with unarmed, thrown, or improvised weapons
• Misty Step / Dimension Door / Thunder Step while grappled