How to check a paper for hallucinated references before you submit

A hallucinated reference is an entry in your reference list that points to a work that does not exist, or that mixes real and invented details so that it no longer identifies one real work: invented authors on a real title, a real DOI attached to a different paper, a venue that never existed. To check a paper before you submit, resolve every identifier in the registry that issued it (a DOI at doi.org, an arXiv ID on arXiv, a PMID in PubMed, an ISBN in a library catalogue) and compare the title, first author, year and venue with what the registry lists. For any reference without an identifier, search for its exact title. Then open each source and confirm it says what you cite it for, because no registry lookup can tell you that. The stakes went up in 2026: arXiv's computer science chair announced a one-year ban for papers with hallucinated references or other "incontrovertible evidence" of unchecked LLM output, ICLR desk-rejected papers that contained them, and ACL sent more than 100 accepted papers back to resubmission.

This guide covers what counts as a hallucinated reference, how common they are, what arXiv, conferences and journals do about them, how to check references by hand, which free checkers exist and what they miss, and how to work with an AI app without letting it invent sources. Every policy, figure and tool below was checked on 19 October 2026 on the source's own page, paper or repository; the sources list notes where we read an archived or proxied copy instead. Policies and tools in this area change quickly, so check the linked page before you rely on a detail.

What counts as a hallucinated reference?#

A reference is hallucinated when its identity fails: no matching work exists, or its details point to a different work from the one written down. Samar Ansari's taxonomy of 100 fabricated citations found in NeurIPS 2025 papers sorts them into five failure modes:

• Total fabrication. "The authors, title, venue, and identifiers (DOI, arXiv ID, URL) are all fabricated."
• Partial attribute corruption. Real and invented details are blended, for example "real authors attributed to papers they did not write", "real paper titles with fabricated authors" or "correct venues with wrong publication years".
• Identifier hijacking. A valid DOI, arXiv ID or URL "points to a real paper, but the authors, title, or other metadata do not match the paper at that identifier".
• Placeholder hallucination. Leftovers such as "Firstname Lastname" or "arXiv:2305.XXXX".
• Semantic hallucination. A plausible title that "fits the research domain conceptually but does not correspond to any real work".

In Ansari's sample, total fabrication was the most common primary type (66%), and every one of the 100 citations showed more than one failure mode at once. GPTZero's table of NeurIPS 2025 examples shows what this looks like in print. One accepted paper cited a real title under the authors "John Doe and Jane Smith", and GPTZero's note says the "arXiv ID links to a different article". Another listed "Firstname Lastname and Others", with "URL or arXiv ID to be updated."

Conferences draw the line in a similar place. ICML 2026 told its area chairs to follow ICLR 2026 guidance (area chair instructions). A reference counts as hallucinated if you cannot find "any document or publication with a title anywhere near the one provided", if the author list includes "added authors that are not actually authors of that publication", or if "there is no evidence that the reference's venue exists". The same guidance says several common errors do not make a reference hallucinated, though authors should still fix them: minor differences in the title, a few misspelled or missing authors, an author list swapped with another paper's ("a common BibTeX error") and a real venue that is the wrong one.

A further problem sits outside these definitions: a real paper, correctly listed, that does not support the sentence that cites it. On 3 October 2026 the publisher of Discover Artificial Intelligence retracted a 2024 review because "A number of references in this article appear to be contextually incorrect, which may suggest generative AI was used in writing this manuscript" (retraction note). The note records that one of the two authors "disagrees with this retraction". A registry lookup cannot catch this kind of error. Only reading the source can.

How common are hallucinated references?#

They are rare per reference but no longer rare per paper in some fields, and every count depends on how its authors defined the problem. The main figures, each with its definition:

• NeurIPS 2025. GPTZero scanned "4841 of the 5290 papers accepted" and published 100 hallucinated citations, each "verified by a human expert" (GPTZero, 21 January 2026). The post gives the number of papers as 51 in its introduction and 53 in its table title; Ansari's paper uses 53, "approx. 1% of all accepted papers". Asked by Fortune, the NeurIPS board said: "Even if 1.1% of the papers have one or more incorrect references due to the use of LLMs, the content of the papers themselves are not necessarily invalidated" (Fortune).
• ICLR 2026 submissions. GPTZero scanned a sample of 300 papers under review. Its tool flagged 90, and "Following human verification, we determined that 50 papers included at least one actual hallucitation" (GPTZero, December 2025).
• Four conferences, several years. Phantom References by Mark Russinovich, Ram Shankar Siva Kumar and Ahmed Salem checked 48,095 accepted papers and 2,614,992 references from ICLR, ICML, NeurIPS and USENIX Security. They counted only "non-existent works and substantial author-list mismatches", excluding venue, year and publication-status drift. Reference-level rates were "usually below 1%", but "in 2025, roughly one in twenty NeurIPS and USENIX Security papers contains at least two likely hallucinated academic-paper-like references under our strict definition". They also found likely hallucinated citations "even among award-winning papers".
• Preprint servers. Zhenyue Zhao and colleagues audited "111 million references across 2.5 million papers in arXiv, bioRxiv, SSRN, and PubMed Central" and give "a conservative estimate of 146,932 hallucinated citations in 2025 alone" (arXiv 2605.07723).
• Biomedicine. An audit led by Maxim Topaz at Columbia University, published in The Lancet on 7 May 2026, checked papers in PubMed Central's open-access set and found 4,046 fake citations among 97.1 million verified references, spread across 2,810 papers (Columbia Nursing). Retraction Watch reports the rate as about one in 277 papers in the first seven weeks of 2026, up from one in 458 in 2025 and one in 2,828 in 2023.
• Fake references from ChatGPT itself. In a 2023 study that asked ChatGPT for short literature reviews, Walters and Wilder found that "55% of the GPT-3.5 citations but just 18% of the GPT-4 citations are fabricated" (Scientific Reports). Those models are no longer current, so the study shows the failure, not today's rate.

These figures cannot be compared with one another. They use different corpora, definitions and verification methods, and some count likely rather than confirmed cases. The Lancet audit has also drawn method criticism: Cochrane's Ella Flemyng told Retraction Watch "we are lacking considerable details about the methods".

What do arXiv, conferences and journals do about hallucinated references?#

arXiv's computer science chair has announced a one-year ban for them, ICLR and ACL have rejected papers over them, and journals have retracted articles. Each rule below is quoted from the venue's own page or, for arXiv, from its section chair's announcement.

The arXiv penalty was announced by Thomas Dietterich, then chair of arXiv's computing section (Times Higher Education), in that thread: "If a submission contains incontrovertible evidence that the authors did not check the results of LLM generation, this means we can't trust anything in the paper." His examples were "hallucinated references, meta-comments from the LLM". arXiv's moderation page, as of 19 October 2026, sets out the underlying rule, that "incorrect references" produced by generative AI are "the responsibility of the author(s)", but does not state the penalty.

Journal rules sit in their author instructions. JAMA's instructions for authors say "AI, LLM, and chatbots should not be used to generate or format references", recommend standard reference managers instead and ask authors to "Check all references for completeness, accuracy, and integrity" (JAMA; our guide to citing AI covers JAMA's other AI rules). A Taylor & Francis spokesperson told Retraction Watch that articles with concerning references go back to the author, and "If these account for more than a small proportion of citations and/or substantially impact the overall integrity of the manuscript, the submission will usually be rejected." Springer Nature's Journal of Academic Ethics retracted an article in a note dated 2 January 2026 because "The majority of the references of this article appear to be non-existent" (retraction note). A month earlier, its corresponding author had told Retraction Watch that he and his co-authors "used ChatGPT to generate the references".

How do you check references by hand?#

Resolve each identifier in its own registry, compare four fields, search titles for the rest, and then read the source. All of the routes below are free.

1. Check the final list. Do this after your last formatting pass. When the Phantom References authors wrote to the authors of papers with at least five likely hallucinated references, several replied that references "were generated or reformatted using LLM-based tools", in some cases "during the camera-ready stage rather than the original submission".
2. DOIs. Paste each DOI into doi.org. If it does not resolve, the DOI Foundation's "Which RA?" service tells you which registration agency holds it: append the DOI to https://doi.org/doiRA/, and the possible answers include "DOI does not exist" (DOI Foundation). If it does resolve, compare the title, first author, year and venue with the landing page. A DOI that opens a different paper is identifier hijacking, not a typo to ignore.
3. arXiv IDs. Open arxiv.org/abs/ followed by the ID, and check the version you mean, written as in "arXiv:0706.0001v2" (arXiv). Every arXiv paper also has a DataCite DOI in the form https://doi.org/10.48550/arXiv.2202.01037 (arXiv DOI help). Authors can add a journal reference or DOI once the paper is published (arXiv), so the abstract page may show the published version too.
4. PMIDs and ISBNs. Look up a PMID in PubMed. For a book, Open Library resolves addresses of the form https://openlibrary.org/isbn/9780140328721 (Open Library); a library catalogue covers what it lacks.
5. Many identifiers at once. Zotero's Add Item by Identifier takes "ISBN, DOI, PubMed ID, arXiv ID, or ADS Bibcode" and fetches the metadata from "Library of Congress, WorldCat, and other catalogs for ISBNs, CrossRef and other registries for DOIs, NCBI PubMed for PubMed IDs, arXiv.org for arXiv IDs". Compare the item it creates with your entry. Crossref's Simple Text Query matches a pasted reference list to Crossref DOIs, with "space for 1,000 references per submission" (Crossref). Its own tip applies to every tool here: "To check the accuracy of a reference, click on the link that is returned with the reference."
6. No identifier. Search the exact title in Crossref Metadata Search, DataCite Commons, DBLP for computer science, OpenAlex, Semantic Scholar, Google Scholar or the ACL Anthology. Apply the conference rule: a title "anywhere near" the cited one is a lead to check, and nothing near it is a strong warning. When you find the work, check the author list and the venue as well.
7. Web pages, software and reports. Enter the address in the Wayback Machine to see archived copies. The date is in the archived address as "yyyymmddhhmmss", so you can cite the copy that matches the date you read it. A page that was never archived may still have existed.
8. Open the source. Find the passage that supports your sentence. This step catches the "contextually incorrect" references that a registry cannot.

Watch the difference between preprint and published versions. Decide which version you read, and make the year, venue and identifier match that version. Phantom References treated venue, year and publication-status differences as "ordinary bibliographic drift", not hallucination, and the ICML guidance asks authors to fix a wrong venue, but an automated checker may still flag the mismatch.

Which free hallucinated reference checkers can you use?#

Several open-source checkers extract your references and look them up in scholarly databases. All of them produce flags for you to review, not verdicts. These are the ones we could open on 19 October 2026:

Two more exist but are not free click-and-check tools. CiteAudit, a research pipeline from arXiv 2602.23452, combines "URL probing, Google Search, Google Scholar, and an LLM judge", and you "need accounts on Google AI Studio (Gemini), OpenAI, and SerpApi" to run it. GPTZero's Hallucination Detector is a commercial service with free sign-up; its plan limits were not readable on its pricing page when we checked. GPTZero says its tool flags "any citations in a document that can't be found online" and that "These flagged citations are not automatically hallucinations".

An independent test by Fidan Badalova and Philipp Mayr of GESIS shows the trade-offs (arXiv 2607.22693, July 2026). They ran five tools on "a manually verified test set of 104 references from three scholarly documents, including 33 problematic references":

In the authors' words, RefChecker and CheckIfExist "identified more problematic references, but they also produced many false positives", while HalluCiteChecker "generated fewer false positives but missed more problematic references". They trace the weaknesses to "reference extraction errors, incomplete metadata, limited database coverage, and inconsistent verification results", and conclude that "the output of these tools should not be treated as a final decision". The test set was small, and several of the tools have been updated since, so treat the table as a picture of trade-offs, not a ranking.

What can a hallucinated reference checker not tell you?#

A checker cannot tell you whether a source supports your sentence, and it cannot tell you for certain that a flagged reference is fake. Five limits recur across the papers and tools above:

• Existence is not support. Phantom References says its measure "makes no claim about whether the surrounding text was generated by an LLM or whether the cited result is true". Mohammad Hosseini told Retraction Watch that the "bigger and more important problem" is AI-generated citations that are not wholly hallucinated but are inaccurate, biased or incomplete. The Discover Artificial Intelligence retraction was about references that exist.
• "Not found" is not "fake". Hallucinator's README warns that "technical reports, books, workshop papers, brand-new work, non-English venues, and software/websites are often missing or indexed with partial metadata". ICLR's own checker would "flag a reference to a paper with a non-English title that a submission's authors had translated to English". Springer Nature's research integrity director, Chris Graf, told Retraction Watch that references "often do not include DOIs, and simple tools to identify hallucinated references can produce false positives".
• DOIs from other agencies. Crossref and DataCite are not the only DOI agencies. The DOI Foundation also lists Airiti, Chinese DOI, CNKI, the Japan Link Center, the Korean DOI Center, mEDRA and the Publications Office of the EU among others (DOI Foundation). A tool that asks only Crossref or DataCite can report a valid DOI from those agencies as missing. Resolve such DOIs at doi.org, which serves the whole DOI system, and use "Which RA?" to see which agency holds them.
• "Found" is not "right". Two works can share a title: Phantom References describes a flag caused by SIGIR-AP 2023 containing "two distinct tutorials with this identical title", where the checker had matched the wrong one. A hijacked DOI or arXiv ID also resolves cleanly to a real paper, so a link check alone passes it.
• AI-text detectors test something else. Badalova and Mayr note that general AI text detectors "are not suitable for evaluating hallucinated citations because they do not verify whether references exist or whether their metadata are correct".

How do you verify references when an AI helped you write?#

Do not let the AI supply references from memory. Give it sources you chose, or tools that query registries, and check every reference it touched yourself, including ones it only reformatted.

Formatting is one way errors get in. The NeurIPS board suggested that "authors may have given an LLM a partial description of a citation and asked the LLM to produce bibtex (a formatted reference)", and authors contacted for Phantom References gave similar explanations. JAMA's rule gives the reason: these tools "may cause problems with the accuracy and integrity of cited references", so it recommends standard reference managers instead.

Asking the chatbot to check itself is not a check. Reviewing earlier studies, Walters and Wilder wrote that ChatGPT "often provides incorrect responses when asked “Is this citation correct” or “Do you fabricate citations?”". If an AI app does the lookup through a tool, the tool's answer comes from a registry, but the app's summary of that answer is still the app's. Some checkers can be connected to AI apps directly: CheckIfExist's README describes using it "inside AI assistants like Claude Desktop or VS Code Copilot through the Model Context Protocol (MCP)".

Then disclose what the AI did. ICLR 2027 recommends disclosing AI use to "identify relevant literature" and to "format references", and arXiv asks authors to report significant use of "text-to-text generative AI". Our guide on how to disclose AI use in research has wording you can adapt. The same habit applies to everything an AI produces for a paper, including what an AI research agent reports about its own runs (why AI agents leave out negative results). The ACL 2026 chairs put the principle plainly: the efficiency these models bring to literature review "does not remove the author's obligation to manually confirm the existence and relevance of every cited source before submission".

A pre-submission checklist for references#

1. Freeze the reference list after the last formatting or BibTeX pass, and check that version.
2. Resolve every DOI at doi.org. For any that fail, check https://doi.org/doiRA/ before you conclude it is fake.
3. Open every arXiv ID, PMID and ISBN in its own registry.
4. For each reference, compare the title, first author, year and venue with the registry record, and correct every field that differs.
5. Search the exact title of each reference that has no identifier, and check the author list of the match.
6. Run one free checker as a second pass, and resolve every flag by hand.
7. Look up web sources in the Wayback Machine, and cite an archived copy if the live page has changed.
8. Make sure each reference names the version you read, preprint or published.
9. Open each source and find the passage that supports your sentence.
10. Record any AI help with finding, writing or formatting references in your disclosure.

Where Papex fits#

When you review a Hypothesis, Discovery or Publication draft in Papex, the References section has a Look up button for each reference. It asks the registry for that identifier: Crossref or DataCite for a DOI, arXiv for an arXiv ID, PubMed for a PMID and Open Library for an ISBN. It then fills in the title, authors, venue and year from what the registry lists, and you can change any of it.

The answer appears as a dated observation, such as "Found in Crossref · 18 Sep 2026", "Title differs: registered as “…”" or "Not found in the registry". Before you sign, Papex warns you, for example "Check 1 reference: 1 not found. You can still sign." It never blocks signing. On a published record, each reference shows Papex's latest observation beside it, outside the signature, with the line "Papex looked these up; that says a source exists, not that it supports the record."

An AI app connected to Papex (see Papex in AI apps) can run the same lookup through Papex's MCP tool papex_lookup_reference. The tool also compares a title it is given with the registered title, and reports whether a web address has a copy in the Internet Archive. Papex never fetches the address itself.

The lookup checks that an identifier exists and what it is registered as. It does not check that a source supports any claim, and DOIs registered with agencies other than Crossref or DataCite currently read "Not found". It is not a plagiarism, AI-text or peer-review check, so reading each source (step 9 above) remains yours.

Sources#

• Compound Deception in Elite Peer Review: A Failure Mode Taxonomy of 100 Fabricated Citations at NeurIPS 2025 (Samar Ansari, arXiv 2602.05930, 5 February 2026), checked 19 October 2026
• GPTZero finds 100 new hallucinations in NeurIPS 2025 accepted papers (GPTZero, 21 January 2026), checked 19 October 2026
• GPTZero uncovers 50+ Hallucinations in ICLR 2026 (GPTZero, December 2025), checked 19 October 2026
• NeurIPS, AI conferences and hallucinated citations (Fortune, 21 January 2026), checked 19 October 2026
• Phantom References: Hallucinated Citations That Survive Peer Review at Top-Tier Conferences (Russinovich, Siva Kumar and Salem, arXiv 2607.00738, v2 6 July 2026), checked 19 October 2026
• LLM hallucinations in the wild: Large-scale evidence from non-existent citations (Zhao et al., arXiv 2605.07723, 8 May 2026), checked 19 October 2026
• Nearly 3,000 peer-reviewed medical papers have fake citations (Columbia University School of Nursing), checked 19 October 2026 (The Lancet's own page blocked automated access)
• One in 277 PubMed-indexed papers in 2026 shows fabricated references, says analysis (Retraction Watch, 7 May 2026), checked 19 October 2026
• Fabrication and errors in the bibliographic citations generated by ChatGPT (Walters and Wilder, Scientific Reports, 2023), checked 19 October 2026
• Thread on arXiv's penalties for unchecked LLM output (Thomas G. Dietterich on X, 14 May 2026), checked 19 October 2026 (read through X's public embed service; the posts are numbered 1/ to 4/ and end/)
• arXiv moderation: policy for authors' use of generative AI language tools (arXiv), checked 19 October 2026
• Ban for authors submitting AI content 'welcome but unenforceable' (Times Higher Education, 20 May 2026), checked 19 October 2026
• ICLR 2026 Response to LLM-Generated Papers and Reviews (ICLR, 19 November 2025), checked 19 October 2026
• A Retrospective on the ICLR 2026 Review Process (ICLR, 31 March 2026), checked 19 October 2026
• AI Policy for Authors (ICLR 2027), checked 19 October 2026
• Area Chair Instructions (ICML 2026), checked 19 October 2026
• Main Track Handbook (NeurIPS 2026), checked 19 October 2026
• ACL Statement on Desk Rejecting Papers with Hallucinated References (ACL 2026, May 2026), checked 19 October 2026
• Instructions for Authors (JAMA), checked 19 October 2026 (Internet Archive copy of 16 September 2026; the live page blocked automated access)
• Retraction Note: Innovative application of artificial intelligence in a multi-dimensional communication research analysis: a critical review (Discover Artificial Intelligence, 3 October 2026), checked 19 October 2026 (read through a text-only reader service; the publisher's page blocked automated access)
• Retraction Note: Whistleblowing in Ethiopian Public Educational Institutions (Journal of Academic Ethics, 2 January 2026), checked 19 October 2026 (read through a text-only reader service; the publisher's page blocked automated access)
• The case of the fake references in an ethics journal (Retraction Watch, 2 December 2025), checked 19 October 2026
• DOI Resolution Documentation: Which RA? (DOI Foundation), checked 19 October 2026
• Existing Registration Agencies (DOI Foundation), checked 19 October 2026
• Understanding the arXiv identifier (arXiv), checked 19 October 2026
• arXiv DOIs (arXiv), checked 19 October 2026
• Adding Journal Reference and DOI (arXiv), checked 19 October 2026
• Books API (Open Library), checked 19 October 2026
• Adding Items to Zotero (Zotero documentation), checked 19 October 2026
• Simple Text Query (Crossref), checked 19 October 2026
• Simple Text Query documentation (Crossref, last updated 6 October 2020), checked 19 October 2026
• Crossref Metadata Search (Crossref), checked 19 October 2026
• DataCite Commons (DataCite Support), checked 19 October 2026
• Using the Wayback Machine (Internet Archive Help Center), checked 19 October 2026
• CheckIfExist: Detecting Citation Hallucinations in the Era of AI-Generated Content (Diletta Abbonato, arXiv 2602.15871), checked 19 October 2026
• CheckIfExist repository and README (GitHub), checked 19 October 2026
• RefChecker repository and README (GitHub), checked 19 October 2026
• Hallucinator repository, README and licence (GitHub), checked 19 October 2026
• HalluCiteChecker: A Lightweight Toolkit for Hallucinated Citation Detection and Verification (Sakai, Kamigaito and Watanabe, arXiv 2604.26835), checked 19 October 2026
• HalluCiteChecker repository (GitHub), checked 19 October 2026
• halref: Hallucinated Reference Finder repository (GitHub), checked 19 October 2026
• CiteScan Space and README (Hugging Face), checked 19 October 2026
• CiteAudit: You Cited It, But Did You Read It? (arXiv 2602.23452), checked 19 October 2026
• CiteAudit repository (GitHub), checked 19 October 2026
• Hallucination Detector (GPTZero), checked 19 October 2026
• Detecting Hallucinated and Suspicious Citations: What Current Tools Can and Cannot Do (Badalova and Mayr, arXiv 2607.22693, v2 28 July 2026), checked 19 October 2026