npp-finder

Filter new Wikipedia pages by whether their references contain URLs — a New Pages Patrol triage utility.

New Pages Patrol reviewers face a persistent backlog of thousands of unreviewed articles. A common red flag during review is references that cite a source without providing a clickable link. These come in several forms:

A page where every single reference lacks a URL is difficult to verify quickly. Reviewers must either own the cited books, chase down print-only journals, or take the citations on faith. These pages deserve extra scrutiny — and npp-finder surfaces them automatically.

1. Pulls all newly created mainspace pages from the last N days via the MediaWiki Action API (list=recentchanges&rctype=new)
2. Batch-fetches each page's raw wikitext, size, revision ID, and categories in one combined query (50 pages per API call via prop=revisions|info|categories)
3. Parses every <ref> tag with mwparserfromhell — proper AST parsing, not fragile regex
4. Resolves named references (<ref name="x" /> → finds the original definition elsewhere on the page)
5. Checks each reference for:
   • Raw http:// or https:// strings in the body text
   • Citation template parameters: url, archive-url, chapter-url, conference-url, contribution-url, transcript-url (and their camelCase variants)
   • Nested templates inside references (e.g. a {{cite web}} inside a <ref> tag)
6. Enriches each page with:
   • Page size (bytes) — from prop=info
   • Infobox presence — parsed via mwparserfromhell for {{Infobox *}} templates
   • Category count — from prop=categories
   • Previous deletions — checks the deletion log for same-title matches
   • Article quality prediction — Lift Wing ML model (Stub / Start / C / B / GA / FA)
   • Creator edit count — from list=users&usprop=editcount
7. Outputs the pages where zero references have a URL — the strict set, with all enrichment fields in the same table

Pages with no references at all are reported separately. Pages with some URLs and some URL-free refs are excluded from the main output because at least one citation is verifiable online.

• Python 3.10+ — recent macOS versions include python3 via Xcode command line tools or Homebrew. To check: python3 --version.
• An internet connection (for the Wikipedia API — no account needed for reads).

git clone https://github.com/fuzheado/npp-finder.git
cd npp-finder
python3 -m venv .venv
source .venv/bin/activate

# Upgrade pip and setuptools inside the venv (avoids build warnings)
pip install --upgrade pip setuptools wheel

# Install npp-finder in editable mode
pip install -e .

⚠️ macOS note: Modern macOS (Sonoma/Sequoia) ships with Python 3.14 via Homebrew, which blocks pip install outside a virtual environment (PEP 668 — externally-managed-environment error). Always activate the venv first with source .venv/bin/activate before running pip install. If you see that error, you forgot the activate step.

npp-finder --days 1 --limit 10

This should list the most recently created Wikipedia pages and highlight any with URL-free references. No Wikipedia login required.

npp-finder --days 7 --limit 100

npp-finder --days 7 --limit 200 --output json > suspicious.json

npp-finder --days 7 --limit 200 --output csv > suspicious.csv

npp-finder --days 7 --limit 200 --output wikitable > patrol-log.wiki

Paste the contents of the file directly into any wiki page. It renders with:

• Article titles wikilinked ([[Title]]) — clickable
• Usernames linked to their user page ([[User:Name|Name]]) — clickable
• class="wikitable sortable" for automatic styling and column sorting
• Summary embedded as an HTML comment at the top

Example output:

{| class="wikitable sortable"
! Date
! Title
! Creator
! Edits
! Size
! Infbx
! Cats
! Del?
! Quality
! Refs
! URL
! Sample ref
|-
| 2026-06-09
| [[LOL: Slutty Bass]]
| [[User:Vvenom974|Vvenom974]]
| 148
| 4,364
| Y
| 6
| N
| Start
| 3
| 0
| <code>[cite Instagram]</code>
|-
| 2026-06-09
| [[Betka Ait Mokran]]
| [[User:Naslechat|Naslechat]]
| 402
| 2,175
| Y
| 11
| N
| Stub
| 2
| 0
| <code>[harvsp]</code>
|}

The Lift Wing quality model adds ~0.15s per page. Use --no-quality to skip it:

npp-finder --days 7 --limit 100 --no-quality

Pages created in last 1 days: 30
Pages with references: 27
Pages with ONLY non-URL references: 2
Pages with no references at all: 3

+----------+------------------+-----------+--------+--------+------+------+------+--------+------+-----+----------------------------+
|   Date  |      Title      |  Creator | Edits |  Size | Infbx| Cats| Del?| Quality| Refs| URL|         Sample ref         |
+----------+------------------+-----------+--------+--------+------+------+------+--------+------+-----+----------------------------+
| 2026-06-09| LOL: Slutty Bass| Vvenom974|    147|  4,366|  Y  |    6|  N  | Start |    3|   0| [cite Instagram]           |
| 2026-06-09| Betka Ait Mokran| Naslechat|    402|  2,175|  Y  |   11|  N  |  Stub |    2|   0| [harvsp]                   |
+----------+------------------+-----------+--------+--------+------+------+------+--------+------+-----+----------------------------+

The rightmost column shows a snippet of one of the URL-free references on the page, with wikitext formatting simplified:

• [harvsp] — a shortened footnote template ({{harvsp|Author|Year}}) that points to a bibliography section
• [Cite journal] — a {{cite journal}} template with no |url= parameter
• [cite Instagram] — an Instagram citation, which is rarely a reliable source
• Venters, Colin C., et al. "Sof... — a plain-text reference with no template wrapping at all (no title, no publisher, no link)

• Read-only. This tool does not tag pages, leave talk page messages, or nominate for deletion. It's a discovery aid, not an automated reviewer.
• Inline refs only. References defined inside <ref> tags are fully analyzed. List-defined references and bibliography-only citations outside <ref> tags are checked but some edge cases may be missed.
• Recent changes window. The recentchanges API has a practical limit of ~30 days of history.
• No authentication needed. For read-only queries, no Wikipedia login is required. --unreviewed-only needs the patrol user right.
• Quality predictions are best-effort. The Lift Wing model may be unavailable or return errors for very new pages. Predictions are informational only and should not be treated as authoritative.