The problem I actually had
In February 2022 I downloaded a course pack that someone had stitched together from lecture PDFs. The file was around eighty pages and roughly a third of them were repeats. The same fifteen slides sat at positions 12, 27, 43, and 61, and the person who assembled it either did not notice or did not care.
I looked for a tool. The web ones asked for money to remove more than a page or two. The desktop ones were installers from sites I did not want to run on my laptop. The open-source ones assumed you had a PDF library toolchain installed and a Stack Overflow tab open.
So I wrote one. One file, JET-PDF.py, 111 lines, two dependencies pinned in requirements.txt (PyPDF2 1.26.0 and tqdm 4.62.3). The README explained the origin story in one line and then, in a spirit I remember well from that year, told Mac users "You are own your own buddy" (typo intact). Apache 2.0 license, sign-off "Made with love by Kaushik" at the bottom of both the script and the README.
Text-equality dedupe
The core loop is not a hash of the rendered page. It is a nested loop over every pair of pages, calling pageObj.extractText() on each, and comparing the extracted text strings for equality.
flowchart LR
A[Input PDF] --> B[extractText per page]
B --> C[Nested loop over pairs]
C --> D[Text equality check]
D --> E[Keep unique pages]
E --> F[Write new PDF]That is O(n²) in page count with a string comparison inside. For a hundred-page slide pack it is fine. For a thousand-page reference book it would be slow, but at the time I did not have a thousand-page reference book to test on.
The choice is defensible even now if you narrow the input class. For born-digital slide decks (LaTeX Beamer, PowerPoint export, Google Slides download), the text layer is stable across a repeat: the same slide extracts to the same string every time. Two visually identical born-digital pages will compare equal.
Where it breaks is scans. If your PDF is a photograph of a book chapter, extractText() returns an empty string, and every empty string compares equal to every other empty string. My little dedupe tool will happily tell you your entire 400-page scanned textbook is one page, repeated. There is no OCR fallback, no guard against extractText() returning "" and no length check. This is a real limitation of the approach, not a bug in the loop.
Duplicates are stored in a dict keyed by (min(p1, p2), max(p1, p2)) so a pair is not counted twice, and then the code walks that dict and appends the second page of each pair (rep[1]) to a duplicates list. The output writer then adds every page not in that list to a new PdfFileWriter and saves as <name>.pdf.
The 2.86 MB Windows .exe
Two commits into the repo, right after the code landed, there is a dist/JET-PDF/ folder committed to git with a PyInstaller bundle inside. The bundle contains JET-PDF.exe at 2.86 MB, libcrypto-1_1.dll, libssl-1_1.dll, mfc140u.dll, base_library.zip, and a bag of PyInstaller runtime files including pip 22.0.4 and altgraph 0.17.2.
The reason it exists is that after I sent the Python file to a friend, the reply was "how do I run this." Then a second friend asked the same thing. So I ran PyInstaller once, checked in the dist/ folder, and now the README could say "grab the exe, double-click, paste the path to your PDF."
The OpenSSL DLLs are there because Python's standard library links against libssl, and PyInstaller bundles everything the interpreter can reach, whether the script uses it or not. My PDF tool does not open a single socket. It still shipped with two SSL DLLs because that was the cost of not making a friend install Python.
I do not know if the bundled .exe still runs on current Windows. It probably does, and it probably has a load of dependency warnings if you look at it with a modern lens. I have not tested it in years.
Bugs I would fix on a re-read
Reading it back four years later, a short list.
There is a branch that prints "None" when the duplicate list is empty. It runs before the list is populated. So it prints "None" even when the file has duplicates that the code then goes on to find and remove. Cosmetic, but a real bug and an easy one.
The except: blocks are bare. except: pass swallows every exception including KeyboardInterrupt, which is the one you actually want to reach the shell.
There is a string that says "{page_1} and {page_2} Can't be compared" without the f prefix. It prints the literal braces. I never noticed because the path that produces the message rarely fires on the PDFs I was feeding it.
There is a dead-code twin called get_duplicates that compares PageObject instances with ==. That comparison uses object identity on PyPDF2's PageObject, so it never returns true for distinct objects, so the function returns an empty dict. The real function is get_duplicates_dir, the one that calls extractText. The dead twin sat there the whole time.
And there is a small tic: from tqdm import tqdm at the top, then tqdm.tqdm(...) in the loop. The second tqdm should not be there. It works because the module also has a tqdm attribute pointing at the class, so tqdm.tqdm still resolves to the class. It just looks wrong.
None of this stopped it from doing what I wanted on the file I wrote it for.
What a v2 would look like
If I picked this up today, three changes.
The first is a perceptual hash on rendered page images for the scanned-PDF case. Rasterise each page at low DPI, compute a pHash, bucket by hash. That covers scans and photos of pages where extractText() is empty. Keep the text-equality path for born-digital pages where it is faster and exact.
The second is the library. PyPDF2 1.26.0 is old. The project renamed to pypdf and the API changed. PdfFileReader and PdfFileWriter became PdfReader and PdfWriter. getPage became pages[i]. extractText became extract_text. A migration is mechanical but touches every call site.
The third is throwing the exe out of git and pushing releases to the Releases tab instead, or better, letting people pip install it. The bundle in dist/ is 2.86 MB per commit that touched it, and the repo carries that weight forever. I did not know about GitHub Releases at the time. Now I do.
The repo is unchanged since 26 March 2022, eight commits, no CI, no tests, no follow-ups. It solved one file for one person and then the person moved on. That is a fair definition of done for a script.