YouTube's Most-Replayed heatmap sat in a closed beta through late 2021 and rolled out to everyone in the first months of 2022. It was the little orange curve under the scrubber that told you which seconds of a video people rewound to. A few months after it went public I wanted the raw numbers behind the drawn curve. I ended up with three GitHub repos that share the same theme and split out of one scratch pad.
This post is the retrospective. What the scrape actually does, why one repo turned into three, and what the intensity curve looked like on a real video.
The three repos
Youtube-heatmap-analysis came first, on 2022-06-12. It is the scratch pad. Four files with different experiments jammed together: a bare scrape in py_yt.py, a copy of the Selenium demo in yt_ipy_to_py.py, a completely unrelated playlist bulk-downloader in on_tp_yt_dw.py using pytube against two hard-coded playlists at 360p and 480p, and a tes.txt sample of a real heatmap. One repo doing four things.
Youtube-heatmap came out on 2022-07-17. About thirty lines. One function, get_heatmap(url), that takes a watch URL and hands back the intensity curve. This is the version I would still point someone at first.
Youtube-heatmap-video shipped the same day. About 172 lines. It reuses the same scrape, then drives Selenium to auto-open the single most-replayed moment of the video in a new tab.
The scrape
YouTube's watch page ships the heatmap data inline in the HTML. If you open view-source on any video with Most-Replayed enabled and search for heatMarkerRenderer, you land on an array of objects that look like this:
{"heatMarkerRenderer":{
"timeRangeStartMillis": 12960,
"markerDurationMillis": 1620,
"heatMarkerIntensityScoreNormalized": 1.0
}}Each entry is one segment of the curve. start_ms, duration_ms, intensity between 0 and 1. String them together and you have the full curve.
The thirty-line version does not try to be clever about it. It grabs the HTML with urllib, slices between the literal {"heatMarkerRenderer": and the next known landmark heatMarkersDecorations, and returns the slice. It is a stringified list, not parsed JSON. String slicing beat json.loads here because the surrounding page is not valid JSON at all, it is a huge JavaScript blob with the array embedded somewhere in the middle. Trying to parse it end-to-end means writing a bracket-matcher first, and I was seventeen and impatient. Slice between two anchor strings, ship it.
The demo URL hardcoded into the library file is x7X9w_GIm1s. I do not remember which video that was.
Scratch, then library, then demo
The order matters. Youtube-heatmap-analysis in June was where I first got the scrape to return data at all, mixed with a completely unrelated pytube experiment because it lived in the same folder on my Desktop. os.chdir('C:\\Users\\Kau\\Desktop\\web') still sits at the top of both scratch and demo files, which is exactly the kind of thing that happens when a student project is one folder and one file until it is not.
A month later I split it. The library repo got the clean thirty lines. The demo repo got the Selenium flow. The scratch repo stayed as it was, still tracking two different projects at once. If I were doing this now I would delete the scratch repo, or archive it. In 2022 I was still learning that git history is not a diary you need to keep.
The peak-clip demo
The 172-line repo is the one I remember being proud of. It answers a question the scrape alone cannot: given a video, what single slice do people rewind to most, and can I just play that?
The logic is straightforward once you have the triples. Find the marker where intensity == 1.0, take its start time, expand a window of plus or minus two times the marker duration around it, and you have a clip roughly ten seconds long centered on the peak.
The hard part is playing that clip. YouTube's own embed API had already gone paid around then, or at least the free tier had shrunk to the point where I could not use it. YouTube also blocks video playback when the referrer is localhost. So the demo does a detour. It opens socialvideoplaza.com or listenonrepeat.com, both of which will play a YouTube video within a start and end time, and drives Selenium to fill in the video ID and the timestamps. The clip pops open in a new tab, cued to the peak.
This is where the code shows its age. It uses find_element_by_xpath and find_element_by_name, which Selenium 4.2 removed and replaced with find_element(By.XPATH, ...). The README itself lists "Refactor un-used libraries" and "Fix old/depreciated methods" under future plans, so I knew it even then. I imported pyperclip and svgpathtools and used neither of them in the shipping path.
flowchart LR
A[watch URL] --> B[HTML fetch]
B --> C[slice heatMarkerRenderer]
C --> D[triples]
D --> E[peak == 1.0]
E --> F[expand plus/minus 2 x duration]
F --> G[socialvideoplaza / listenonrepeat]
G --> H[clip in new tab]What the curve actually looked like
The tes.txt sample in the scratch repo captured a real scrape from video SR-3V5PcJVU. Ninety-nine markers. The peak sat at marker 8, at 12.96 seconds in, at intensity 1.0. Everything after that decayed almost monotonically toward zero by the end of the video.
That shape is not universal. Long tutorial videos have multiple bumps where people rewind to the actual demonstration. Music videos often peak in the chorus. But for this specific video the answer was simple: the audience cared about the first twenty seconds and then drifted off. Being able to see that from ninety-nine numbers, rather than eyeballing the curve YouTube drew, was the whole point.
What would break today
I have not re-run any of this in 2026. A few things would need attention before it worked again.
The Selenium bindings in the demo repo will fail at import. find_element_by_xpath and find_element_by_name were removed years ago. Every call site needs to move to the find_element(By.SOMETHING, selector) form.
pytube, which the scratch repo used for the unrelated playlist downloader, has been in and out of "abandoned" and "revived" states multiple times as YouTube changes its player. It may or may not work on a given week.
And I do not know whether the scrape itself still returns data. heatMarkerRenderer might have moved to a different key, or moved off the initial HTML and into a lazy /youtubei/v1/next fetch, or been gated behind login. The right way to check is to view-source on a current watch page and search for the string. I have not done that check for this post. If someone reads this and finds it still works, or finds the exact place it broke, I would like to hear about it.
Three repos, one theme, one scratch pad, and a scrape that was fun while it lasted.