Some sites have online videos playable with an embedded JavaScript player. It would be nice to store those files for offline viewing...

Firefox can produce a .har file (HTTP Archive file), a JSON-formatted log that records all network traffic. The particular site I was interested in used something called THEOplayer. It fetches chunks of content periodically. The chunks seem part of an MPEG transport stream (ts). Each chunk is requested with a signature on the request (which I did not care to compute). However, all requests for all chunks are listed upfront.

This is what a single request looks like:

ttps://chunk-oci-us-phoenix-1-vop1.fastly.mux.com/v1/chunk/EO...1Z/0.ts?skid=default&signature=Nm..MA==&zone=2

And the list of chunks is given here in the HAR file:

          "content": {
            "mimeType": "application/vnd.apple.mpegurl",
            "size": 35377,
            "text": "#EXTM3U\n#EXT-X-VERSION:3\n#EXT-X-TARGETDURATION:6\n#EXT-X-PLAYLIST-TYPE:VOD\n#EXTINF:5,\nhttps://chunk...

Probably there's tools that already do the job, but here's my attempt

Let's get to work (using copilot): prompt #1:

create python script that opens a .har file (firefox browser network log), file the list of video content urls (content with mimetype "application/vnd.apple.mpegurl", text element), extract the list of urls with /chunk/ and download all chunks and concatinate into .ts file

prompt #2:

there is a file sample.har to test

Resulting Python code:

#!/usr/bin/env python3
"""Extract HLS chunk URLs from a Firefox .har file, download and concatenate into a .ts file."""
import argparse
import json
import sys
from concurrent.futures import ThreadPoolExecutor
from urllib.request import Request, urlopen
from urllib.parse import urljoin

MIME = "application/vnd.apple.mpegurl"


def find_chunk_urls(har_path):
    with open(har_path, encoding="utf-8") as f:
        har = json.load(f)
    urls = []
    for entry in har["log"]["entries"]:
        content = entry.get("response", {}).get("content", {})
        if MIME not in (content.get("mimeType") or "").lower():
            continue
        text = content.get("text")
        if not text:
            continue
        if content.get("encoding") == "base64":
            import base64
            text = base64.b64decode(text).decode("utf-8", "replace")
        base = entry["request"]["url"]
        for line in text.splitlines():
            line = line.strip()
            if line and not line.startswith("#") and "/chunk/" in line:
                urls.append(urljoin(base, line))
    # de-duplicate, keep order
    return list(dict.fromkeys(urls))


def fetch(url, retries=3):
    for i in range(retries):
        try:
            req = Request(url, headers={"User-Agent": "Mozilla/5.0"})
            with urlopen(req, timeout=30) as r:
                return r.read()
        except Exception as e:
            if i == retries - 1:
                raise
            print(f"retry {url}: {e}", file=sys.stderr)


def main():
    p = argparse.ArgumentParser()
    p.add_argument("har")
    p.add_argument("-o", "--output", default="output.ts")
    p.add_argument("-j", "--jobs", type=int, default=8)
    a = p.parse_args()

    urls = find_chunk_urls(a.har)
    if not urls:
        sys.exit("No /chunk/ URLs found")
    print(f"{len(urls)} chunks")
    with open(a.output, "wb") as out, ThreadPoolExecutor(a.jobs) as ex:
        for i, data in enumerate(ex.map(fetch, urls), 1):
            out.write(data)
            print(f"\r{i}/{len(urls)}", end="", flush=True)
    print(f"\nWrote {a.output}")


if __name__ == "__main__":
    main()

And it works. (End of story)

The resulting MPEG file can be played/viewed using ffplay, other tools have some sync issues.

posted at: 08:32 | path: / | permanent link