Add MPDF to your app

If your app opens PDFs, it already opens MPDF files. To also play the music, read one attachment and follow a few rules. Nothing in the pages changes, so your rendering code stays as it is.

What's in the file

A normal PDF with an embedded file named mpdf.json (and, optionally, audio files). The manifest lists tracks and page cues:

mpdf.jsonJSON

{
  "mpdf": "1.0",
  "title": "Piano Book 2",
  "tracks": [
    { "id": "t1", "title": "Canon in D", "src": "youtube:dQw4w9WgXcQ" },
    { "id": "t2", "title": "Arirang",    "src": "file:track2.m4a" }
  ],
  "cues": [ { "page": 1, "track": "t1" }, { "page": 4, "track": "t2", "at": 32.5 } ]
}

The four steps

Find the attachment named mpdf.json. No attachment means a plain PDF; do nothing.
Parse it and show the tracks. Ignore fields and src schemes you don't know.
Play: youtube:ID through the official YouTube player (kept visible), file:name from the embedded file of that name, https: as a URL.
On page change, if a cue exists for the new page, switch to its track at at seconds; otherwise keep playing.

Rules that keep the format trustworthy: never draw over the page; the player is docked outside the page or is a movable, collapsible panel; no autoplay before a user gesture. Details in the spec (three pages, CC BY).

Reading the manifest

JavaScriptpdf.js

// Browser or Node, with pdf.js (any version that has getAttachments)
const pdf = await pdfjsLib.getDocument({ data: bytes }).promise
const att = (await pdf.getAttachments()) || {}
const raw = att['mpdf.json']                       // no attachment -> a plain PDF, nothing to do
if (raw) {
  const manifest = JSON.parse(new TextDecoder().decode(raw.content))
  // manifest.tracks: [{ id, title, src }]   src = "youtube:ID" | "file:name" | "https://..."
  // manifest.cues:   [{ page, track, at }]  when page becomes current, play track from `at` seconds
  // embedded audio:  att[name].content       for src "file:name"
}

Pythonpypdf

# Python 3.8+, pip install pypdf
from pypdf import PdfReader
import json

reader = PdfReader("score.pdf")
att = reader.attachments                           # {name: [bytes]}
if "mpdf.json" in att:
    manifest = json.loads(att["mpdf.json"][0])
    tracks = {t["id"]: t for t in manifest["tracks"]}
    for cue in manifest.get("cues", []):
        print(cue["page"], tracks[cue["track"]]["src"], cue.get("at", 0))

SwiftPDFKit

// iOS / macOS with PDFKit (reference; walks the flat /Names form of the EmbeddedFiles tree)
import PDFKit

func mpdfManifest(_ doc: PDFDocument) -> [String: Any]? {
    guard let cg = doc.documentRef, let catalog = cg.catalog else { return nil }
    var names: CGPDFDictionaryRef?, ef: CGPDFDictionaryRef?, arr: CGPDFArrayRef?
    guard CGPDFDictionaryGetDictionary(catalog, "Names", &names), let names,
          CGPDFDictionaryGetDictionary(names, "EmbeddedFiles", &ef), let ef,
          CGPDFDictionaryGetArray(ef, "Names", &arr), let arr else { return nil }
    for i in stride(from: 0, to: CGPDFArrayGetCount(arr), by: 2) {
        var key: CGPDFStringRef?, spec: CGPDFDictionaryRef?, efd: CGPDFDictionaryRef?, stream: CGPDFStreamRef?
        guard CGPDFArrayGetString(arr, i, &key), let key,
              (CGPDFStringCopyTextString(key) as String?) == "mpdf.json",
              CGPDFArrayGetDictionary(arr, i + 1, &spec), let spec,
              CGPDFDictionaryGetDictionary(spec, "EF", &efd), let efd,
              CGPDFDictionaryGetStream(efd, "F", &stream), let stream else { continue }
        var fmt = CGPDFDataFormat.raw
        guard let data = CGPDFStreamCopyData(stream, &fmt) as Data? else { return nil }
        return try? JSONSerialization.jsonObject(with: data) as? [String: Any]
    }
    return nil
}

Producing a file

Pythonpypdf

# Producing one: attach the manifest (and any audio) to a normal PDF
from pypdf import PdfWriter
w = PdfWriter(clone_from="score.pdf")
w.add_attachment("mpdf.json", manifest_bytes)
w.add_attachment("track2.m4a", audio_bytes)   # optional
w.write("score.pdf")                            # still a PDF; name it .pdf

The JavaScript and Python reference libraries (about 80 lines each, MIT) do both directions with validation, and sample files exercise every case. Conformance: Level 0 opens the PDF (you already do), Level 1 plays the tracks, Level 2 follows page cues and registers the .mpdf extension.

Questions or a viewer to list on this site: open an issue on GitHub.