localVideo — findings

Notes from building a local image/video generation app on Apple Silicon. Everything here was measured on one machine unless stated otherwise:

M5 Max · 128 GB unified memory · macOS 26.4 · ComfyUI 0.29.0 · torch 2.13.0 · Python 3.12.13 Models: LTX 2.3 22B (BF16) for video, Krea 2 for images.

Numbers from a single machine and a single library. They are real measurements, not benchmarks — treat them as orders of magnitude, not league tables.

1. The FP8 wall on Apple Silicon

Almost every curated model weight you can download is FP8. Metal has no float8 kernels. An FP8 tensor on the GPU cannot be converted, copied off, or even reinterpreted with .view(). It fails at load, not at inference:

RuntimeError: Undefined type Float8_e4m3fn

The same tensors work fine on CPU. There is no userspace workaround for the GPU path — you cannot cast your way out, because the cast itself is the unsupported operation.

This is why so many "local generation" tools hand a Mac user a 30 GB download that crashes on load.

The inversion that fixes it

FP8 exists to squeeze large models into 24 GB consumer NVIDIA cards. That constraint does not apply to a Mac with unified memory.

• torch.mps.recommended_max_memory() on this machine: 115.4 GB
• LTX 2.3 22B, uncompressed BF16: ~46 GB

So the fix is to stop compressing. Use the BF16 build the quantised one was derived from. The "compatibility" format is the problem; the "too big" format is the solution.

Blog angle: the counterintuitive one — on a big-memory Mac, the larger download is the one that works.

ComfyUI flags that matter

PYTORCH_ENABLE_MPS_FALLBACK=1 PYTORCH_MPS_HIGH_WATERMARK_RATIO=0.0 \
  python main.py --use-split-cross-attention --fp32-vae

• Never --bf16-vae — breaks the LTX audio VAE.
• Never --force-fp16 — black frames on recent macOS.
• PYTORCH_MPS_HIGH_WATERMARK_RATIO=0.0 lifts the allocator's default ~0.75×RAM

cap, which you need for a 46 GB model.

2. Measured generation cost

Fixed overhead dominates short image jobs

1 step costs 17 s and 4 steps costs 20 s. Three extra sampling steps cost 3 seconds; the job costs 17 seconds before it samples anything. Model load, text encode and VAE decode swamp the actual diffusion at low step counts.

Practical consequence: dropping from 4 steps to 1 saves ~15%, not ~75%. People optimise the wrong number.

The attention ceiling is a hard wall, not a slowdown

Refined (2× upscale) video is bounded by attention memory:

(W/32) · (H/32) · (frames/8 + 1)  >  ~45,000 tokens  →  allocation aborts

This is worth computing before submitting rather than discovering as a crash. It also means "longer" and "bigger" trade against each other on a fixed budget — you can have a long small clip or a short large one.

3. Making an extension not look like a cut

Generating a continuation from the last frame of a clip produces a visible seam. The interesting finding is why.

A single still tells the model where things are, but not where they were going. Motion restarts from rest at the join. It is not a colour or detail problem — it is a velocity problem.

LTXVAddGuide accepts a multi-frame video guide, not just one image. Feeding the last 9 frames instead of the last 1 carries velocity across the join.

Measured on the same source clip:

Honest caveat: this did not reach the ~1.5× ratio where a cut becomes genuinely invisible. The residual difference is tone, not motion — the re-rendered frames land about 1.2/255 darker. Dissolving across the overlapping frames hides that step, and it measures better than either a hard cut or a blind crossfade, because you are blending two takes of the same instants rather than two different moments.

Guides are cropped to 8n+1 frames, so 1, 9, and 17 are the usable depths.

The "it only zooms out" mystery

Early clips all drifted backwards regardless of prompt. The cause was not prompting — it was LTXVPreprocess(img_compression), which is the motion knob. ComfyUI's default is 35; this app had shipped 18. Lower compression produced less motion, which read as a slow zoom-out.

Blog angle: a parameter named "image compression" is the motion control. The name tells you nothing.

4. Drawing a lineage graph correctly

The app draws generation history as a left-to-right DAG: an image spawns clips, a clip spawns extensions, an extension gets mended back onto its source.

The bug that makes chains "fall upward"

A tidy-tree pass placed each node, then pushed it down to clear its column — but left the descendants it had already placed behind. This is Reingold–Tilford missing its modifier/shift step. The deeper a chain ran, the further it lagged, so every chain visibly climbed away from the origin it came from.

Measured over a real 68-node library, in row units:

The off-centre and spread going to exactly 0 is not tuning. Placing each node at the centre of the bounding block it produced — rather than the midpoint of its first and last child — makes every subtree symmetric about its own root by induction. "Centred in its grouping" falls out of the algorithm.

Recursion depth is the number of columns

A textbook recursive contour implementation blew the stack at ~5,300 columns. The non-recursive version (pre-order flatten, then read backwards for post-order) does:

• a 5,000-column chain in 5 ms
• an 8,191-node binary tree in under 1 ms

A generation history grows without bound, so this is not hypothetical.

Hang nodes off their deepest parent, not their first

A "mend" joins a clip and that clip's own extension — it has two parents. Hanging it off the deepest one makes the chain read left-to-right instead of doubling back. Mend edge length dropped from 1.39 → 0.32 rows mean.

5. Export formats: measure, don't estimate

For a 20.2 s clip at 896 px:

GIF is ~15× the size of VP9 for the same seconds. A full-resolution GIF of a 20 s clip is tens of megabytes and shareable nowhere, which is why hard fps and width ceilings are the only honest option.

Why not estimate sizes from bitrate

Because GIF size depends almost entirely on how much the frames move, and generated clips range from a static lantern to a whip pan. Bitrate maths would be instant and wrong exactly where it matters. Producing all three formats and reporting real bytes costs ~1.9 s and is then cached — a confident "about 3 MB" that arrives as 11 MB is worse than a two-second wait.

VP9 is unusable at defaults

-deadline good -cpu-used 4 -row-mt 1 is the difference between 0.9 s for a 10 s clip and minutes. Worth knowing before concluding VP9 is too slow to offer.

GIF needs a generated palette

Without palettegen/paletteuse, the encoder falls back to a generic 256-colour table and gradients band badly — which is most of what generated clips are.

One ffmpeg pass, not N

For a 12-frame filmstrip, tile=12x1 in a single pass takes 97 ms. Pulling twelve frames individually is twelve process spawns and twelve full decodes — a GIF has no keyframe index, so every seek re-decodes from the start.

6. What a real library looks like

Composition of a working 137-item library, which shaped several design decisions:

"Re-run this from the same source frame" is a minority path. Most items have no source frame at all. A feature that promises it universally is lying for the majority — it has to route by origin and disable itself with a reason otherwise.

Most settings were never chosen

116 of the items never recorded a motion value. The server default simply applied. Any UI that shows "Motion: 55" for those is presenting an accident as a decision.

This forced a three-state model rather than the obvious two:

• set — recorded, differs from the default: the user chose it
• default — recorded, matches the default
• assumed — never recorded: the default applied and nobody agreed to it

A default that had never once run

video.js declared baseStrength: 0.95. But the request validator materialised strength: 1 before the builder could ever reach its own fallback — making the 0.95 unreachable dead code. The library proved it: 74 of 82 clips recorded strength 1, and not one recorded 0.95, while the UI slider sat at "0.95" and called it the default.

Blog angle: a default value that has never executed, in a UI that advertises it, is invisible until you count what actually landed on disk.

Payload weight is mostly repetition

/api/lineage for this library: 110,310 bytes. Adding raw generation params per node would add +76% — and models{} alone accounts for 32,674 bytes: six identical filename strings repeated once per node. A whitelist plus a payload-level dedupe table costs +19–27% instead.

Most items are unconnected

68 nodes with 29 apparent origins collapsed to 50 nodes and 11 origins once items with no parent and no children were hidden. Roughly a quarter of a real library is one-offs that bury the lineage the graph exists to show.

7. Bugs worth writing up

A collection of failures that were invisible rather than loud — the interesting category.

Number(null) is 0, and 0 is finite. URLSearchParams.get() returns null for an absent parameter. A naive Number.isFinite guard therefore accepts it and clamps an absent speed to the 0.25 floor — silently returning slow motion for a request that never mentioned speed.

A silent no-op from a lookup that returned the wrong map. Graph node clicks resolved through the raw fetched data, while the DOM was built from the rendered view. Nodes that existed only in the view (in-flight jobs) returned null and the click quietly did nothing.

A deliberate design choice becoming a bug one feature later. In-flight "ghost" nodes deliberately never wrote themselves onto their parent's children array — correct, because a finished job would otherwise leave a dangling id forever. Two features later, chain-selection walked children and silently skipped every running job. The original comment explaining why is what made the second bug findable.

A graph walk that overshoots its own filter. A reachability helper marked ids as visited before checking whether they resolved, so under a filter it returned nodes one hop outside the visible set. In a "file these into a group" feature, that means filing items the user cannot see.

A fast path that fired too eagerly. A stream-copy shortcut triggered on any "repeat once at 1× speed" — including requests that changed container or dropped audio. Asking for WebM would have handed back an untouched MP4.

Audio reintroduced after being declined. A ping-pong filter chain rebuilt its audio branch from "does the source have audio" rather than "did we resolve an audio label", re-adding a track the caller had just asked to drop.

ffmpeg rejects an empty -filter_complex. A plain transcode pushes no filters, so the flag must be omitted entirely rather than passed empty.

An extension guard that corrupted by coercing. An upload path forced any unknown extension to .png — so a GIF was storable, just stored as GIF bytes under a name nothing could decode.

A whitelist that silently dropped new fields. A normaliser copied a fixed list of keys; twice, a new field was added upstream and simply never arrived, with no error anywhere.

A positional index that retargeted itself. A lightbox tracked the open item by list position. When a newly generated item shifted the newest-first list, actions silently applied to a different item — including downloads and "extend this".

Common thread: none of these threw. They returned undefined, 0, null, or the wrong-but-valid thing. The ones that surfaced did so because a number was counted against real data, not because anything crashed.

8. Design positions worth defending

Things that came out of the above and are arguable enough to be interesting.

Measure, don't estimate, when the estimate can be wrong in the direction that matters. Two seconds of waiting beats a confident wrong number someone plans around.

Three states beat two whenever "unset" is possible. Anywhere a default can apply silently, a UI that shows only "changed / unchanged" fabricates intent.

A format conversion is not a new artifact. A loop of a clip is something you can extend again and earns a place in the history graph. A WebM of a clip that already exists is a delivery copy — filing it would add an identical-looking node per platform someone exports for.

Raw material is not the user's work. A pasted GIF stays out of the gallery. Putting someone else's image among a user's own output is a category error.

Frame zero is almost never the interesting frame. Given an animation as input, where you continue from should be the user's decision, not a silent default.

An explicit statement beats a heuristic. When a user has manually filed items into a group, a "hide unconnected items" heuristic must not then hide them. Grey the heuristic out and say why, rather than silently overriding either one.

Reproducing any of this

The app is MIT, zero runtime npm dependencies, no build step: <https://github.com/MassiveNoobie/localVideo>

Requirements: macOS on Apple Silicon (36 GB+ recommended, 64 GB+ for comfortable video), Node.js 22+, a running ComfyUI, and BF16 model weights.