Baberg/ltx-2.5-22b-ic-lora-cel-character

LTX-2.5 22B IC-LoRA — live-action → 2D cel character

Converts live-action footage so that one central character becomes hand-drawn 2D cel animation while everything else — background, props, bystanders — stays photographic.

Trained at 1152×672, which is the resolution it should be used at.

Example

Live-action reference on the left, generated output on the right. One take, no retouching — the man is fully cel-drawn while the rain-streaked glass and the neon behind him stay photographic.

Generated from this prompt:

Live-action footage in a tight close-up on a weathered older man with a grey beard standing inside a glass phone booth at night, pressing a black telephone handset to his ear as his face crumples and he weeps with his eyes squeezed shut. Rain streams down the glass behind him and out-of-focus red and amber neon signs glow in the wet street beyond.

---

What is in this repo

ltx25-iclora-cartoon-cum3250.safetensors   the LoRA (rank 32, 312 MB)
config.yaml                                the training config
clean/                                     11 demo renders, 1152×672, output only
side_by_side/                              the same 11, 2304×672, reference | output

Every clip in clean/ and side_by_side/ was generated with the checkpoint in this repo, at 1152×672, seed 42, 30 inference steps, CFG 3.0, STG 1.0, guidance rescale 0.7.

---

How to use it

This is an IC-LoRA video-to-video model. Every generation takes two inputs together:

1. a live-action reference clip — the conditioning video
2. a text prompt describing that clip

There is no trigger word.

Prompt template

Live-action footage in a [SHOT TYPE — close-up / medium close-up / medium shot] of [THE ONE CHARACTER THAT WILL BE TURNED INTO 2D — describe them distinctly: age, hair, clothing, distinguishing features], [WHAT THAT CHARACTER IS DOING]. [SETTING, LIGHTING AND BACKGROUND — everything here STAYS PHOTOGRAPHIC, do not name other people].

Only the character in the second bracket is converted. Everything in the last bracket stays live-action — but anyone you name there will also be converted, so describe the background as places and objects, not people.

Working example:

Live-action footage in a tight close-up on a weathered older man with a grey beard standing inside a glass phone booth at night, pressing a black telephone handset to his ear as his face crumples and he weeps with his eyes squeezed shut. Rain streams down the glass behind him and out-of-focus red and amber neon signs glow in the wet street beyond.

Rules that matter

Do name exactly one character; describe them distinctly enough that they cannot be confused with anyone else on screen; open directly with the action or a visual detail; be generous with detail — small objects, textures, secondary movement, background elements.

Don't name other people in the shot; don't preface with "The video opens with…", "The scene shows…", "We see…"; and don't try to protect anything with the prompt. Phrases like "the background stays photographic" or "only the woman is animated" have no effect. What gets converted is learned from the training data, not steered by wording — this was tested directly by rendering the same clip with a prompt naming three people and again naming only one; the output was unchanged.

Inference settings

Resolution	1152×672, 97 frames, 24 fps
Inference steps	30
Video CFG scale	3.0
Video STG scale	1.0
Guidance rescale	0.7
Negative prompt	none

Resolution matters. The style is learned at a specific bucket and does not survive being asked for much more. Render at 1152×672 and upscale afterwards if you need a larger delivery size.

Running it

Rendering goes through the LTX-2 trainer's own validation sampler — no third-party inference code is required:

git clone https://github.com/Lightricks/LTX-2.git ltx2
cd ltx2 && git checkout 400fd31 && uv sync

uv run --directory packages/ltx-trainer python scripts/train.py config.yaml \
  --disable-progress-bars

with config.yaml set to optimization.steps: 1, validation.interval: 999, validation.skip_initial_validation: false, and one validation.samples entry per clip:

validation:
  samples:
    - prompt: "Live-action footage in a tight close-up on ..."
      conditions:
        - type: reference
          video: /path/to/your_clip.mp4
          downscale_factor: 1
          temporal_scale_factor: 1
          include_in_output: true      # writes reference | output side by side
  video_dims: [1152, 672, 97]
  inference_steps: 30
  video_cfg_scale: 3.0
  video_stg_scale: 1.0
  guidance_rescale: 0.7
  generate_audio: false

Use the DEV transformer, never the distilled one. The distilled checkpoint produces burnt, overcooked results with this LoRA. Some pipeline examples in the LTX-2 repo default to distilled — override them and assert the resolved path before every launch.

---

The dataset

A fully synthetic paired-video dataset, created from scratch. Nothing was filmed or scraped.

Cinematic live-action clips were generated, each built around a single clearly identifiable central character. For each clip the first frame was extracted, the central character in that frame was replaced with a 2D cel-animated version of the same character in the same pose and framing, and the clip was then re-animated from that edited frame so the character stays 2D cel throughout the shot while the surroundings remain photographic.

That yields both halves of a training pair from the same shot: the original clip is the reference (conditioning input), the re-animated clip is the target (generation goal). Because both derive from one source, framing, camera motion and timing match exactly — the only difference is the character's rendering.

20 pairs / 40 clips were used for training, conformed to 768×448, 24 fps, 97 frames, silent.

The dataset is public: Baberg/ltx-2.5-ic-lora-cartoon-dataset — it holds 27 pairs, of which these 20 were used. The other seven were dropped because their targets restyled more than 60% of the frame instead of isolating one character, which taught the model to repaint whole scenes. Removing them, with nothing else changed, cut the share of frame repainted on unseen clips from 33–51% down to 9–24%. See EXCLUDED.md in the dataset for the list and the reasoning; the seven also serve as held-out evaluation clips, since this model has never seen them.

Captions were generated with Google Gemini 3.5 Flash (gemini-3.5-flash), captioning the reference halves. The caption instruction requires the exact phrase "live-action footage" and requires the one central character to be described distinctly enough to be unambiguous.

Two deliberate properties of the conversion: character reflections are stylised along with the character, and the targets add a drawn cast shadow on the ground — so the model learns the character's effect on the scene, not just a repaint of their silhouette.

---

How it was trained

Two stages. A base run at 768×448, then a resolution adaptation to 1152×672.

Stage 1 — base, 768×448

Base model	LTX-2.5 22B — dev transformer (never distilled)
Text encoder	gemma4-12b-with-proj-ltx-2.5-bf16
Mode	IC-LoRA video-to-video, flexible strategy
Conditions	reference p=1.0, first_frame p=0.2
Bucket	768×448×97 @ 24 fps
Rank / alpha	32 / 32
Learning rate	2e-4, linear schedule
Steps	3000
Precision	bf16, gradient checkpointing on
Hardware	1× RTX PRO 6000 Blackwell 96 GB
Wall clock	4h 22m, peak 57.0 GB

Stage 2 — resolution adaptation, 1152×672

The checkpoint in this repo. It warm-starts from stage 1's step-3000 weights and continues for 1000 more steps at the higher bucket, so cumulative training is 4000 steps. The style is already learned at this point; the adaptation only teaches it to hold at more latent tokens.

Starting weights	stage 1, step 3000
Bucket	1152×672×97 @ 24 fps (19,656 latent tokens vs 8,736)
Learning rate	1e-4 — half the base rate
Steps	1000 (cumulative 4000)
Everything else	unchanged from stage 1
Wall clock	4h 47m, 13.73 s/step, peak 68.25 GB

The lower learning rate is deliberate. A converged LoRA being moved to a new resolution needs small nudges; restarting at the full 2e-4 risks destroying what already works.

Checkpoint selection. Checkpoints were saved every 125 steps and swept — each rendered on the same held-out clips with identical settings, then scored on how much of the frame changed relative to the reference. The checkpoints are not equally good:

Adaptation step	Cumulative	Frame repainted
125	3125	38.5%	over-converts — repaints whole scene
250	3250	14.8%	selected — matches the base model's native strength
375	3375	39.5%	over-converts
500	3500	11.9%	clean, slightly under-converted
625	3625	13.8%	clean

The first ~400 steps are unstable while the LoRA absorbs the resolution change; it settles from step 500 on. The last checkpoint is not the best one — step 250 (cumulative 3250) was selected on measured evidence, and is what ships here.

---

Evaluation

Renders are produced side by side (reference | output) and scored by the share of frame whose pixels differ from the reference by more than 28 in any channel — lower is better, provided the character is still fully converted. A second figure, the ratio of mean signed difference to mean absolute difference, separates contained conversion (0.7–0.8) from whole-scene repainting (0.4).

Measured on clips excluded from training:

Clip	Frame repainted	Ratio
Woman on the phone	18.2%	0.81	cleanest containment measured
Bartender with a shaker	14.8%	0.76
Boy with a chicken	13.6%	0.66	held prop survives
Dancer mid-turn	10.6%	0.60
Phone booth	33.5%	0.55	high because the figure fills the frame
Subway busker	21.5%	0.41	crowd scene — see limitations

---

Limitations

Multi-person scenes convert everyone. If several people are in frame, all of them are converted, not only the one named in the prompt. This is learned behaviour and prompting cannot override it — tested directly. Use shots with one clear subject.

Crowds pick up outlines. People moving through the background behind the subject acquire ink outlines rather than staying photographic.

Face size drives facial detail. Eyes render cleanly — proper lid strokes, defined creases — when the face is large in frame. In wide shots, where the eye spans barely more than one latent cell of the VAE's 32× compressed grid, they degrade to soft, undefined shapes. Close and medium-close framing gives markedly better results. This is a property of the training data's shot distribution, not of the resolution: training at a higher bucket did not improve it.

Aspect ratio matters. The model is trained at 1.714 (1152×672). Rendering at other aspect ratios reframes the shot.

---

Credits

Base model LTX-2.5 by Lightricks. Trained with the LTX-2 trainer at commit 400fd31.