FrankLiuDundun/latentsync-finetune-lora-v1

LoRA-finetuned LatentSync UNet, merged back into the base checkpoint. Built with scripts/merge_lora.py.

How to use

from latentsync.pipelines.lipsync_pipeline import LipsyncPipeline

pipeline = LipsyncPipeline.from_pretrained(
    "ByteDance/LatentSync-1.5",       # base components
    unet_path="FrankLiuDundun/latentsync-finetune-lora-v1/latentsync_unet.pt",   # this merged file
)

or via the Gradio Fine-tune Studio (gradio_finetune.py) by selecting this repo's latentsync_unet.pt as the inference checkpoint.

Files in this repo

• latentsync_unet.pt — drop-in replacement for the base UNet
• adapter/ — original peft-format LoRA adapter (if you want to keep training)
• unet_config.yaml — UNet architectural config
• scheduler/ — DDIM scheduler

Provenance

• LoRA rank: 48
• Base checkpoint: checkpoints/latentsync_unet.pt
• LoRA adapter: /root/autodl-tmp/latentsync_finetune/unet/train_lora-2026_07_20-14:03:57/checkpoints/checkpoint-4500
• UNet config: configs/unet/stage2.yaml

Caveats

This model is a community finetune of the base LatentSync checkpoint. Use the same audio-video preprocessing as the base model. No guarantees are made about identity preservation, side-face robustness, or lip-sync on out-of-distribution sources.