domain	archive
title	Heartmula
verification	metadata-normalized
source	skill-pipeline
status	draft
created	2026-05-09

背景

（此 lesson 从 skill heartmula 自动提取，待补全）

根因

（待补充）

修复

HeartMuLa - Open-Source Music Generation

Overview

HeartMuLa is a family of open-source music foundation models (Apache-2.0) that generates music conditioned on lyrics and tags, with multilingual support. Generates full songs from lyrics + tags. Comparable to Suno for open-source. Includes:

HeartMuLa - Music language model (3B/7B) for generation from lyrics + tags
HeartCodec - 12.5Hz music codec for high-fidelity audio reconstruction
HeartTranscriptor - Whisper-based lyrics transcription
HeartCLAP - Audio-text alignment model

When to Use

User wants to generate music/songs from text descriptions
User wants an open-source Suno alternative
User wants local/offline music generation
User asks about HeartMuLa, heartlib, or AI music generation

Hardware Requirements

Minimum: 8GB VRAM with --lazy_load true (loads/unloads models sequentially)
Recommended: 16GB+ VRAM for comfortable single-GPU usage
Multi-GPU: Use --mula_device cuda:0 --codec_device cuda:1 to split across GPUs
3B model with lazy_load peaks at ~6.2GB VRAM

Installation Steps

1. Clone Repository

cd ~/  # or desired directory
git clone https://github.com/HeartMuLa/heartlib.git
cd heartlib

2. Create Virtual Environment (Python 3.10 required)

uv venv --python 3.10 .venv
. .venv/bin/activate
uv pip install -e .

3. Fix Dependency Compatibility Issues

IMPORTANT: As of Feb 2026, the pinned dependencies have conflicts with newer packages. Apply these fixes:

# Upgrade datasets (old version incompatible with current pyarrow)
uv pip install --upgrade datasets

# Upgrade transformers (needed for huggingface-hub 1.x compatibility)
uv pip install --upgrade transformers

4. Patch Source Code (Required for transformers 5.x)

Patch 1 - RoPE cache fix in src/heartlib/heartmula/modeling_heartmula.py:

In the setup_caches method of the HeartMuLa class, add RoPE reinitialization after the reset_caches try/except block and before the with device: block:

# Re-initialize RoPE caches that were skipped during meta-device loading
from torchtune.models.llama3_1._position_embeddings import Llama3ScaledRoPE
for module in self.modules():
    if isinstance(module, Llama3ScaledRoPE) and not module.is_cache_built:
        module.rope_init()
        module.to(device)

Why: from_pretrained creates model on meta device first; Llama3ScaledRoPE.rope_init() skips cache building on meta tensors, then never rebuilds after weights are loaded to real device.

Patch 2 - HeartCodec loading fix in src/heartlib/pipelines/music_generation.py:

Add ignore_mismatched_sizes=True to ALL HeartCodec.from_pretrained() calls (there are 2: the eager load in __init__ and the lazy load in the codec property).

Why: VQ codebook initted buffers have shape [1] in checkpoint vs [] in model. Same data, just scalar vs 0-d tensor. Safe to ignore.

5. Download Mod

验证