Skip to content

skill-harvest-llama-cpp.md

Latest commit

 

History

History
102 lines (76 loc) · 3.21 KB

skill-harvest-llama-cpp.md

File metadata and controls

102 lines (76 loc) · 3.21 KB
Raw
domain archive
title Llama Cpp
verification metadata-normalized
source skill-pipeline
status draft
created 2026-05-09

背景

（此 lesson 从 skill llama-cpp 自动提取，待补全）

根因

（待补充）

修复

llama.cpp + GGUF

Use this skill for local GGUF inference, quant selection, or Hugging Face repo discovery for llama.cpp.

When to use

  • Run local models on CPU, Apple Silicon, CUDA, ROCm, or Intel GPUs
  • Find the right GGUF for a specific Hugging Face repo
  • Build a llama-server or llama-cli command from the Hub
  • Search the Hub for models that already support llama.cpp
  • Enumerate available .gguf files and sizes for a repo
  • Decide between Q4/Q5/Q6/IQ variants for the user's RAM or VRAM

Model Discovery workflow

Prefer URL workflows before asking for hf, Python, or custom scripts.

  1. Search for candidate repos on the Hub:
    • Base: https://huggingface.co/models?apps=llama.cpp&sort=trending
    • Add search=<term> for a model family
    • Add num_parameters=min:0,max:24B or similar when the user has size constraints
  2. Open the repo with the llama.cpp local-app view:
    • https://huggingface.co/<repo>?local-app=llama.cpp
  3. Treat the local-app snippet as the source of truth when it is visible:
    • copy the exact llama-server or llama-cli command
    • report the recommended quant exactly as HF shows it
  4. Read the same ?local-app=llama.cpp URL as page text or HTML and extract the section under Hardware compatibility:
    • prefer its exact quant labels and sizes over generic tables
    • keep repo-specific labels such as UD-Q4_K_M or IQ4_NL_XL
    • if that section is not visible in the fetched page source, say so and fall back to the tree API plus generic quant guidance
  5. Query the tree API to confirm what actually exists:
    • https://huggingface.co/api/models/<repo>/tree/main?recursive=true
    • keep entries where type is file and path ends with .gguf
    • use path and size as the source of truth for filenames and byte sizes
    • separate quantized checkpoints from mmproj-*.gguf projector files and BF16/ shard files
    • use https://huggingface.co/<repo>/tree/main only as a human fallback
  6. If the local-app snippet is not text-visible, reconstruct the command from the repo plus the chosen quant:
    • shorthand quant selection: llama-server -hf <repo>:<QUANT>
    • exact-file fallback: llama-server --hf-repo <repo> --hf-file <filename.gguf>
  7. Only suggest conversion from Transformers weights if the repo does not already expose GGUF files.

Quick start

Install llama.cpp

# macOS / Linux (simplest)
brew install llama.cpp
winget install llama.cpp
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build
cmake --build build --config Release

Run directly from the Hugging Face Hub

llama-cli -hf bartowski/Llama-3.2-3B-Instruct-GGUF:Q8_0
llama-server -hf bartowski/Llama-3.2-3B-Instruct-GGUF:Q8_0

Run an exact GGUF file from the Hub

Use this when the tree API shows custom file naming or the exact HF snippet is missing.

llama-server \
    --hf-repo microsoft/Phi-3-mini-4k-instruct-gguf

## 验证

（待补充）