The conversion step was the product. Hugging Face is trying to delete it.
Hugging Face’s blog, “Transformers now runs llama.cpp quants”, is a library note with a laptop consequence. Pick a GGUF on the Hub, pass model_id and gguf_file into from_pretrained, generate on the machine you already have. To make that pleasant, they reuse ggml kernels through a kernels library and cut overhead in generate. First target: local inference on Apple Silicon, starting with the Qwen3.5 architecture.
Analytics Insight’s September 27 tools roundup is the dated breadcrumb: on September 22, Transformers added support for GGUF models through llama.cpp quantizations. Treat that as the calendar stamp, not as a benchmark. explainx.ai’s recap is the interpretation layer, with a disclaimer that they did not independently verify speed.
We already compared vLLM, TensorRT-LLM, and ONNX Runtime. That piece was servers. This one is the Python process on a Mac, and whether you still have to leave Python to get the quantized file.
What the blog actually shipped
GGUF is llama.cpp’s format. The llama.cpp team publishes quantized checkpoints under ggml-org. Unsloth, LM Studio Community, and bartowski publish more. Hugging Face says those GGUF models have been downloaded millions of times. The new work is not a new format. It is Transformers agreeing to run the files people already fetch.
from_pretrained is the API they want you to keep. If you fine-tune in Transformers, you already live there. The old path was: train, export, quantize, run llama.cpp or a wrapper UI. The new path, for the architectures they have wired, is: load the GGUF in the same process. That only matters if it is fast enough that you do not immediately export anyway.
They say compatibility is useless if the model is unpleasant, so they reuse ggml’s kernels rather than reimplementing a slow Python loop. Initial focus is Apple Silicon. Metal is in the table of contents (“Reusing ggml’s Metal kernels”). There is a section on keeping CPU and GPU working together. There is a limitations section. Read that before you rewrite a service.
Qwen3.5 is the first architecture named. That is a narrower launch than “Transformers runs GGUF now” in a Slack title. If your checkpoint is not in the first wave, you are still on the old path. Architecture support is a checklist, not a slogan. The blog is explicit that each architecture still needs integration and validation, and that the initial GGUF examples are text generation.
Kernels operate on tensors. They do not require the whole model to arrive as a GGUF file. Hugging Face flags a later path: vision, audio, multimodal models reusing attention, normalization, and matmul kernels without a full llama.cpp implementation. That is a roadmap sentence. It is not a date. Do not write a multimodal milestone around it.
The speed claim is a recap, not a llama-bench paste
explainx.ai’s post is dated around September 23 in their own “as of” line. The claim: Transformers closed the performance gap with llama.cpp for GGUF. The consequence they care about: a Python app that wanted Hub + tokenizers + training no longer has to eject to llama.cpp for the decode loop.
They also say the thing Hugging Face cannot say as loudly. llama.cpp remains a small C/C++ binary for machines that should not grow a Python dependency tree. Embedded, constrained edge, anything where RAM is the product. It remains the reference implementation for GGUF. A lot of local UIs speak llama.cpp because of that, not because they hate transformers.
explainx.ai notes they did not independently verify the speed claim. So neither should your slide. If this is a production decision, run llama-bench against generate on the same Qwen3.5 GGUF, same Mac, same prompt length. Hugging Face put “Benchmarking against llama.cpp” in the blog’s own outline. Use their section, then run yours.
Until you have those numbers, the true shipping change is operational: one process, one tokenizer stack, no export script for the architectures they enabled. That is already worth a day of engineering if your team bleeds time on conversion. It is not worth a rewrite if your production box never had Python.
September 22 was a packaging week, not a model week
Analytics Insight stuffed three dates into one listicle.
Transformers GGUF support: September 22. That is this article.
JupyterLab 4.6.4: September 22, security fixes. Notebook 7.6.3: security release, same week. If you run notebooks on a shared server, that is the patch you apply before you care about GGUF. A local-LLM demo on an unpatched Jupyter is a side channel with extra steps.
TensorFlow 2.22.0 RC: September 22, with 2.21.0 back in March. Mentioned here only so you do not confuse a TensorFlow RC with a Hugging Face runtime change. Different trees.
The listicle also waves at PyTorch 2.14 from September 2, which is outside this week’s window and already had its own release blog. If you are on 2.14 because of Inductor and distributed, that is a different Monday. vLLM’s speed-of-light talk is still the server-side number to argue with. GGUF-in-Transformers is the laptop-side API to argue with.
Safetensors as the default weight container did not go away. Liquid’s draft model this week even ships Safetensors and GGUF side by side. Transformers learning GGUF does not mean you should start saving training runs as GGUF. GGUF is a quantized inference file. Safetensors is still how you hand a trainer a weight dump. Mixing those jobs is how you get a one-star GitHub issue.
What from_pretrained buys you in a real repo
A typical Python demo that “just runs a GGUF” is a pile of subprocess calls into llama-cli, plus a second tokenizer you hope matches. Tests flake when the binary is missing. CI images grow a compiler. Hugging Face’s loader puts the quantized weights behind the same call you already mock in unit tests. That is the engineering win even if llama-bench later shows a 5 percent gap.
It also puts you back on PyTorch’s memory allocator, Python’s GIL, and whatever CUDA or Metal build you installed. llama.cpp’s pitch was that you could avoid that. If your Mac already has a working transformers environment, you are not avoiding it. You are reusing it. If your Mac does not, installing PyTorch to run one Qwen3.5 GGUF is a worse hello-world than brew install llama.cpp.
Quant types will be the first footgun. bartowski and Unsloth publish a supermarket of Q4_K_M, IQ2, and whatever landed this month. The blog says they reuse ggml kernels. It does not say every quant type on the Hub is wired. Start with the file names Hugging Face uses in their own examples. When a load throws, that is a kernel gap, not a reason to write a tweet about the feature being fake.
Serving is a separate heading on their page: “Serve GGUF with your preferred interface.” Read that as “we know you have a server already.” It is not a promise that Transformers is now vLLM. If you need continuous batching, you are still in the engine comparison we already published. If you need a single-user notebook, generate is the interface.
Metal on Apple Silicon is the reason this launched there first. ggml’s Metal kernels already exist because llama.cpp had to run on M-series machines without a CUDA story. Transformers borrowing them is the unglamorous version of “we are not rewriting GPU code.” Good. Borrowed kernels still need a thread budget. The blog’s CPU-plus-GPU section is the warning that unified memory is not an excuse to ignore copies.
Do not fine-tune into GGUF as a training format. Train, export safetensors, then quantize if you must. Loading GGUF for inference in the same Python process is the new trick. Saving GGUF from a half-finished trainer is how you corrupt a weekend.
When you should still call llama.cpp
Call it when there is no Python. A Pi, a kiosk, a sidecar written in Go that shells out to llama-server, a desktop app that already embeds llama.cpp. Transformers will not make those smaller.
Call it when your architecture is not in the first Transformers GGUF wave. Qwen3.5 on Apple Silicon is the documented happy path. Mystery GGUF from a Discord quant is not.
Call it when the UI you already standardized on is LM Studio or a llama.cpp server and nobody on the team wants to own a PyTorch runtime. “We can do it in Transformers now” is not a migration plan. It is an option.
Call it when you have already measured and Transformers loses. The recap says parity. Parity is a claim. Claims die in llama-bench.
Stay in Transformers when the rest of the job is Python. Retrieval, evaluation, a FastAPI process that already imports transformers, a fine-tune that needs to smoke-test the quantized file without a second toolchain. That is the conversion step dying. That is the blog’s actual gift.
Stay in vLLM or a dedicated server engine when the job is many concurrent users. A Mac running generate is not a scheduler. We already had that argument in the inference-engine comparison. Do not reopen it because GGUF got a Python loader.
A Monday checklist
Print the architecture list from the Hugging Face blog, not from a screenshot of Analytics Insight. If your model is on it, load one GGUF with from_pretrained on the Mac you actually own. Time a short prompt and a long prompt. Time the same files in llama.cpp. Write the two numbers in the PR, not “feels similar.”
If your model is not on the list, do not open a rewrite ticket. Open a watch. Architecture support will land as follow-up work. The blog said that.
Patch Jupyter if you have not. The GGUF loader does not make a 4.6.4 security release less true.
Keep llama.cpp installed. You will still need it for the box that cannot pip install. You will still need it for the UI that never heard of PyTorch. You will still need it the first time Transformers throws on a quant type the kernels do not wrap.
If you maintain a Hub space that already shells out to llama.cpp, do not rip it out this week. Add a branch that loads the same gguf_file through Transformers on a Mac runner and compare tokens per second plus RSS. If Transformers wins and the dependency set is already on the runner, delete the subprocess. If it loses, you spent an afternoon and kept the binary. That is a cheaper experiment than a rewrite announced in Slack.
Watch the acknowledgments and the limitations on the Hugging Face page for the next architecture name after Qwen3.5. The second name is when this stops being a preview. Until then, document it as “Qwen3.5 GGUF on Apple Silicon in Transformers,” not as “we standardized on GGUF-in-Python.”
The story is not that llama.cpp lost. The story is that Python got a way to read the files llama.cpp already won. Use that on the laptop. Leave the C++ where the laptop is not.
Discussion
Leave a comment
No comments yet
Be the first to start the conversation.