I can run MiMo 2.5 at about 9 tokens/sec, on 128GB RAM, a 7800 and a 3090 in an SFF rig. That’s a fantastic 310B model. I’m requantizing it right now, to see if I can speed it up with Dflash.
Still fantastic models can be run on 64GB or 32GB CPU RAM, as long as you have some GPU. We’re awash in sparse models these days.
I’ve got 128GB RAM + 24GB VRAM on a 4090. I’ve managed to get a 400B parameter model running on a single board computer with 64GB RAM by using MMAP. But I want to run Kimi K3 locally so I would need a lot more RAM / bandwidth
You can’t realistically run Kimi (as it’s a 1T+ model), but you’re set. There’s a glut of excellent 120B-300B models for you to choose from.
I’m quantizing MiMo 2.5 specifically to cram into 24GB/128GB as I type this. It’s tight, and will use up most of your memory, but it’s fantastically smart and plenty fast. The quantization won’t finish cooking until tomorrow, but I’ll upload it to huggingface then.
It’s specifically optimized for hybrid (CPU + GPU) inference on Nvidia desktops; you’ll get MUCH faster speeds than mainline llama.cpp or anything based on it. It also supports some more exotic quantization type; as an example, I’m quantizing MiMo 2.5 as a hybrid quant, with the dense layers at IQ6K/Q8_0 and the sparse experts as an IQ3_KT “trellis” quant type. This should yield a higher fidelity quantization than a typical Q3 GGUF while taking less RAM, at the cost of taking forever to quantize and a slight speed hit.
Some other quantization types (like the KS or R4 types) are specifically configured to be fast on CPU.
There are specialized “quant cookers” that make GGUFs specifically for ik_llama.cpp, like:
For anyone else reading this: none of this is applicable to you if you have an AMD/Intel GPU, or an older Nvidia GPU, or less than a certain amount of RAM, or a non AVX2 CPU or… well, there’s all sorts of caveats.
The optimal runtime is different for everyone. As an example, exllamav3 is WAY better than llama.cpp on modern Nvidia GPUs until you get above a certain amount of CPU RAM; then using all that RAM for hybrid inference makes more sense. On AMD, different backends work better on different GPUs, and… well, you get the point. Basically all LLM running advice is irrelevant without specifics of your hardware, even this post will be obsolete in a month.
Same, getting ~3 trillion parameters in consumer hardware is rough.
If Nvidia has any foresight they’ll see the writing on the wall and start getting higher memory Spark style SMB inference machines, few people in the long run are going to pay retail API token costs,
I just wish I could buy enough memory to run one of these models locally. Specially Kimi K3
How much RAM do you have?
I can run MiMo 2.5 at about 9 tokens/sec, on 128GB RAM, a 7800 and a 3090 in an SFF rig. That’s a fantastic 310B model. I’m requantizing it right now, to see if I can speed it up with Dflash.
Still fantastic models can be run on 64GB or 32GB CPU RAM, as long as you have some GPU. We’re awash in sparse models these days.
I’ve got 128GB RAM + 24GB VRAM on a 4090. I’ve managed to get a 400B parameter model running on a single board computer with 64GB RAM by using MMAP. But I want to run Kimi K3 locally so I would need a lot more RAM / bandwidth
Oh that’s perfect!
You can’t realistically run Kimi (as it’s a 1T+ model), but you’re set. There’s a glut of excellent 120B-300B models for you to choose from.
I’m quantizing MiMo 2.5 specifically to cram into 24GB/128GB as I type this. It’s tight, and will use up most of your memory, but it’s fantastically smart and plenty fast. The quantization won’t finish cooking until tomorrow, but I’ll upload it to huggingface then.
But there are others existing quants would fit, like Deepseek Flash IQ3_S: https://huggingface.co/unsloth/DeepSeek-V4-Flash-GGUF/tree/main/UD-IQ3_S
Or… well, any of these! https://huggingface.co/models?num_parameters=min%3A128B%2Cmax%3A256B&library=gguf&sort=modified
The key for you is to run the ik_llama.cpp fork: https://github.com/ikawrakow/ik_llama.cpp/
It’s specifically optimized for hybrid (CPU + GPU) inference on Nvidia desktops; you’ll get MUCH faster speeds than mainline llama.cpp or anything based on it. It also supports some more exotic quantization type; as an example, I’m quantizing MiMo 2.5 as a hybrid quant, with the dense layers at IQ6K/Q8_0 and the sparse experts as an IQ3_KT “trellis” quant type. This should yield a higher fidelity quantization than a typical Q3 GGUF while taking less RAM, at the cost of taking forever to quantize and a slight speed hit.
Some other quantization types (like the KS or R4 types) are specifically configured to be fast on CPU.
There are specialized “quant cookers” that make GGUFs specifically for ik_llama.cpp, like:
https://huggingface.co/ubergarm
https://huggingface.co/AesSedai
https://huggingface.co/sigargv/Laguna-M.1-GGUF
https://huggingface.co/models?other=ik_llama.cpp&sort=modified
Mine will be here once I upload it:
https://huggingface.co/Downtown-Case
For anyone else reading this: none of this is applicable to you if you have an AMD/Intel GPU, or an older Nvidia GPU, or less than a certain amount of RAM, or a non AVX2 CPU or… well, there’s all sorts of caveats.
The optimal runtime is different for everyone. As an example, exllamav3 is WAY better than llama.cpp on modern Nvidia GPUs until you get above a certain amount of CPU RAM; then using all that RAM for hybrid inference makes more sense. On AMD, different backends work better on different GPUs, and… well, you get the point. Basically all LLM running advice is irrelevant without specifics of your hardware, even this post will be obsolete in a month.
Same, getting ~3 trillion parameters in consumer hardware is rough.
If Nvidia has any foresight they’ll see the writing on the wall and start getting higher memory Spark style SMB inference machines, few people in the long run are going to pay retail API token costs,
Especially giving the lack of trust organizations rightfully have in big AI companies
Exactly, having the mustard toddler pull fable because of, who knows? That burned trust in the west’s AI companies.
And there is also the question of what they’re doing with the data you send into the models.