I didn’t even know they train their own models.
Have been playing around with this on my laptop’s Radeon 780M, with 16GB of RAM assigned to the GPU. llama.cpp with Vulkan reaches a comfortable 30t/s TG with up to 1000t/s PP when running Q8_0 with unquantized KV-cache.
In the few rather simple summarisation tasks I gave it so far it performed better than expected, but of course it’s very small MoE and you shouldn’t expect too much. Looking forward to evaluating this in Pi on small agentic coding tasks.
Have some of you been playing around with it?
Is it better than Ling Tiny 3.0 8b-a1b?
That was my first instinct too, but that a1b is just so attractive for old DDR4 cpu inference. a2.5b is a bit of an awkward size when qwen 35b_a3b exists
I find the overall parameter size interesting. On 16GB VRAM the whole thing fits at an excellent quantisation without having to offload anything to system RAM and taking hits on PP & TG.
Yeah it’s a really nice size for older gaming computers. Of course it can still be beaten by smaller models from other labs, so that kinda defeats the purpose, but it’s nice in theory
According to the officially reported benchmarks, Ling wins on GPQA Diamond (73.4 vs. 64.6) and BFCLv4 (62.72 vs. 62.3). No other overlaps.
That being said, I’m neither trusting benchmarks alone nor have tested any of the two extensively enough to have a solid impression.