I didn’t even know they train their own models.
Have been playing around with this on my laptop’s Radeon 780M, with 16GB of RAM assigned to the GPU. llama.cpp with Vulkan reaches a comfortable 30t/s TG with up to 1000t/s PP when running Q8_0 with unquantized KV-cache.
In the few rather simple summarisation tasks I gave it so far it performed better than expected, but of course it’s very small MoE and you shouldn’t expect too much. Looking forward to evaluating this in Pi on small agentic coding tasks.
Have some of you been playing around with it?
I find the overall parameter size interesting. On 16GB VRAM the whole thing fits at an excellent quantisation without having to offload anything to system RAM and taking hits on PP & TG.
Yeah it’s a really nice size for older gaming computers. Of course it can still be beaten by smaller models from other labs, so that kinda defeats the purpose, but it’s nice in theory