What models are currently good for running coding tasks? I just ran Qwen3-14B-Q6_K.gguf with llama.cpp on my card with 16GB of vram (+32GB ddr4), but I get really close to filling the entire vram on a single short conversation, so I am looking for some (smaller) alternatives to test.

I might throw OpenCode container in the mix next, if that is relevant information.

spoiler 
podman run --rm --replace --pull=newer \
  --name llama \
  -p 8080:8080 \
  -v ./llama_models:/models:Z \
  --device /dev/dri/card1:/dev/dri/card1 \
  --device /dev/dri/renderD128:/dev/dri/renderD128 \
  ghcr.io/ggml-org/llama.cpp:full-vulkan \
  --server \
  -m /models/Qwen3-14B-Q6_K.gguf \
  -ngl 99 \
  -fa on \
  -c 16384 \
  --temp 0.6 \
  --top-k 20 \
  --top-p 0.95 \
  --jinja \
  --host 0.0.0.0 --port 8080
  • Iced Raktajino@startrek.websiteEnglish
    5·
    17 days ago

    I run gemma4:26b in 16 GB of RAM. It’s slow on my test rig with only 2 GB VRAM but it should fit 16 GB VRAM fine. I have one of those AMD BC-250 crypto mining units setup as a gaming rig, but my plan was to also run ollama on it. gemma4:26b was the model I planned to make the default. I haven’t messed with it yet since I’m playing through my Steam catalog that was waiting for me to have a PC that could run them lol.

  • Avid Amoeba@lemmy.caEnglish
    4·
    16 days ago

    Qwen 3.6 35B. It’s A3B so it fits with space to spare for context. Just make sure you have --cpu-moe.

    • Svinhufvud@sopuli.xyzOPEnglish
      3·
      16 days ago

      Thanks! Why the --cpu-moe flag though? And can you share your parameter section (the after --server part), if you’re running llama?

      • Avid Amoeba@lemmy.caEnglish
        4·
        16 days ago

        I use model preset file but that just contains the equivalent of command line options. Here’s what I have for Qwen:

        [Qwen3.6-35B-A3B-MTP-Q4_K_XL]                                                                                                                
m = /models/Qwen3.6-35B-A3B-MTP/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf                                                                              
mmproj = /models/Qwen3.6-35B-A3B-MTP/mmproj-BF16.gguf                                                                                        
spec-type = draft-mtp                                                                                                                        
spec-draft-n-max = 2                                                                                                                         
chat-template-kwargs = {"preserve_thinking": true}                                                                                           
temp = 1.0                                                                                                                                   
top-p = 0.95                                                                                                                                 
top-k = 20                                                                                                                                   
min-p = 0.0                                                                                                                                  
presence-penalty = 1.5                                                                                                                       
repeat-penalty = 1.0

        I don’t use cpu-moe because I have enough VRAM for the whole model. If you have 16GB VRAM, you add cpu-moe which makes llama.cpp put only the active layers on the GPU. Keeps the rest of them in system RAM. Then it swaps them around as needed. The result is lower but still decent speed. On my hw, this model does 90-100tps when fully in VRAM. When using cpu-moe, I think it falls to 40-50tps, if I remember correctly.