What local LLM-s you use?

[csendesmark]

Hello Forum,
I occasionally run LLM-s locally with LM Studio.
And I mostly use
DeepSeek-R1-Distill-Qwen-14B-GGUF (Q8_0) ~42 token/s
phi-4-GGUF (Q8_0) ~42 token/s
DeepSeek-R1-Distill-Qwen-32B-GGUF (Q6_K) ~3,2 token/s
Llama-3.3-70B-Instruct-GGUF (4Q_K_M) ~1,9 token/s
Using DeepSeek for logical and math problems while Phi and Llama for language question usually.
Really curious what do you use and how fast is it on your rig.

 

[Ultron1337]

I've used many models from 1B tinyllama to mixtral:8x22b.
I've found Phi4:14B and Gemma2:27B most useful.
My 7800XT runs these models Q4_K at these speeds:
phi4:27B - 42 tokens/s
gemma2:27B - 8.5 tokens/s

For reference
llama3.3:70B - 1.5 tokens/s painfully slow
llama3.2:3.2B - 115 tokens/s, super fast, but super dumb

I use ollama for running models, it has nice API that can be used with special Python module. I run Open WebUI in my QNAP container, that is connected to my models via same API. Another QNAP container runs nginx reverse proxy, to make the WebUI connection secure (HTTP->HTTPS).

I must admit much bigger online models like GPT-4o, Gemini 2 Pro etc are obviously superior to local ones, but the local ones get surprising amount stuff done and are fun to play with.

 

[Easy Rhino]

I am running ollama here on my Fedora 41 desktop with an AMD RX 6750 XT. I was able to pass the Environment variable that enables that GPU despite it not directly being supported. I mostly am running deepseek-coder-v2 for AI chat inside VSCodium and Qwen2.5-coder:1.5 for code completion. I get 75-80 tokens per second which is pretty good.

edit: i get 83 tps with llama3.2:3b

 

[csendesmark]

Made a second round with the same models but with a fresh start not opening any other programs
I am not sure why my system get this "slow" regarding AI
DS 14B Q8 from 42 -> 73 token/s
Phi4 Q8 from 42 still 42 token/s
DS 32B Q6 from 3.2 to 3.3 token/s
Llama3.3 70B Q3 from 1.9 to 2.6 token/s
I understand that the browser using some VRAM, but how is that LM Studio not using priority and offloading data not currently in use to system memory?

phi4:27B - 42 tokens/s

Same on mine, even with fresh start - strange
Llama 3.3 performance also looks strange

 

[Easy Rhino]

Made a second round with the same models but with a fresh start not opening any other programs
I am not sure why my system get this "slow" regarding AI
DS 14B Q8 from 42 -> 73 token/s
Phi4 Q8 from 42 still 42 token/s
DS 32B Q6 from 3.2 to 3.3 token/s
Llama3.3 70B Q3 from 1.9 to 2.6 token/s
I understand that the browser using some VRAM, but how is that LM Studio not using priority and offloading data not currently in use to system memory?

Same on mine, even with fresh start - strange
Llama 3.3 performance also looks strange

Try llama3.2:3b and see what you get?

 

[Ultron1337]

Llama3.3 70B Q3 from 1.9 to 2.6 token/s

Llama 3.3 performance also looks strange

That model needs about 42GB of VRAM to run properly on a PC. A lot of model is loaded into about 10x slower system RAM, so performance falls kinda off the cliff here. Fluctuations in performance are to be expected, depending if answer came mostly from VRAM or RAM.
Measuring LLM TPS always varies a bit. If question is complex and requires long answer it takes LLM longer. If LLM temperature is set to creative, it will give different answers to same question, making exact measuring complicated.

Phi and Llama for language question usually.

Phi4 and low level Llamas are terrible at my native language.

Last years I've not been much of a gamer, but with LLMs I am tempted to upgrade my GPU to 24GB 7900XTX or 32GB version of 9070XT if that rumor comes alive. All current nVidia offerings seem to be low bang for the buck options from LLM perspective, but GB10 based desktop computer is very interesting.

 

[csendesmark]

Try llama3.2:3b and see what you get?

I did run Llama-3.2-3B-Instruct-GGUF chosed the
Q8_0 quantization scoring 136 token/s
Q4_K_M 154 token/s
Which quantization did you choose? It is also important as you can see on the values.

 

[Ultron1337]

I did run Llama-3.2-3B-Instruct-GGUF chosed the
Q8_0 quantization scoring 136 token/s
Q4_K_M 154 token/s
Which quantization did you choose? It is also important as you can see on the values.

https://ollama.com/library/llama3.2:3b is Q4_K_M

 

[DR4G00N]

Tried playing around with LM studio a bit with my 3060 6GB Laptop GPU & Instinct MI25 16GB together. It was a lot easier to set up than I expected it to be.

Performance on the smaller model is quite lacking with the MI25 because the 3060 can't use it's Tensor cores I think. The 22GB VRAM pool makes up for that on larger models though.

Vulkan (3060 + MI25)
DS Qwen 7B 4Q_K_M ~20t/s
DS Qwen 32B 4Q_K_M ~8t/s

CUDA (3060 only)
DS Qwen 7B 4Q_K_M ~43t/s
DS Qwen 32B 4Q_K_M ~2.1t/s (Obviously way too big for it)