• Welcome to TechPowerUp Forums, Guest! Please check out our forum guidelines for info related to our community.

What local LLM-s you use?

johnspack

johnspack

Here For Good!
Joined
Oct 6, 2007
Messages
6,055 (0.95/day)
Location
Nelson B.C. Canada
System Specs
System Name System2 Blacknet , System1 Blacknet2
Processor System2 Threadripper 1920x, System1 2699 v3
Motherboard System2 Asrock Fatality x399 Professional Gaming, System1 Asus X99-A
Cooling System2 Noctua NH-U14 TR4-SP3 Dual 140mm fans, System1 AIO
Memory System2 64GBS DDR4 3000, System1 32gbs DDR4 2400
Video Card(s) System2 GTX 980Ti System1 GTX 970
Storage System2 4x SSDs + NVme= 2.250TB 2xStorage Drives=8TB System1 3x SSDs=2TB
Display(s) 1x27" 1440 display 1x 24" 1080 display
Case System2 Some Nzxt case with soundproofing...
Audio Device(s) Asus Xonar U7 MKII
Power Supply System2 EVGA 750 Watt, System1 XFX XTR 750 Watt
Mouse Logitech G900 Chaos Spectrum
Keyboard Ducky
Software Archlinux, Manjaro, Win11 Ent 24h2
Benchmark Scores It's linux baby!
I actually have that model, but would like to go up a bit, maybe q8? I also see llama 70b. But don't see any download links....
I have to find models that will fit in 64gbs of ram.
 
tpa-pr

tpa-pr

Joined
Jan 12, 2023
Messages
276 (0.36/day)
System Specs
System Name IZALITH (or just "Lith")
Processor AMD Ryzen 7 7800X3D (4.2Ghz base, 5.0Ghz boost, -30 PBO offset)
Motherboard Gigabyte X670E Aorus Master Rev 1.0
Cooling Deepcool Gammaxx AG400 Single Tower
Memory Corsair Vengeance 64GB (2x32GB) 6000MHz CL40 DDR5 XMP (XMP enabled)
Video Card(s) PowerColor Radeon RX 7900 XTX Red Devil OC 24GB (2.39Ghz base, 2.99Ghz boost, -30 core offset)
Storage 2x1TB SSD, 2x2TB SSD, 2x 8TB HDD
Display(s) Samsung Odyssey G51C 27" QHD (1440p 165Hz) + Samsung Odyssey G3 24" FHD (1080p 165Hz)
Case Corsair 7000D Airflow Full Tower
Audio Device(s) Corsair HS55 Surround Wired Headset/LG Z407 Speaker Set
Power Supply Corsair HX1000 Platinum Modular (1000W)
Mouse Logitech G502 X LIGHTSPEED Wireless Gaming Mouse
Keyboard Keychron K4 Wireless Mechanical Keyboard
Software Arch Linux
Divide Overflow said:
Get your own data center cards and leave my gaming GPUs alone!
Click to expand...
Never fear friend, my card is primarily for gaming. The AI stuff is just for experimenting from time-to-time :)

Anyway, I decided to download llama3.3, but unfortunately I don't have the VRAM to run it. It maxed out my VRAM and any responses were INCREDIBLY slow. So I suspect i'll need to stick to smaller models.
 
csendesmark

csendesmark

Joined
Mar 11, 2008
Messages
1,100 (0.18/day)
Location
Hungary / Budapest
System Specs
System Name Kincsem
Processor AMD Ryzen 9 9950X
Motherboard ASUS ProArt X870E-CREATOR WIFI
Cooling Be Quiet Dark Rock Pro 5
Memory Kingston Fury KF560C32RSK2-96 (2×48GB 6GHz)
Video Card(s) Sapphire AMD RX 7900 XT Pulse
Storage Samsung 990PRO 2TB + Samsung 980PRO 2TB + FURY Renegade 2TB+ Adata 2TB + WD Ultrastar HC550 16TB
Display(s) Acer QHD 27"@144Hz 1ms + UHD 27"@60Hz
Case Cooler Master CM 690 III
Power Supply Seasonic 1300W 80+ Gold Prime
Mouse Logitech G502 Hero
Keyboard HyperX Alloy Elite RGB
Software Windows 10-64
Benchmark Scores https://valid.x86.fr/9qw7iq https://valid.x86.fr/4d8n02 X570 https://www.techpowerup.com/gpuz/g46uc
johnspack said:
I actually have that model, but would like to go up a bit, maybe q8? I also see llama 70b. But don't see any download links....
I have to find models that will fit in 64gbs of ram.
Click to expand...
It is all there:
DeepSeek-R1-Distill-Qwen-32B-Q8_0
Llama-3.3-70B-Instruct-GGUF

Alternatively you can download it with LM Studio like this:
1740048755321.png

Super convenient :toast:
 
Last edited:
johnspack

johnspack

Here For Good!
Joined
Oct 6, 2007
Messages
6,055 (0.95/day)
Location
Nelson B.C. Canada
System Specs
System Name System2 Blacknet , System1 Blacknet2
Processor System2 Threadripper 1920x, System1 2699 v3
Motherboard System2 Asrock Fatality x399 Professional Gaming, System1 Asus X99-A
Cooling System2 Noctua NH-U14 TR4-SP3 Dual 140mm fans, System1 AIO
Memory System2 64GBS DDR4 3000, System1 32gbs DDR4 2400
Video Card(s) System2 GTX 980Ti System1 GTX 970
Storage System2 4x SSDs + NVme= 2.250TB 2xStorage Drives=8TB System1 3x SSDs=2TB
Display(s) 1x27" 1440 display 1x 24" 1080 display
Case System2 Some Nzxt case with soundproofing...
Audio Device(s) Asus Xonar U7 MKII
Power Supply System2 EVGA 750 Watt, System1 XFX XTR 750 Watt
Mouse Logitech G900 Chaos Spectrum
Keyboard Ducky
Software Archlinux, Manjaro, Win11 Ent 24h2
Benchmark Scores It's linux baby!
Thanks yeah I finally found more downloads. Right now I have to use Koboldcpp, and it doesn't have the download feature. LMStudio was failing on me, so I switched.
Although after some time the model f's up, but in Kobold I just use start new session and it clears up.
Yep, now have DeepSeek-R1-Distill-Qwen-32B-Q8_0 running just fine. Not bad for an ancient computer!
Oh and Q8 is using around 35gbs of ram.

It's a bit slow... not really using my gpu as much as I'd like:
1740111467983.png
 
Last edited:
R

Rover4444

Joined
Nov 23, 2023
Messages
72 (0.16/day)
johnspack said:
Thanks yeah I finally found more downloads. Right now I have to use Koboldcpp, and it doesn't have the download feature. LMStudio was failing on me, so I switched.
Although after some time the model f's up, but in Kobold I just use start new session and it clears up.
Yep, now have DeepSeek-R1-Distill-Qwen-32B-Q8_0 running just fine. Not bad for an ancient computer!
Oh and Q8 is using around 35gbs of ram.

It's a bit slow... not really using my gpu as much as I'd like:
View attachment 385850
Click to expand...
The more layers you can put on VRAM the faster it'll perform. Use the Q4 quants or check how much of your VRAM is being used.
 
R-T-B

R-T-B

Supporter
Joined
Aug 20, 2007
Messages
21,800 (3.41/day)
Location
Olympia, WA
System Specs
System Name Pioneer
Processor Ryzen 9 9950X
Motherboard MSI MAG X670E Tomahawk Wifi
Cooling Noctua NH-D15 + A whole lotta Sunon, Phanteks and Corsair Maglev blower fans...
Memory 128GB (4x 32GB) G.Skill Flare X5 @ DDR5-4000 (Running 1:1:1 to FCLK)
Video Card(s) XFX RX 7900 XTX Speedster Merc 310
Storage Intel 5800X Optane 800GB boot, +2x Crucial P5 Plus 2TB PCIe 4.0 NVMe SSDs, 1x 2TB Seagate Exos 3.5"
Display(s) 55" LG 55" B9 OLED 4K Display
Case Thermaltake Core X31
Audio Device(s) TOSLINK->Schiit Modi MB->Asgard 2 DAC Amp->AKG Pro K712 Headphones or HDMI->B9 OLED
Power Supply FSP Hydro Ti Pro 850W
Mouse Logitech G305 Lightspeed Wireless
Keyboard WASD Code v3 with Cherry Green keyswitches + PBT DS keycaps
Software Gentoo Linux x64
Divide Overflow said:
Get your own data center cards and leave my gaming GPUs alone!
Click to expand...

Why not both? These guys likely game too given the audience here. You are being mad at the wrong group.
 
csendesmark

csendesmark

Joined
Mar 11, 2008
Messages
1,100 (0.18/day)
Location
Hungary / Budapest
System Specs
System Name Kincsem
Processor AMD Ryzen 9 9950X
Motherboard ASUS ProArt X870E-CREATOR WIFI
Cooling Be Quiet Dark Rock Pro 5
Memory Kingston Fury KF560C32RSK2-96 (2×48GB 6GHz)
Video Card(s) Sapphire AMD RX 7900 XT Pulse
Storage Samsung 990PRO 2TB + Samsung 980PRO 2TB + FURY Renegade 2TB+ Adata 2TB + WD Ultrastar HC550 16TB
Display(s) Acer QHD 27"@144Hz 1ms + UHD 27"@60Hz
Case Cooler Master CM 690 III
Power Supply Seasonic 1300W 80+ Gold Prime
Mouse Logitech G502 Hero
Keyboard HyperX Alloy Elite RGB
Software Windows 10-64
Benchmark Scores https://valid.x86.fr/9qw7iq https://valid.x86.fr/4d8n02 X570 https://www.techpowerup.com/gpuz/g46uc
Rover4444 said:
The more layers you can put on VRAM the faster it'll perform. Use the Q4 quants or check how much of your VRAM is being used.
Click to expand...
Or the other way,
It is more and more painful if you put more and more layers into your system RAM :roll:
R-T-B said:
Why not both? These guys likely game too given the audience here. You are being mad at the wrong group.
Click to expand...
Yeah!
PC is a general computer it can do it all,
You can load and unload those programs on demand! :toast:

Wanted to try out yesterday but did not have the smirki/UIGEN-T1-Qwen-7b is doing good job with match problems, with language, not that great.
And it is quite fast with 74 token/s for me.
 
You must log in or register to reply here.
Top