What local LLM-s you use?

[csendesmark]

@AusWolf
Local LLM-s are very important, I really reject the trend everything getting "cloud" based, micro$oft wants even you windows account to be online...
But after you posted 3 times, you could tell us what LLM-s you use, and maybe some performance data too!

 

[Ultron1337]

Maybe I'm a little bit behind on stuff but... Got to ask... What's the point of this for any regular home user?

It's like asking "whats the point of using your brain ?". First time in recorded history humanity has the ability to use thinking tool. It can supercharge almost any skill you have. Just as human brain can be used in nearly limitless ways, same applies to local LLM. Use it to check kids homework, use it to make your homework, help analyze scientific papers, write code for you, explain why vitamin K is good, count starts in the sky, analyze insurance offerings etc etc etc. I am not even going to pretend I know even a fraction of uses cases local LLMs will have next 10 years, but I know its going to be wild. On a level how internet changed our lives (yeah some of us grew up without internet).

 

[JWNoctis]

Maybe I'm a little bit behind on stuff but... Got to ask... What's the point of this for any regular home user?

That sounds interesting. Can you explain?

At least the larger, 70B+ models, are typically sufficiently knowledgeable that you can ask them some complicated questions and expect a reasonable and not necessarily banal and expected answer. There are things you do not want to send to commercial services, most typically personal information. Some of the latest advancements made even 70B scale models competent with illusion-shattering problems previous generations of models have difficulty with, like how many r's in strawberry, how many boys does Mary have when one of them is gross, et cetera.

Now they are usually useful for common math and programming problems when used with care, can explore human philosophy and condition quite competently, and can tell stories of some interest with the right prompt. They are also useful for getting familiar with what LLM output looked like. Half of the internet looks LLM generated these days.

Some open-weight models are capable of API use, such as those provided by the framework it is running on, including requesting web services. Usefulness of such capabilities is apparently unremarkable given other limitations, and for that matter, the state of the internet and search engine results these days. That require support by the framework the model is running on, and is usually the only time the model - note, not the framework - would access the Internet.

They can also provide some silly fun, especially the roleplay-finetuned ones, for uses where hallucination actually provides some emulation of creativity. Think of it as a text-based holodeck. Throw in an image generator and it is text and image. You typically don't want a lot of those elsewhere, as well: As with all things requiring an account, all things you put into a networked service would be recorded by the provider, and linked to you. Not everyone feel comfortable with the nothing-to-hide mentality even when they really don't, and more than a few have objections to their interactions and personal info being used to train future commercial AI models.

Personally, I've sized my setup to be able to run a "future larger model" in early 2024, which would turn out to be mistral-large-2407, 123B, quantized. The best correctness and general task performance is probably currently achieved by the LLAMA 3 70B distilled version of DeepSeek R1. Anything larger would be costly and impractical for the moment, to me. Might as well make them useful while they are there.

 

[csendesmark]

70B+ models, are typically sufficiently knowledgeable that you can ask them some complicated questions and expect a reasonable and not necessarily banal and expected answer.

Did you skip DeepSeek?

 

[AusWolf]

@AusWolf
Local LLM-s are very important, I really reject the trend everything getting "cloud" based, micro$oft wants even you windows account to be online...
But after you posted 3 times, you could tell us what LLM-s you use, and maybe some performance data too!

I'm not using anything. I didn't even know that you could run them locally until recently. I'm only trying to learn what use it is, to see whether it's something I'd want to do or not.

At least the larger, 70B+ models, are typically sufficiently knowledgeable that you can ask them some complicated questions and expect a reasonable and not necessarily banal and expected answer. There are things you do not want to send to commercial services, most typically personal information. Some of the latest advancements made even 70B scale models competent with illusion-shattering problems previous generations of models have difficulty with, like how many r's in strawberry, how many boys does Mary have when one of them is gross, et cetera.

Now they are usually useful for common math and programming problems when used with care, can explore human philosophy and condition quite competently, and can tell stories of some interest with the right prompt. They are also useful for getting familiar with what LLM output looked like. Half of the internet looks LLM generated these days.

Some open-weight models are capable of API use, such as those provided by the framework it is running on, including requesting web services. Usefulness of such capabilities is apparently unremarkable given other limitations, and for that matter, the state of the internet and search engine results these days. That require support by the framework the model is running on, and is usually the only time the model - note, not the framework - would access the Internet.

They can also provide some silly fun, especially the roleplay-finetuned ones, for uses where hallucination actually provides some emulation of creativity. Think of it as a text-based holodeck. Throw in an image generator and it is text and image. You typically don't want a lot of those elsewhere, as well: As with all things requiring an account, all things you put into a networked service would be recorded by the provider, and linked to you. Not everyone feel comfortable with the nothing-to-hide mentality even when they really don't, and more than a few have objections to their interactions and personal info being used to train future commercial AI models.

Personally, I've sized my setup to be able to run a "future larger model" in early 2024, which would turn out to be mistral-large-2407, 123B, quantized. The best correctness and general task performance is probably currently achieved by the LLAMA 3 70B distilled version of DeepSeek R1. Anything larger would be costly and impractical for the moment, to me. Might as well make them useful while they are there.

Text-based holodeck running locally on your PC... Now that caught my attention!

I'm just having a hard time imagining it. LLM still lives in my head as a glorified search engine.

 

[csendesmark]

I'm not using anything.

It is "current year"
Try starting here: https://lmstudio.ai

 

[JWNoctis]

Did you skip DeepSeek?

The way they do chain-of-thought makes for interesting reading. I think they are the first to do it well enough in an open-weight model, too. Wherever they might be from, I do not have quite sufficient trust to send any hosted services anything confidential or profiling.

Even "free" services come with the implicit permission of using your interaction for any number of further purposes buried in the user agreement assuming it is followed, and God forbid if there is a data breach.

FWIW 70B Q6_k quantized models are a bit more than ~0.9 token/s to almost 1.2 token/s on my setup running on official distribution of Ollama 0.5.7. Latest llama.cpp compiled from source gives ~1.2 token/s.

Text-based holodeck running locally on your PC... Now that caught my attention!

I'm just having a hard time imagining it. LLM still lives in my head as a glorified search engine.

To be fair, they are still even worse than that when used for factual stuff without verification. And whatever it is that the various search engines are integrating, they certainly aren't doing it quite right yet.

They do have uses where even current models could play to their strengths though, and even the smaller models have the superhuman passing familiarity with almost everything anyone would - or could - ever have seen in text on a computer display. As long as you don't try something too unusual, they'd often do fine.

 

[The King]

Maybe I'm a little bit behind on stuff but... Got to ask... What's the point of this for any regular home user?

This video helped me get started with running llm locally. It should also answer many of the questions you raised.

 

[csendesmark]

This video helped me get started with running llm locally. It should also answer many of the questions you raised.

OMG, this scare on the first seconds of the video....
instant downvote from me for these kind of "content"
There is no worries when you run it locally on your local program.
I would never install DeepSeek's app on my phone tho...

 

[ShiBDiB]

I'm still fairly new to the scene, but have a phi-4 and deepseek-r1 instance locally. Right now just use them when I run into coding issues or need some inspiration. Was using a lot of Grok to fill that role, but the local stuff is neat.