Running Local LLMs with Ollama on openSUSE Tumbleweed
-
This post did not contain any content.
-
This post did not contain any content.wrote last edited by [email protected]
I hate to drone on about this again, but:
-
ollama is getting less and less open, and (IMO) should not be used. If that doesn't concern you, you should be using LM Studio anyway.
-
The model sizes they mention are mostly for old models no-one should be using. The only exception is a 70B MoE (Hunyuan), but I think ollama doesn't even support that?
-
The quantization methods they mention are (comparatively) primitive and low performance, not cutting edge.
-
It mentions q8_0 twice, nonsensically... Um, it makes me think this article is AI slop?
I'm glad opensuse is promoting local LLM usage, but please... not ollama, and be more specific.
And don't use ollama to write it without checking
-
-
I hate to drone on about this again, but:
-
ollama is getting less and less open, and (IMO) should not be used. If that doesn't concern you, you should be using LM Studio anyway.
-
The model sizes they mention are mostly for old models no-one should be using. The only exception is a 70B MoE (Hunyuan), but I think ollama doesn't even support that?
-
The quantization methods they mention are (comparatively) primitive and low performance, not cutting edge.
-
It mentions q8_0 twice, nonsensically... Um, it makes me think this article is AI slop?
I'm glad opensuse is promoting local LLM usage, but please... not ollama, and be more specific.
And don't use ollama to write it without checking
wrote last edited by [email protected]Is there any background information available on ollama becoming less open? It's marked MIT licensed in the repo of my Linux distribution and on their Github.
-
-
Is there any background information available on ollama becoming less open? It's marked MIT licensed in the repo of my Linux distribution and on their Github.
wrote last edited by [email protected]It's kinda a hundred little things all pointing in a bad direction:
https://old.reddit.com/r/LocalLLaMA/comments/1kg20mu/so_why_are_we_shing_on_ollama_again/
I would summarize it as "AI Bro" like behavior:
-
Signs in the code they are preparing a commercial version of Ollama, likely dumping the free version as a bait and switch.
-
Heavy online marketing.
-
"Reinventing"the wheel" to shut out competition, even when base llama.cpp already has it implemented, like with modelfiles and the ollama API.
-
A lot of inexplicable forked behavior.
Beyond that:
-
Misnaming models for hype reasons, like the tiny deepseek distils as "Deepseek"
-
Technical screw ups with the backend, chat templates and such hidden from users, so there's no apparent reason why models are misbehaving.
-
Not actually contributing to the core development of the engine.
-
Social media scummery.
-
Treating the user as 'dumb' by hiding things like the default hard 2048-token context window.
-
Not keeping up with technical innovations, like newer quantizations, SWA, batching, other backend stuff.
-
Bad default quantizations, even beyond the above. For instance, no Google QATs (last I checked), no imatrix, no dynamic quants.
I could go on forever about more specific dramas, and I don't even remember the half of them. But there are plenty of technical and moral reasons to stay away.
LM Studio is much better put together if you want 1-click. Truly open solutions that are more DIY (and reward you with dramatically better performance from the understanding/learning) are the way if you have the time/patience to burn.
-
-
It's kinda a hundred little things all pointing in a bad direction:
https://old.reddit.com/r/LocalLLaMA/comments/1kg20mu/so_why_are_we_shing_on_ollama_again/
I would summarize it as "AI Bro" like behavior:
-
Signs in the code they are preparing a commercial version of Ollama, likely dumping the free version as a bait and switch.
-
Heavy online marketing.
-
"Reinventing"the wheel" to shut out competition, even when base llama.cpp already has it implemented, like with modelfiles and the ollama API.
-
A lot of inexplicable forked behavior.
Beyond that:
-
Misnaming models for hype reasons, like the tiny deepseek distils as "Deepseek"
-
Technical screw ups with the backend, chat templates and such hidden from users, so there's no apparent reason why models are misbehaving.
-
Not actually contributing to the core development of the engine.
-
Social media scummery.
-
Treating the user as 'dumb' by hiding things like the default hard 2048-token context window.
-
Not keeping up with technical innovations, like newer quantizations, SWA, batching, other backend stuff.
-
Bad default quantizations, even beyond the above. For instance, no Google QATs (last I checked), no imatrix, no dynamic quants.
I could go on forever about more specific dramas, and I don't even remember the half of them. But there are plenty of technical and moral reasons to stay away.
LM Studio is much better put together if you want 1-click. Truly open solutions that are more DIY (and reward you with dramatically better performance from the understanding/learning) are the way if you have the time/patience to burn.
Thanks. I'll factor that in next time someone asks me for a recommendation. I personally have Kobold.CPP on my machine, that seems to be more transparent toward such things.
-
-
Thanks. I'll factor that in next time someone asks me for a recommendation. I personally have Kobold.CPP on my machine, that seems to be more transparent toward such things.
wrote last edited by [email protected]Kobold.cpp is fantastic. Sometimes there are more optimal ways to squeeze models into VRAM (depends on the model/hardware), but TBH I have no complaints.
I would recommend croco.cpp, a drop-in fork: https://github.com/Nexesenex/croco.cpp
It has support for more the advanced quantization schemes of ik_llama.cpp. Specifically, you can get really fast performance offloading MoEs, and you can also use much higher quality quantizations, with even ~3.2bpw being relatively low loss. You'd have to make the quants yourself, but it's quite doable... just poorly documented, heh.
The other warning I'd have is that some of it's default sampling presets are fdfunky, if only because they're from the old days of Pygmalion 6B and Llama 1/2. Newer models like much, much lower temperature and rep penalty.
-
Kobold.cpp is fantastic. Sometimes there are more optimal ways to squeeze models into VRAM (depends on the model/hardware), but TBH I have no complaints.
I would recommend croco.cpp, a drop-in fork: https://github.com/Nexesenex/croco.cpp
It has support for more the advanced quantization schemes of ik_llama.cpp. Specifically, you can get really fast performance offloading MoEs, and you can also use much higher quality quantizations, with even ~3.2bpw being relatively low loss. You'd have to make the quants yourself, but it's quite doable... just poorly documented, heh.
The other warning I'd have is that some of it's default sampling presets are fdfunky, if only because they're from the old days of Pygmalion 6B and Llama 1/2. Newer models like much, much lower temperature and rep penalty.
Thanks for the random suggestion! Installed it already. Sadly as a drop-in replacement it doesn't provide any speedup on my old machine, it's exactly the same number of tokens per second... Guess I have to learn about the ik_llama.cpp and pick a different quantization of my favourite model.
-
I hate to drone on about this again, but:
-
ollama is getting less and less open, and (IMO) should not be used. If that doesn't concern you, you should be using LM Studio anyway.
-
The model sizes they mention are mostly for old models no-one should be using. The only exception is a 70B MoE (Hunyuan), but I think ollama doesn't even support that?
-
The quantization methods they mention are (comparatively) primitive and low performance, not cutting edge.
-
It mentions q8_0 twice, nonsensically... Um, it makes me think this article is AI slop?
I'm glad opensuse is promoting local LLM usage, but please... not ollama, and be more specific.
And don't use ollama to write it without checking
It also sets context length to 2k by default iirc, which breaks a lot of tasks, and gives a general bad first impression to users who are likely using local models for the first time.
-
-
It also sets context length to 2k by default iirc, which breaks a lot of tasks, and gives a general bad first impression to users who are likely using local models for the first time.
Yes, and it's hard to undo, and not obvious!
-
Thanks for the random suggestion! Installed it already. Sadly as a drop-in replacement it doesn't provide any speedup on my old machine, it's exactly the same number of tokens per second... Guess I have to learn about the ik_llama.cpp and pick a different quantization of my favourite model.
wrote last edited by [email protected]What model size/family? What GPU? What context length? There are many different backends with different strengths, but I can tell you the optimal way to run it and the quantization you should run with a bit more specificity, heh.
-
What model size/family? What GPU? What context length? There are many different backends with different strengths, but I can tell you the optimal way to run it and the quantization you should run with a bit more specificity, heh.
wrote last edited by [email protected]CPU-only. It's an old Xeon workstation without any GPU, since I mostly do one-off AI tasks at home and I never felt any urge to buy one (yet). Model size woul be something between 7B and 32B with that. Context length is something like 8128 tokens. I have a bit less than 30GB of RAM to waste since I'm doing other stuff on that machine as well.
And I'm picky with the models. I dislike the condescending tone of ChatGPT and newer open-weight models. I don't want it to blabber or praise me for my "genious" ideas. It should be creative, have some storywriting abilities, be uncensored and not overly agreeable. Best model I found for that is Mistral-Nemo-Instruct. And I currently run a Q4_K_M quant of it. That does about 2.5 t/s on my computer (which isn't a lot, but somewhat acceptable for what I do). Mistral-Nemo isn't the latest and greatest any more. But I really prefer it's tone of speaking and it performs well on a wide variety of tasks. And I mostly do weird things with it. Let it give me creative advice, be a dungeon master or an late 80s text adventure. Or mimick a radio moderator and feed it into TTS for a radio show. Or write a book chapter or a bad rap song. I'm less concerned with the popular AI use-cases like answer factual questions or write computer code. So I'd like to switch to a newer, more "intelligent" model. But that proves harder than I imagined.
(Occasionally I do other stuff as well, but that's a far and in-between. So I'll rent a datacenter GPU on runpod.io for a few bucks an hour. That's the main reason why I didn't buy an own GPU yet.)
-
CPU-only. It's an old Xeon workstation without any GPU, since I mostly do one-off AI tasks at home and I never felt any urge to buy one (yet). Model size woul be something between 7B and 32B with that. Context length is something like 8128 tokens. I have a bit less than 30GB of RAM to waste since I'm doing other stuff on that machine as well.
And I'm picky with the models. I dislike the condescending tone of ChatGPT and newer open-weight models. I don't want it to blabber or praise me for my "genious" ideas. It should be creative, have some storywriting abilities, be uncensored and not overly agreeable. Best model I found for that is Mistral-Nemo-Instruct. And I currently run a Q4_K_M quant of it. That does about 2.5 t/s on my computer (which isn't a lot, but somewhat acceptable for what I do). Mistral-Nemo isn't the latest and greatest any more. But I really prefer it's tone of speaking and it performs well on a wide variety of tasks. And I mostly do weird things with it. Let it give me creative advice, be a dungeon master or an late 80s text adventure. Or mimick a radio moderator and feed it into TTS for a radio show. Or write a book chapter or a bad rap song. I'm less concerned with the popular AI use-cases like answer factual questions or write computer code. So I'd like to switch to a newer, more "intelligent" model. But that proves harder than I imagined.
(Occasionally I do other stuff as well, but that's a far and in-between. So I'll rent a datacenter GPU on runpod.io for a few bucks an hour. That's the main reason why I didn't buy an own GPU yet.)
Try any of Latitude's series. They're 'uninhibited' dungeonmaster models, but they should be smart enough (and retain enough of that personality) for some flexibility:
https://huggingface.co/LatitudeGames
Perhaps more optimally for your hardware, try this:
https://huggingface.co/Gryphe/Pantheon-Proto-RP-1.8-30B-A3B
It's trained from Qwen A3B base, not instruct. Base models usually don't have the severe ChatGPT-isms you describe, hence while I haven't personally tried this model, it seems promising. And it should be fast on your Xeon.
-
CPU-only. It's an old Xeon workstation without any GPU, since I mostly do one-off AI tasks at home and I never felt any urge to buy one (yet). Model size woul be something between 7B and 32B with that. Context length is something like 8128 tokens. I have a bit less than 30GB of RAM to waste since I'm doing other stuff on that machine as well.
And I'm picky with the models. I dislike the condescending tone of ChatGPT and newer open-weight models. I don't want it to blabber or praise me for my "genious" ideas. It should be creative, have some storywriting abilities, be uncensored and not overly agreeable. Best model I found for that is Mistral-Nemo-Instruct. And I currently run a Q4_K_M quant of it. That does about 2.5 t/s on my computer (which isn't a lot, but somewhat acceptable for what I do). Mistral-Nemo isn't the latest and greatest any more. But I really prefer it's tone of speaking and it performs well on a wide variety of tasks. And I mostly do weird things with it. Let it give me creative advice, be a dungeon master or an late 80s text adventure. Or mimick a radio moderator and feed it into TTS for a radio show. Or write a book chapter or a bad rap song. I'm less concerned with the popular AI use-cases like answer factual questions or write computer code. So I'd like to switch to a newer, more "intelligent" model. But that proves harder than I imagined.
(Occasionally I do other stuff as well, but that's a far and in-between. So I'll rent a datacenter GPU on runpod.io for a few bucks an hour. That's the main reason why I didn't buy an own GPU yet.)
wrote last edited by [email protected]One more thing, you don't have to get something shiny and new to speed LLMs up. Even if you have like a 4-6GB GPU collecting dust somehwere, you can still use it to partially offload MoE models to great effect.
-
Try any of Latitude's series. They're 'uninhibited' dungeonmaster models, but they should be smart enough (and retain enough of that personality) for some flexibility:
https://huggingface.co/LatitudeGames
Perhaps more optimally for your hardware, try this:
https://huggingface.co/Gryphe/Pantheon-Proto-RP-1.8-30B-A3B
It's trained from Qwen A3B base, not instruct. Base models usually don't have the severe ChatGPT-isms you describe, hence while I haven't personally tried this model, it seems promising. And it should be fast on your Xeon.
wrote last edited by [email protected]Big thanks! I'm always looking for recommendations. I'll check them out. It's going to take me some time, since it's very subjective. I used to look at numbers and scores, but they just don't mean a lot. So I need to use every one for a while and see whether I like what they write. The MoE model is quite an improvement in speed already. It's 3 times faster...
-
Big thanks! I'm always looking for recommendations. I'll check them out. It's going to take me some time, since it's very subjective. I used to look at numbers and scores, but they just don't mean a lot. So I need to use every one for a while and see whether I like what they write. The MoE model is quite an improvement in speed already. It's 3 times faster...
wrote last edited by [email protected]Turn it into an ik_llama.cpp k quant, and you should be able to squeeze even more out!
FYI you can find more models like this by looking up a base model (not the instruct) of interest, then clicking on the 'finetunes' category. For example:
https://huggingface.co/models?other=base_model%3Afinetune%3AQwen%2FQwen3-30B-A3B-Base&sort=modified
This one's also the perfect size for you, but has no finetunes yet: https://huggingface.co/baidu/ERNIE-4.5-VL-28B-A3B-Base-PT
One other thing. A lot of folks (like me) tend to use the base models, not instruct finetunes, in completion mode since they tend to be devoid of AI slop. But you have to prompt them different than a regular LLM: instead of multi turn conversation, you write out a starting block of text for them to 'latch onto', and get them to continue it from your cursor.
But prompt them right, and they will do literally whatever you want, devoid of any sycophancy or guardrails.
Mikupad is great for this since it also shows token probablities. So you can, for instance, click on a critial word, and see what 'choices' the LLM was considering internally as a set of branches, and regenerate from there.
-
Try any of Latitude's series. They're 'uninhibited' dungeonmaster models, but they should be smart enough (and retain enough of that personality) for some flexibility:
https://huggingface.co/LatitudeGames
Perhaps more optimally for your hardware, try this:
https://huggingface.co/Gryphe/Pantheon-Proto-RP-1.8-30B-A3B
It's trained from Qwen A3B base, not instruct. Base models usually don't have the severe ChatGPT-isms you describe, hence while I haven't personally tried this model, it seems promising. And it should be fast on your Xeon.
The 30B-A3Bs I've tried have been suuuuuuuper repetitive. Do you have any specific settings to recommend to get them to work well?
-
Kobold.cpp is fantastic. Sometimes there are more optimal ways to squeeze models into VRAM (depends on the model/hardware), but TBH I have no complaints.
I would recommend croco.cpp, a drop-in fork: https://github.com/Nexesenex/croco.cpp
It has support for more the advanced quantization schemes of ik_llama.cpp. Specifically, you can get really fast performance offloading MoEs, and you can also use much higher quality quantizations, with even ~3.2bpw being relatively low loss. You'd have to make the quants yourself, but it's quite doable... just poorly documented, heh.
The other warning I'd have is that some of it's default sampling presets are fdfunky, if only because they're from the old days of Pygmalion 6B and Llama 1/2. Newer models like much, much lower temperature and rep penalty.
Kobold is great. Never heard of Croco. I'll have to look into it.
-
Kobold is great. Never heard of Croco. I'll have to look into it.
It's great, albiet not super useful unless you make your own quantizations (or find the few K-quant/trellis quant GGUFs hidden on huggingface).
-
The 30B-A3Bs I've tried have been suuuuuuuper repetitive. Do you have any specific settings to recommend to get them to work well?
wrote last edited by [email protected]Random thing, I did not get a notification for this comment, I stumbled upon it. This happens all the time, and it makes me wonder how many replies I miss...
I don't run A3B specifically, but for Qwen3 32B Instruct I put something like "vary your prose; avoid repetitive vocabulary and sentence structure" in the system prompt, run at least 0.5 DRY, and maybe some dynamic sampler like mirostat if supported. Too much regular rep penalty makes it dumb, unfortunately.
But I have much better luck with base model derived models. Look up the finetunes you tried, and see if they were trained from A3B instruct or base. Qwen3 Instruct is pretty overtuned.
-
Random thing, I did not get a notification for this comment, I stumbled upon it. This happens all the time, and it makes me wonder how many replies I miss...
I don't run A3B specifically, but for Qwen3 32B Instruct I put something like "vary your prose; avoid repetitive vocabulary and sentence structure" in the system prompt, run at least 0.5 DRY, and maybe some dynamic sampler like mirostat if supported. Too much regular rep penalty makes it dumb, unfortunately.
But I have much better luck with base model derived models. Look up the finetunes you tried, and see if they were trained from A3B instruct or base. Qwen3 Instruct is pretty overtuned.
They may have been based on Instruct. It left such a bad impression, I didn't play around with them much. Good to know for the future, though. I haven't used DRY or mirostat really in the past, but I'll try them next time I look at the Qwen3s.
