Faster Ollama alternative
-
Vllm unfortunately doesn't support switching the model without a restart.
-
Are you using a tiny model (1.5B-7B parameters)? ollama pulls 4bit quant by default. It looks like vllm does not used quantized models by default so this is likely the difference.
-
I would not recommend LocalAI. There documentation is somewhat lacking and it’s an all in one utility with many moving parts. The parts also tend to break, quite often.
-
It was multiple models, mainly 32-70B
-
Can you try setting the
num_ctxandnum_predictusing a Modelfile with ollama? -
I’ve read about this method in the GitHub issues, but to me it seemed impractical to have different models just to change the context size, and that was the point I started looking for alternatives
-
You can overwrite the model by using the same name instead of creating one with a new name if it bothers you. Either way there is no duplication of the llm model file
-
Isn't this a model? https://github.com/deepseek-ai/DeepSeek-V3
(Honest question, not an expert in AI)
-
Yes, Deepseek V3 is a model. But what I was trying to say, you download the file. But then what? Just having the file stored on your harddisk doesn't do much. You need to run it. That's called "inference" in machine learning/AI terms. The repository you linked, contains some example code how to do it with Huggingface's Transformer library. But there are quite some frameworks out there for running AI models. Ollama would be another one. And it's not just some example code where to start with your own Python program, but a ready-made project/framework with tools and frontends available and an interface for other software to hook into.
And generally, you need some software to actually do something. And how fast it is, depends on the software used, the hardware it's executed on. And in this case also on the size of the AI model and its architecture. But yeah, Deepseek V3 has some tricks up it's sleeves to make it very efficient. Though, it is really big for home use. I think we're looking at a six-figure price for the hardware to run it. Usually, people use Deepseek R1 models. Or other smaller AI models if they run them themselves.
-
I see, had no idea! Thanks for the detailed answer!