Login Register

Running Local LLMs: A Comparison of Ollama, llama.cpp, LM Studio, and vLLM

DaDesktop Team ·

There are multiple avenues for running Large Language Models (LLMs) locally. Some tools are engineered to simplify the onboarding process, while others prioritise granular control or are architected to handle high-volume user traffic. Selecting the right solution hinges on your specific needs: are you looking for a straightforward local chat experience, a highly configurable inference engine, or a robust production API?

Ollama

Ollama stands out as one of the most accessible methods for initiating local model execution. The process is streamlined: install the software, fetch a model, and execute it via the command line. Additionally, it offers a local API endpoint for integration with other applications and tools.

Advantages:

  • Streamlined installation and model management
  • User-friendly command-line interface
  • OpenAI-compatible API endpoints
  • Supports GPU acceleration for NVIDIA, AMD, Apple Silicon, and Vulkan-based systems
  • Modelfiles allow for customising model behaviour and parameters
  • Handles concurrent requests efficiently when sufficient memory is available

Limitations:

  • Offers less low-level control compared to llama.cpp
  • Model management is tightly integrated with the Ollama ecosystem
  • May not be the optimal choice for maximum serving throughput or distributed inference scenarios

Complexity: Low. Ideal for users who wish to get a model up and running quickly without navigating complex inference settings.

llama.cpp

llama.cpp is a lightweight C/C++ inference engine designed to execute models efficiently across diverse hardware configurations. It utilises GGUF models and provides extensive control over model loading and execution parameters.

Advantages:

  • Granular control over context, GPU offloading, batching, threading, quantization, and other inference configurations
  • Extensive hardware support, including CUDA, HIP, Metal, Vulkan, and SYCL
  • Supports a wide range of quantization levels, from low-bit formats to 8-bit
  • Ability to split models across multiple GPUs
  • Utilises both CPU and GPU resources when a model exceeds available VRAM
  • Includes llama-server for an OpenAI-compatible API

Limitations:

  • Requires more configuration than Ollama or LM Studio
  • GGUF models typically need to be downloaded and managed separately
  • Optimising performance often requires a deep understanding of inference parameters

Complexity: Medium. Well-suited for users who desire precise control over model execution or wish to experiment with performance and quantization strategies.

LM Studio

LM Studio is a desktop application designed for downloading, configuring, and running local LLMs. It features a graphical interface that simplifies model discovery and management of key settings such as GPU offloading and context size.

Advantages:

  • Intuitive graphical user interface
  • Searches and downloads models directly via Hugging Face
  • Displays model and resource details prior to loading
  • Offers an OpenAI-compatible API server
  • Capable of running models headlessly via the llmster server
  • Supports GGUF models through llama.cpp and MLX models on Apple Silicon

Limitations:

  • Less low-level control than using llama.cpp directly
  • Desktop application format may be less suited to certain server deployments
  • Not primarily designed for large-scale, multi-user serving environments

Complexity: Low. A strong choice for those who prefer experimenting with local models without extensive command-line interaction.

vLLM

vLLM is architected specifically for serving LLMs to applications and multiple concurrent users. Its primary strength lies in efficient high-concurrency serving, leveraging techniques such as PagedAttention, continuous batching, prefix caching, and distributed inference.

Advantages:

  • High throughput for handling multiple simultaneous requests
  • Continuous batching and efficient KV-cache management
  • OpenAI-compatible API server
  • Direct compatibility with many Hugging Face models
  • Supports various quantization methods, including FP8, INT4, GPTQ, AWQ, GGUF, and others
  • Supports tensor, pipeline, expert, and other parallelism strategies
  • Designed for production-grade inference and serving

Limitations:

  • More complex setup and configuration process
  • Primarily targeted at Linux environments
  • Generally overkill for a single user running one model interactively
  • Hardware and model compatibility must be verified prior to deployment

Complexity: High. Best suited for professionals deploying inference services rather than simply running models on personal computers.

Which solution should you choose?

  • Simply want to run a model easily: Opt for Ollama or LM Studio. Choose Ollama for a command-line focused approach with a simple API, or LM Studio if you prefer a graphical interface.
  • Seeking control over inference: Select llama.cpp. It offers direct control over model loading, quantization, context, GPU offloading, and other specific settings.
  • Requiring a local API: Ollama, llama.cpp, or LM Studio are all viable options, as each provides OpenAI-compatible APIs.
  • Need to serve many users: vLLM is the recommended choice. Its continuous batching and distributed inference capabilities are specifically designed for this use case.
  • Interested in experimenting with different quantizations: Consider llama.cpp or LM Studio.

Run it on DaDesktop

If your local GPU hardware is insufficient, you can execute these tools on a DaDesktop cloud desktop. Select a GPU with adequate VRAM for your target model, launch the desktop environment, and install your preferred inference software.

Ollama and LM Studio are excellent for creating a simple local environment. llama.cpp provides deeper control over hardware and inference settings. vLLM is the appropriate choice when you need to expose a model as a high-throughput API.

View available GPUs to compare VRAM and other specifications.