DocsGetting StartedIkigenia PlatformLearning HubArcaniaGuidesUsing LlamaCPP

Using LLaMA.cpp

Learn how to set up and configure LLaMA.cpp as a local AI backend for Arcania

You can find the full llama.cpp documentation here.

Installation Steps

  1. Clone the repo:

    git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp

  2. Download the model: For example, we will use OpenChat 3.5 model, which is what is used on the demo instance. There are many models to choose from.

    Navigate to TheBloke/openchat_3.5-GGUF and download one of the models, such as openchat_3.5.Q5_K_M.gguf. Place this file inside the ./models directory.

  3. Build the server:

    make llama-server

  4. Run the server: Read the llama.cpp documentation for more information on the server options. Or run ./server --help.

    ./llama-server -t 4 -c 4096 -ngl 35 -b 512 --mlock -m models/openchat_3.5.Q5_K_M.gguf

Configuration

Enable the server in the client:

settings -> ChatBot -> ChatBot Backend -> LLaMA.cpp
Using KoboldCPPUsing Llava