Posts on

llama.cpp guide - Running LLMs locally, on any hardware, from scratch

Mon, 28 Oct 2024 00:00:00 +0000

No LLMs were harmed during creation of this post.

so, i started playing with LLMs…

…and it’s pretty fun. I was very skeptical about the AI/LLM “boom” back when it started. I thought, like many other people, that they are just mostly making stuff up, and generating uncanny-valley-tier nonsense. Boy, was i wrong. I’ve used ChatGPT once or twice, to test the waters - it made a pretty good first impression, despite hallucinating a bit. That was back when GPT3.5 was the top model. We came a pretty long way since then.

However, despite ChatGPT not disappointing me, i was still skeptical. Everything i’ve wrote, and every piece of response was fully available to OpenAI, or whatever other provider i’d want to use. This is not a big deal, but it tickles me in a wrong way, and also means i can’t use LLMs for any work-related non-open-source stuff. Also, ChatGPT is free only to a some degree - if i’d want to go full-in on AI, i’d probably have to start paying. Which, obviously, i’d rather avoid.

At some point i started looking at open-source models. I had no idea how to use them, but the moment i saw the sizes of “small” models, like Llama 2 7B, i’ve realized that my RTX 2070 Super with mere 8GB of VRAM would probably have issues running them (i was wrong on that too!), and running them on CPU would probably yield very bad performance. And then, i’ve bought a new GPU - RX 7900 XT, with 20GB of VRAM, which is definitely more than enough to run small-to-medium LLMs. Yay!

Now my issue was finding some software that could run an LLM on that GPU. CUDA was the most popular back-end - but that’s for NVidia GPUs, not AMD. After doing a bit of research, i’ve found out about ROCm and found LM Studio. And this was exactly what i was looking for - at least for the time being. Great UI, easy access to many models, and the quantization - that was the thing that absolutely sold me into self-hosting LLMs. Existence of quantization made me realize that you don’t need powerful hardware for running LLMs! You can even run LLMs on RaspberryPi’s at this point (with llama.cpp too!) Of course, the performance will be abysmal if you don’t run the LLM with a proper backend on a decent hardware, but the bar is currently not very high.

If you came here with intention of finding some piece of software that will allow you to easily run popular models on most modern hardware for non-commercial purposes - grab LM Studio, read the next section of this post, and go play with it. It fits this description very well, just make sure to use appropriate back-end for your GPU/CPU for optimal performance.

However, if you:

Want to learn more about llama.cpp (which LM Studio uses as a back-end), and LLMs in general
Want to use LLMs for commercial purposes (LM Studio’s terms forbid that)
Want to run LLMs on exotic hardware (LM Studio provides only the most popular backends)
Don’t like closed-source software (which LM Studio, unfortunately, is) and/or don’t trust anything you don’t build yourself
Want to have access to latest features and models as soon as possible

you should find the rest of this post pretty useful!

but first - some disclaimers for expectation management

Before i proceed, i want to make some stuff clear. This “FAQ” answers some questions i’d like to know answers to before getting into self-hosted LLMs.

Do I need RTX 2070 Super/RX 7900 XT ot similar mid/high-end GPU to do what you did here?

No, you don’t. I’ll elaborate later, but you can run LLMs with no GPU at all. As long as you have reasonably modern hardware (by that i mean at least a decent CPU with AVX support) - you’re compatible. But remember - your performance may vary.

What performance can I expect?

This is a very hard question to answer directly. The speed of text generation depends on multiple factors, but primarily

matrix operations performance on your hardware
memory bandwidth
model size

I’ll explain this with more details later, but you can generally get reasonable performance from the LLM by picking model small enough for your hardware. If you intend to use GPU, and it has enough memory for a model with it’s context - expect real-time text generation. In case you want to use both GPU and CPU, or only CPU - you should expect much lower performance, but real-time text generation is possible with small models.

What quality of responses can I expect?

That heavily depends on your usage and chosen model. I can’t answer that question directly, you’ll have to play around and find out yourself. A rule of thumb is “larger the model, better the response” - consider the fact that size of SOTA (state-of-the-art) models, like GPT-4 or Claude, is usually measured in hundreds of billions of parameters. Unless you have multiple GPUs or unreasonable amount of RAM and patience - you’ll most likely be restricted to models with less than 20 billion parameters. From my experience, 7-8B models are pretty good for generic purposes and programming - and they are not very far from SOTA models like GPT-4o or Claude in terms of raw quality of generated responses, but the difference is definitely noticeable. Keep in mind that the choice of a model is only a part of the problem - providing proper context and system prompt, or fine-tuning LLMs can do wonders.

Can i replace ChatGPT/Claude/[insert online LLM provider] with that?

Maybe. In theory - yes, but in practice - it depends on your tools. llama.cpp provides OpenAI-compatible server. As long as your tools communicate with LLMs via OpenAI API, and you are able to set custom endpoint, you will be able to use self-hosted LLM with them.

prerequisites

Reasonably modern CPU. If you’re rocking any Ryzen, or Intel’s 8th gen or newer, you’re good to go, but all of this should work on older hardware too.
Optimally, a GPU. More VRAM, the better. If you have at least 8GB of VRAM, you should be able to run 7-8B models, i’d say that it’s reasonable minimum. Vendor doesn’t matter, llama.cpp supports NVidia, AMD and Apple GPUs (not sure about Intel, but i think i saw a backend for that - if not, Vulkan should work).
If you’re not using GPU or it doesn’t have enough VRAM, you need RAM for the model. As above, at least 8GB of free RAM is recommended, but more is better. Keep in mind that when only GPU is used by llama.cpp, RAM usage is very low.

In the guide, i’ll assume you’re using either Windows or Linux. I can’t provide any support for Mac users, so they should follow Linux steps and consult the llama.cpp docs wherever possible.

Some context-specific formatting is used in this post:

Parts of this post where i’ll write about Windows-specific stuff will have this background. You’ll notice they’re much longer than Linux ones - Windows is a PITA. Linux is preferred. I will still explain everything step-by-step for Windows, but in case of issues - try Linux.

And parts where i’ll write about Linux-specific stuff will have this background.

building the llama

In docs/build.md, you’ll find detailed build instructions for all the supported platforms. By default, llama.cpp builds with auto-detected CPU support. We’ll talk about enabling GPU and advanced CPU support later, first - let’s try building it as-is, because it’s a good baseline to start with, and it doesn’t require any external dependencies. To do that, we only need a C++ toolchain, CMake and Ninja.

If you are very lazy, you can download a release from Github and skip building steps. Make sure to download correct version for your hardware/backend. If you have troubles picking, i recommend following the build guide anyway - it’s simple enough and should explain what you should be looking for. Keep in mind that release won’t contain Python scripts that we’re going to use, so if you’ll want to quantize models manually, you’ll need to get them from repository.

On Windows, i recommend using MSYS to setup the environment for building and using llama.cpp. Microsoft Visual C++ is supported too, but trust me on that - you’ll want to use MSYS instead (it’s still a bit of pain in the ass, Linux setup is much simpler). Follow the guide on the main page to install MinGW for x64 UCRT environment, which you probably should be using. CMake, Ninja and Git can be installed in UCRT MSYS environment like that:

pacman -S git mingw-w64-ucrt-x86_64-cmake mingw-w64-ucrt-x86_64-ninja

However, if you’re using any other toolchain (MSVC, or non-MSYS one), you should install CMake, Git and Ninja via winget:

winget install cmake git.git ninja-build.ninja

You’ll also need Python, which you can get via winget. Get the 3.12 version, because 3.13 does not have working PyTorch package yet.

DO NOT USE PYTHON FROM MSYS, IT WILL NOT WORK PROPERLY DUE TO ISSUES WITH BUILDING llama.cpp DEPENDENCY PACKAGES! We’re going to be using MSYS only for building llama.cpp, nothing more.

If you’re using MSYS, remember to add it’s /bin (C:\msys64\ucrt64\bin by default) directory to PATH, so Python can use MinGW for building packages. Check if GCC is available by opening PowerShell/Command line and trying to run gcc --version. Also; check if it’s correct GCC by running where.exe gcc.exe and seeing where the first entry points to. Reorder your PATH if you’ll notice that you’re using wrong GCC.

If you’re using MSVC - ignore this disclaimer, it should be “detectable” by default.

winget install python.python.3.12

I recommend installing/upgrading pip, setuptools and wheel packages before continuing.

python -m pip install --upgrade pip wheel setuptools

On Linux, GCC is recommended, but you should be able to use Clang if you’d prefer by setting CMAKE_C_COMPILER=clang and CMAKE_CXX_COMPILER=clang++ variables. You should have GCC preinstalled (check gcc --version in terminal), if not - get latest version for your distribution using your package manager. Same applies to CMake, Ninja, Python 3 (with setuptools, wheel and pip) and Git.

Let’s start by grabbing a copy of llama.cpp source code, and moving into it.

Disclaimer: this guide assumes all commands are ran from user’s home directory (/home/[yourusername] on Linux, C:/Users/[yourusername] on Windows). You can use any directory you’d like, just keep in mind that if “starting directory” is not explicitly mentioned, start from home dir/your chosen one.

If you’re using MSYS, remember that MSYS home directory is different from Windows home directory. Make sure to use cd (without arguments) to move into it after starting MSYS.

(If you have previously configured SSH auth w/ GitHub, use git@github.com:ggerganov/llama.cpp.git instead of the URL below)

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
git submodule update --init --recursive

Now we’ll use CMake to generate build files, build the project, and install it. Run the following command to generate build files in build/ subdirectory:

cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/your/install/dir -DLLAMA_BUILD_TESTS=OFF -DLLAMA_BUILD_EXAMPLES=ON -DLLAMA_BUILD_SERVER=ON

There’s a lot of CMake variables being defined, which we could ignore and let llama.cpp use it’s defaults, but we won’t:

CMAKE_BUILD_TYPE is set to release for obvious reasons - we want maximum performance.
CMAKE_INSTALL_PREFIX is where the llama.cpp binaries and python scripts will go. Replace the value of this variable, or remove it’s definition to keep default value.
- On Windows, default directory is c:/Program Files/llama.cpp. As above, you’ll need admin privileges to install it, and you’ll have to add the bin/ subdirectory to your PATH to make llama.cpp binaries accessible system-wide. I prefer installing llama.cpp in $env:LOCALAPPDATA/llama.cpp (C:/Users/[yourusername]/AppData/Local/llama.cpp), as it doesn’t require admin privileges.
- On Linux, default directory is /usr/local. You can ignore this variable if that’s fine with you, but you’ll need superuser permissions to install the binaries there. If you don’t have them, change it to point somewhere in your user directory and add it’s bin/ subdirectory to PATH.
LLAMA_BUILD_TESTS is set to OFF because we don’t need tests, it’ll make the build a bit quicker.
LLAMA_BUILD_EXAMPLES is ON because we’re gonna be using them.
LLAMA_BUILD_SERVER - see above. Note: Disabling LLAMA_BUILD_EXAMPLES unconditionally disables building the server, both must be ON.

If you’d ignore all of those variables, the default would be used - in case of LLAMA_BUILD_* variables, all of those is ON (if you’re building llama.cpp as standalone app, which you most likely are). CMAKE_BUILD_TYPE is usually set to Release by default, but not always, so i’d rather set it manually. CMAKE_INSTALL_PREFIX is not necessary at all if you don’t intend to install the executables and you’re fine with adding the build dir’s /bin subdirectory to PATH.

Now, let’s build the project. Replace X with amount of cores your CPU has for faster compilation. In theory, Ninja should automatically use all available cores, but i still prefer passing this argument manually.

cmake --build build --config Release -j X

Building should take only a few minutes. After that, we can install the binaries for easier usage.

cmake --install build --config Release

Now, after going to CMAKE_INSTALL_PREFIX/bin directory, we should see a list of executables and Python scripts:

/c/Users/phoen/llama-build/bin
❯ l
Mode Size Date Modified Name
-a--- 203k 7 Nov 16:14 convert_hf_to_gguf.py
-a--- 3.9M 7 Nov 16:18 llama-batched-bench.exe
-a--- 3.9M 7 Nov 16:18 llama-batched.exe
-a--- 3.4M 7 Nov 16:18 llama-bench.exe
-a--- 3.9M 7 Nov 16:18 llama-cli.exe
-a--- 3.2M 7 Nov 16:18 llama-convert-llama2c-to-ggml.exe
-a--- 3.9M 7 Nov 16:18 llama-cvector-generator.exe
-a--- 3.9M 7 Nov 16:18 llama-embedding.exe
-a--- 3.9M 7 Nov 16:18 llama-eval-callback.exe
-a--- 3.9M 7 Nov 16:18 llama-export-lora.exe
-a--- 3.0M 7 Nov 16:18 llama-gbnf-validator.exe
-a--- 1.2M 7 Nov 16:18 llama-gguf-hash.exe
-a--- 3.0M 7 Nov 16:18 llama-gguf-split.exe
-a--- 1.1M 7 Nov 16:18 llama-gguf.exe
-a--- 3.9M 7 Nov 16:18 llama-gritlm.exe
-a--- 3.9M 7 Nov 16:18 llama-imatrix.exe
-a--- 3.9M 7 Nov 16:18 llama-infill.exe
-a--- 4.2M 7 Nov 16:18 llama-llava-cli.exe
-a--- 3.9M 7 Nov 16:18 llama-lookahead.exe
-a--- 3.9M 7 Nov 16:18 llama-lookup-create.exe
-a--- 1.2M 7 Nov 16:18 llama-lookup-merge.exe
-a--- 3.9M 7 Nov 16:18 llama-lookup-stats.exe
-a--- 3.9M 7 Nov 16:18 llama-lookup.exe
-a--- 4.1M 7 Nov 16:18 llama-minicpmv-cli.exe
-a--- 3.9M 7 Nov 16:18 llama-parallel.exe
-a--- 3.9M 7 Nov 16:18 llama-passkey.exe
-a--- 4.0M 7 Nov 16:18 llama-perplexity.exe
-a--- 3.0M 7 Nov 16:18 llama-quantize-stats.exe
-a--- 3.2M 7 Nov 16:18 llama-quantize.exe
-a--- 3.9M 7 Nov 16:18 llama-retrieval.exe
-a--- 3.9M 7 Nov 16:18 llama-save-load-state.exe
-a--- 5.0M 7 Nov 16:19 llama-server.exe
-a--- 3.0M 7 Nov 16:18 llama-simple-chat.exe
-a--- 3.0M 7 Nov 16:18 llama-simple.exe
-a--- 3.9M 7 Nov 16:18 llama-speculative.exe
-a--- 3.1M 7 Nov 16:18 llama-tokenize.exe

Don’t feel overwhelmed by the amount, we’re only going to be using few of them. You should try running one of them to check if the executables have built correctly, for example - try llama-cli --help. We can’t do anything meaningful yet, because we lack a single critical component - a model to run.

getting a model

The main place to look for models is HuggingFace. You can also find datasets and other AI-related stuff there, great site.

We’re going to use SmolLM2 here, a model series created by HuggingFace and published fairly recently (1st November 2024). The reason i’ve chosen this model is the size - as the name implies, it’s small. Largest model from this series has 1.7 billion parameters, which means that it requires approx. 4GB of system memory to run in raw, unquantized form (excluding context)! There are also 360M and 135M variants, which are even smaller and should be easily runnable on RaspberryPi or a smartphone.

There’s but one issue - llama.cpp cannot run “raw” models directly. What is usually provided by most LLM creators are original weights in .safetensors or similar format. llama.cpp expects models in .gguf format. Fortunately, there is a very simple way of converting original model weights into .gguf - llama.cpp provides convert_hf_to_gguf.py script exactly for this purpose! Sometimes the creator provides .gguf files - for example, two variants of SmolLM2 are provided by HuggingFace in this format. This is not a very common practice, but you can also find models in .gguf format uploaded there by community. However, i’ll ignore the existence of pre-quantized .gguf files here, and focus on quantizing our models by ourselves here, as it’ll allow us to experiment and adjust the quantization parameters of our model without having to download it multiple times.

Grab the content of SmolLM2 1.7B Instruct repository (you can use 360M Instruct or 135M Instruct version instead, if you have less than 4GB of free (V)RAM - or use any other model you’re already familiar with, if it supports transformers), but omit the LFS files - we only need a single one, and we’ll download it manually.

Why Instruct, specifically? You might have noticed that there are two variants of all those models - Instruct and the other one without a suffix. Instruct is trained for chat conversations, base model is only trained for text completion and is usually used as a base for further training. This rule applies to most LLMs, but not all, so make sure to read the model’s description before using it!

If you’re using Bash/ZSH or compatible shell:

MSYS uses Bash by default, so it applies to it too. From now on, assume that Linux commands work on MSYS too, unless i explicitly say otherwise.

GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/HuggingFaceTB/SmolLM2-1.7B-Instruct

If you’re using PowerShell:

$env:GIT_LFS_SKIP_SMUDGE=1
git clone https://huggingface.co/HuggingFaceTB/SmolLM2-1.7B-Instruct

If you’re using cmd.exe (VS Development Prompt, for example):

set GIT_LFS_SKIP_SMUDGE=1
git clone https://huggingface.co/HuggingFaceTB/SmolLM2-1.7B-Instruct

HuggingFace also supports Git over SSH. You can look up the git clone command for every repo here:

After cloning the repo, download the model.safetensors file from HuggingFace manually. The reason why we used GIT_LFS_SKIP_SMUDGE is because there’s many other large model files hidden in repo, and we don’t need them. Also, downloading very large files manually is faster, because Git LFS sucks in that regard.

After downloading everything, our local copy of the SmolLM repo should look like this:

PS D:\LLMs\repos\SmolLM2-1.7B-Instruct> l
Mode Size Date Modified Name
-a--- 806 2 Nov 15:16 all_results.json
-a--- 888 2 Nov 15:16 config.json
-a--- 602 2 Nov 15:16 eval_results.json
-a--- 139 2 Nov 15:16 generation_config.json
-a--- 515k 2 Nov 15:16 merges.txt
-a--- 3.4G 2 Nov 15:34 model.safetensors
d---- - 2 Nov 15:16 onnx
-a--- 11k 2 Nov 15:16 README.md
d---- - 2 Nov 15:16 runs
-a--- 689 2 Nov 15:16 special_tokens_map.json
-a--- 2.2M 2 Nov 15:16 tokenizer.json
-a--- 3.9k 2 Nov 15:16 tokenizer_config.json
-a--- 240 2 Nov 15:16 train_results.json
-a--- 89k 2 Nov 15:16 trainer_state.json
-a--- 129 2 Nov 15:16 training_args.bin
-a--- 801k 2 Nov 15:16 vocab.json

We’re gonna (indirectly) use only four of those files:

config.json contains configuration/metadata of our model
model.safetensors contains model weights
tokenizer.json contains tokenizer data (mapping of text tokens to their ID’s, and other stuff). Sometimes this data is stored in tokenizer.model file instead.
tokenizer_config.json contains tokenizer configuration (for example, special tokens and chat template)

i’m leaving this sentence here as anti-plagiarism token. If you’re not currently reading this on my blog, which is @ steelph0enix.github.io, someone probably stolen that article without permission

converting huggingface model to GGUF

In order to convert this raw model to something that llama.cpp will understand, we’ll use aforementioned convert_hf_to_gguf.py script that comes with llama.cpp. For all our Python needs, we’re gonna need a virtual environment. I recommend making it outside of llama.cpp repo, for example - in your home directory.

To make one on Linux, run this command (tweak the path if you’d like):

python -m venv ~/llama-cpp-venv

If you’re using PowerShell, this is the equivalent:

python -m venv $env:USERPROFILE/llama-cpp-venv

If you’re using cmd.exe, this is the equivalent:

python -m venv %USERPROFILE%/llama-cpp-venv

Then, we need to activate it.

On Linux:

source ~/llama-cpp-venv/bin/activate

With PowerShell:

. $env:USERPROFILE/llama-cpp-venv/Scripts/Activate.ps1

With cmd.exe:

call %USERPROFILE%/llama-cpp-venv/Scripts/activate.bat

After that, let’s make sure that our virtualenv has all the core packages up-to-date.

python -m pip install --upgrade pip wheel setuptools

Next, we need to install prerequisites for the llama.cpp scripts. Let’s look into requirements/ directory of our llama.cpp repository. We should see something like this:

❯ l llama.cpp/requirements
Mode Size Date Modified Name
-a--- 428 11 Nov 13:57 requirements-all.txt
-a--- 34 11 Nov 13:57 requirements-compare-llama-bench.txt
-a--- 111 11 Nov 13:57 requirements-convert_hf_to_gguf.txt
-a--- 111 11 Nov 13:57 requirements-convert_hf_to_gguf_update.txt
-a--- 99 11 Nov 13:57 requirements-convert_legacy_llama.txt
-a--- 43 11 Nov 13:57 requirements-convert_llama_ggml_to_gguf.txt
-a--- 96 11 Nov 13:57 requirements-convert_lora_to_gguf.txt
-a--- 48 11 Nov 13:57 requirements-pydantic.txt
-a--- 13 11 Nov 13:57 requirements-test-tokenizer-random.txt

As we can see, there’s a file with deps for our script! To install dependencies from it, run this command:

python -m pip install --upgrade -r llama.cpp/requirements/requirements-convert_hf_to_gguf.txt

If pip failed during build, make sure you have working C/C++ toolchain in your PATH.

If you’re using MSYS for that, don’t. Go back to PowerShell/cmd, install Python via winget and repeat the setup. As far as i’ve tested it, Python deps don’t detect the platform correctly on MSYS and try to use wrong build config. This is what i warned you about earlier.

Now we can use the script to create our GGUF model file. Start with printing the help and reading the options.

python llama.cpp/convert_hf_to_gguf.py --help

If this command printed help, you can continue. Otherwise make sure that python’s virtualenv is active and dependencies are correctly installed, and try again. To convert our model we can simply pass the path to directory with model’s repository and, optionally, path to output file. We don’t need to tweak the quantization here, for maximum flexibility we’re going to create a floating-point GGUF file which we’ll then quantize down. That’s because llama-quantize offers much more quantization options, and this script picks optimal floating-point format by default.

To create GGUF file from our downloaded HuggingFace repository with SmolLM2 (Replace SmolLM2-1.7B-Instruct with your path, if it’s different) run this command:

python llama.cpp/convert_hf_to_gguf.py SmolLM2-1.7B-Instruct --outfile ./SmolLM2.gguf

If everything went correctly, you should see similar output:

INFO:hf-to-gguf:Loading model: SmolLM2-1.7B-Instruct
INFO:gguf.gguf_writer:gguf: This GGUF file is for Little Endian only
INFO:hf-to-gguf:Exporting model...
INFO:hf-to-gguf:gguf: loading model part 'model.safetensors'
INFO:hf-to-gguf:token_embd.weight, torch.bfloat16 --> F16, shape = {2048, 49152}
INFO:hf-to-gguf:blk.0.attn_norm.weight, torch.bfloat16 --> F32, shape = {2048}
...
INFO:hf-to-gguf:blk.9.attn_q.weight, torch.bfloat16 --> F16, shape = {2048, 2048}
INFO:hf-to-gguf:blk.9.attn_v.weight, torch.bfloat16 --> F16, shape = {2048, 2048}
INFO:hf-to-gguf:output_norm.weight, torch.bfloat16 --> F32, shape = {2048}
INFO:hf-to-gguf:Set meta model
INFO:hf-to-gguf:Set model parameters
INFO:hf-to-gguf:gguf: context length = 8192
INFO:hf-to-gguf:gguf: embedding length = 2048
INFO:hf-to-gguf:gguf: feed forward length = 8192
INFO:hf-to-gguf:gguf: head count = 32
INFO:hf-to-gguf:gguf: key-value head count = 32
INFO:hf-to-gguf:gguf: rope theta = 130000
INFO:hf-to-gguf:gguf: rms norm epsilon = 1e-05
INFO:hf-to-gguf:gguf: file type = 1
INFO:hf-to-gguf:Set model tokenizer
INFO:gguf.vocab:Adding 48900 merge(s).
INFO:gguf.vocab:Setting special token type bos to 1
INFO:gguf.vocab:Setting special token type eos to 2
INFO:gguf.vocab:Setting special token type unk to 0
INFO:gguf.vocab:Setting special token type pad to 2
INFO:gguf.vocab:Setting chat_template to {% for message in messages %}{% if loop.first and messages[0]['role'] != 'system' %}{{ '<|im_start|>system
You are a helpful AI assistant named SmolLM, trained by Hugging Face<|im_end|>
' }}{% endif %}{{'<|im_start|>' + message['role'] + '
' + message['content'] + '<|im_end|>' + '
'}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant
' }}{% endif %}
INFO:hf-to-gguf:Set model quantization version
INFO:gguf.gguf_writer:Writing the following files:
INFO:gguf.gguf_writer:SmolLM2.gguf: n_tensors = 218, total_size = 3.4G
Writing: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3.42G/3.42G [00:15<00:00, 215Mbyte/s]
INFO:hf-to-gguf:Model successfully exported to SmolLM2.gguf

quantizing the model

Now we can finally quantize our model! To do that, we’ll use llama-quantize executable that we previously compiled with other llama.cpp executables. First, let’s check what quantizations we have available.

llama-quantize --help

As of now, llama-quantize --help shows following types:


usage: llama-quantize [--help] [--allow-requantize] [--leave-output-tensor] [--pure] [--imatrix] [--include-weights] [--exclude-weights] [--output-tensor-type] [--token-embedding-type] [--override-kv] model-f32.gguf [model-quant.gguf] type [nthreads]

...

Allowed quantization types:
 2 or Q4_0 : 4.34G, +0.4685 ppl @ Llama-3-8B
 3 or Q4_1 : 4.78G, +0.4511 ppl @ Llama-3-8B
 8 or Q5_0 : 5.21G, +0.1316 ppl @ Llama-3-8B
 9 or Q5_1 : 5.65G, +0.1062 ppl @ Llama-3-8B
 19 or IQ2_XXS : 2.06 bpw quantization
 20 or IQ2_XS : 2.31 bpw quantization
 28 or IQ2_S : 2.5 bpw quantization
 29 or IQ2_M : 2.7 bpw quantization
 24 or IQ1_S : 1.56 bpw quantization
 31 or IQ1_M : 1.75 bpw quantization
 36 or TQ1_0 : 1.69 bpw ternarization
 37 or TQ2_0 : 2.06 bpw ternarization
 10 or Q2_K : 2.96G, +3.5199 ppl @ Llama-3-8B
 21 or Q2_K_S : 2.96G, +3.1836 ppl @ Llama-3-8B
 23 or IQ3_XXS : 3.06 bpw quantization
 26 or IQ3_S : 3.44 bpw quantization
 27 or IQ3_M : 3.66 bpw quantization mix
 12 or Q3_K : alias for Q3_K_M
 22 or IQ3_XS : 3.3 bpw quantization
 11 or Q3_K_S : 3.41G, +1.6321 ppl @ Llama-3-8B
 12 or Q3_K_M : 3.74G, +0.6569 ppl @ Llama-3-8B
 13 or Q3_K_L : 4.03G, +0.5562 ppl @ Llama-3-8B
 25 or IQ4_NL : 4.50 bpw non-linear quantization
 30 or IQ4_XS : 4.25 bpw non-linear quantization
 15 or Q4_K : alias for Q4_K_M
 14 or Q4_K_S : 4.37G, +0.2689 ppl @ Llama-3-8B
 15 or Q4_K_M : 4.58G, +0.1754 ppl @ Llama-3-8B
 17 or Q5_K : alias for Q5_K_M
 16 or Q5_K_S : 5.21G, +0.1049 ppl @ Llama-3-8B
 17 or Q5_K_M : 5.33G, +0.0569 ppl @ Llama-3-8B
 18 or Q6_K : 6.14G, +0.0217 ppl @ Llama-3-8B
 7 or Q8_0 : 7.96G, +0.0026 ppl @ Llama-3-8B
 33 or Q4_0_4_4 : 4.34G, +0.4685 ppl @ Llama-3-8B
 34 or Q4_0_4_8 : 4.34G, +0.4685 ppl @ Llama-3-8B
 35 or Q4_0_8_8 : 4.34G, +0.4685 ppl @ Llama-3-8B
 1 or F16 : 14.00G, +0.0020 ppl @ Mistral-7B
 32 or BF16 : 14.00G, -0.0050 ppl @ Mistral-7B
 0 or F32 : 26.00G @ 7B
 COPY : only copy tensors, no quantizing

Let’s decode this table. From the left, we have IDs and names of quantization types - you can use either when calling llama-quantize. After the :, there’s a short description that in most cases shows either the example model’s size and perplexity, or the amount of bits per tensor weight (bpw) for that specific quantization. Perplexity is a metric that describes how certain the model is about it’s predictions. We can think about it like that: lower perplexity -> model is more certain about it’s predictions -> model is more accurate. This is a daily reminder that LLMs are nothing more than overcomplicated autocompletion algorithms. The “bits per weight” metric tells us the average size of quantized tensor’s weight. You may think it’s strange that those are floating-point values, but we’ll see the reason for that soon.

Now, the main question that needs to be answered is “which quantization do we pick?”. And the answer is “it depends”. My rule of thumb for picking quantization type is “the largest i can fit in my VRAM, unless it’s too slow for my taste” and i recommend this approach if you don’t know where to start! Obviously, if you don’t have/want to use GPU, replace “VRAM” with “RAM” and “largest i can fit” with “largest i can fit without forcing the OS to move everything to swap”. That creates another question - “what is the largest quant i can fit in my (V)RAM”? And this depends on the original model’s size and encoding, and - obviously - the amount of (V)RAM you have. Since we’re using SmolLM2, our model is relatively small. GGUF file of 1.7B-Instruct variant in BF16 format weights 3.4GB. Most models you’ll encounter will be encoded in either BF16 or FP16 format, rarely we can find FP32-encoded LLMs. That means most of models have 16 bits per weight by default. We can easily approximate the size of quantized model by multiplying the original size with approximate ratio of bits per weight. For example, let’s assume we want to know how large will SmolLM2 1.7B-Instruct be after Q8_0 quantization. Let’s assume Q8_0 quant uses 8 bits per word, which means the ratio is simply 1/2, so our model should weight ~1.7GB. Let’s check that!

The first argument is source model, second - target file. Third is the quantization type, and last is the amount of cores for parallel processing. Replace N with amount of cores in your system and run this command:

llama-quantize SmolLM2.gguf SmolLM2.q8.gguf Q8_0 N

You should see similar output:

main: build = 4200 (46c69e0e)
main: built with gcc (GCC) 14.2.1 20240910 for x86_64-pc-linux-gnu
main: quantizing 'SmolLM2.gguf' to 'SmolLM2.q8.gguf' as Q8_0 using 24 threads
llama_model_loader: loaded meta data with 37 key-value pairs and 218 tensors from SmolLM2.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv 0: general.architecture str = llama
llama_model_loader: - kv 1: general.type str = model
llama_model_loader: - kv 2: general.name str = SmolLM2 1.7B Instruct
llama_model_loader: - kv 3: general.finetune str = Instruct
llama_model_loader: - kv 4: general.basename str = SmolLM2
llama_model_loader: - kv 5: general.size_label str = 1.7B
llama_model_loader: - kv 6: general.license str = apache-2.0
llama_model_loader: - kv 7: general.base_model.count u32 = 1
llama_model_loader: - kv 8: general.base_model.0.name str = SmolLM2 1.7B
llama_model_loader: - kv 9: general.base_model.0.organization str = HuggingFaceTB
llama_model_loader: - kv 10: general.base_model.0.repo_url str = https://huggingface.co/HuggingFaceTB/...
llama_model_loader: - kv 11: general.tags arr[str,4] = ["safetensors", "onnx", "transformers...
llama_model_loader: - kv 12: general.languages arr[str,1] = ["en"]
llama_model_loader: - kv 13: llama.block_count u32 = 24
llama_model_loader: - kv 14: llama.context_length u32 = 8192
llama_model_loader: - kv 15: llama.embedding_length u32 = 2048
llama_model_loader: - kv 16: llama.feed_forward_length u32 = 8192
llama_model_loader: - kv 17: llama.attention.head_count u32 = 32
llama_model_loader: - kv 18: llama.attention.head_count_kv u32 = 32
llama_model_loader: - kv 19: llama.rope.freq_base f32 = 130000.000000
llama_model_loader: - kv 20: llama.attention.layer_norm_rms_epsilon f32 = 0.000010
llama_model_loader: - kv 21: general.file_type u32 = 32
llama_model_loader: - kv 22: llama.vocab_size u32 = 49152
llama_model_loader: - kv 23: llama.rope.dimension_count u32 = 64
llama_model_loader: - kv 24: tokenizer.ggml.model str = gpt2
llama_model_loader: - kv 25: tokenizer.ggml.pre str = smollm
llama_model_loader: - kv 26: tokenizer.ggml.tokens arr[str,49152] = ["<|endoftext|>", "<|im_start|>", "<|...
llama_model_loader: - kv 27: tokenizer.ggml.token_type arr[i32,49152] = [3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, ...
llama_model_loader: - kv 28: tokenizer.ggml.merges arr[str,48900] = ["Ġ t", "Ġ a", "i n", "h e", "Ġ Ġ...
llama_model_loader: - kv 29: tokenizer.ggml.bos_token_id u32 = 1
llama_model_loader: - kv 30: tokenizer.ggml.eos_token_id u32 = 2
llama_model_loader: - kv 31: tokenizer.ggml.unknown_token_id u32 = 0
llama_model_loader: - kv 32: tokenizer.ggml.padding_token_id u32 = 2
llama_model_loader: - kv 33: tokenizer.chat_template str = {% for message in messages %}{% if lo...
llama_model_loader: - kv 34: tokenizer.ggml.add_space_prefix bool = false
llama_model_loader: - kv 35: tokenizer.ggml.add_bos_token bool = false
llama_model_loader: - kv 36: general.quantization_version u32 = 2
llama_model_loader: - type f32: 49 tensors
llama_model_loader: - type bf16: 169 tensors
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon RX 7900 XT (AMD open-source driver) | uma: 0 | fp16: 1 | warp size: 64
[ 1/ 218] output_norm.weight - [ 2048, 1, 1, 1], type = f32, size = 0.008 MB
[ 2/ 218] token_embd.weight - [ 2048, 49152, 1, 1], type = bf16, converting to q8_0 .. size = 192.00 MiB -> 102.00 MiB
[ 3/ 218] blk.0.attn_k.weight - [ 2048, 2048, 1, 1], type = bf16, converting to q8_0 .. size = 8.00 MiB -> 4.25 MiB
[ 4/ 218] blk.0.attn_norm.weight - [ 2048, 1, 1, 1], type = f32, size = 0.008 MB
[ 5/ 218] blk.0.attn_output.weight - [ 2048, 2048, 1, 1], type = bf16, converting to q8_0 .. size = 8.00 MiB -> 4.25 MiB
[ 6/ 218] blk.0.attn_q.weight - [ 2048, 2048, 1, 1], type = bf16, converting to q8_0 .. size = 8.00 MiB -> 4.25 MiB
[ 7/ 218] blk.0.attn_v.weight - [ 2048, 2048, 1, 1], type = bf16, converting to q8_0 .. size = 8.00 MiB -> 4.25 MiB
...
[ 212/ 218] blk.23.attn_output.weight - [ 2048, 2048, 1, 1], type = bf16, converting to q8_0 .. size = 8.00 MiB -> 4.25 MiB
[ 213/ 218] blk.23.attn_q.weight - [ 2048, 2048, 1, 1], type = bf16, converting to q8_0 .. size = 8.00 MiB -> 4.25 MiB
[ 214/ 218] blk.23.attn_v.weight - [ 2048, 2048, 1, 1], type = bf16, converting to q8_0 .. size = 8.00 MiB -> 4.25 MiB
[ 215/ 218] blk.23.ffn_down.weight - [ 8192, 2048, 1, 1], type = bf16, converting to q8_0 .. size = 32.00 MiB -> 17.00 MiB
[ 216/ 218] blk.23.ffn_gate.weight - [ 2048, 8192, 1, 1], type = bf16, converting to q8_0 .. size = 32.00 MiB -> 17.00 MiB
[ 217/ 218] blk.23.ffn_norm.weight - [ 2048, 1, 1, 1], type = f32, size = 0.008 MB
[ 218/ 218] blk.23.ffn_up.weight - [ 2048, 8192, 1, 1], type = bf16, converting to q8_0 .. size = 32.00 MiB -> 17.00 MiB
llama_model_quantize_internal: model size = 3264.38 MB
llama_model_quantize_internal: quant size = 1734.38 MB

main: quantize time = 2289.97 ms
main: total time = 2289.97 ms

And yeah, we were more-or-less correct, it’s 1.7GB! BUT this is only the model’s size, we also have to consider the memory requirements for the context. Fortunately, context doesn’t require massive amounts of memory - i’m not really sure how much exactly it eats up, but it’s safe to assume that we should have at least 1GB memory for it. So, taking all that in account, we can approximate that to load this model to memory with the context, we need at least 3GB of free (V)RAM. This is not so bad, and most modern consumer GPUs have at least this amount (even the old GTX 1060 3GB should be able to run this model in Q8_0 quant). However, if that’s still too much, we can easily go lower! Reducing the amount of bits per weight via quantization not only reduces the model’s size, but also increases the speed of data generation. Unfortunately, it also makes the model more stupid. The change is gradual, you may not notice it when going from Q8_0 to Q6_K, but going below Q4 quant can be noticeable. I strongly recommend experimenting on your own with different models and quantization types, because your experience may be different from mine!

Oh, by the way - remember that right now our llama.cpp build will use CPU for calculations, so the model will reside in RAM. Make sure you have at least 3GB of free RAM before trying to use the model, if you don’t - quantize it with smaller quant, or get a smaller version.

Anyway, we got our quantized model now, we can finally use it!

running llama.cpp server

If going through the first part of this post felt like pain and suffering, don’t worry - i felt the same writing it. That’s why it took a month to write. But, at long last we can do something fun.

Let’s start, as usual, with printing the help to make sure our binary is working fine:

llama-server --help

You should see a lot of options. Some of them will be explained here in a bit, some of them you’ll have to research yourself. For now, the only options that are interesting to us are:

-m, --model FNAME model path (default: `models/$filename` with filename from `--hf-file`
 or `--model-url` if set, otherwise models/7B/ggml-model-f16.gguf)
 (env: LLAMA_ARG_MODEL)
--host HOST ip address to listen (default: 127.0.0.1)
 (env: LLAMA_ARG_HOST)
--port PORT port to listen (default: 8080)
 (env: LLAMA_ARG_PORT)

Run llama-server with model’s path set to quantized SmolLM2 GGUF file. If you don’t have anything running on 127.0.0.1:8080, you can leave the host and port on defaults. Notice that you can also use environmental variables instead of arguments, so you can setup your env and just call llama-server.

llama-server -m SmolLM2.q8.gguf

You should see similar output after running this command:

build: 4182 (ab96610b) with cc (GCC) 14.2.1 20240910 for x86_64-pc-linux-gnu
system info: n_threads = 12, n_threads_batch = 12, total_threads = 24

system_info: n_threads = 12 (n_threads_batch = 12) / 24 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | LLAMAFILE = 1 | AARCH64_REPACK = 1 |

main: HTTP server is listening, hostname: 127.0.0.1, port: 8080, http threads: 23
main: loading model
srv load_model: loading model 'SmolLM2.q8.gguf'
llama_model_loader: loaded meta data with 37 key-value pairs and 218 tensors from SmolLM2.q8.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv 0: general.architecture str = llama
llama_model_loader: - kv 1: general.type str = model
llama_model_loader: - kv 2: general.name str = SmolLM2 1.7B Instruct
llama_model_loader: - kv 3: general.finetune str = Instruct
llama_model_loader: - kv 4: general.basename str = SmolLM2
llama_model_loader: - kv 5: general.size_label str = 1.7B
llama_model_loader: - kv 6: general.license str = apache-2.0
llama_model_loader: - kv 7: general.base_model.count u32 = 1
llama_model_loader: - kv 8: general.base_model.0.name str = SmolLM2 1.7B
llama_model_loader: - kv 9: general.base_model.0.organization str = HuggingFaceTB
llama_model_loader: - kv 10: general.base_model.0.repo_url str = https://huggingface.co/HuggingFaceTB/...
llama_model_loader: - kv 11: general.tags arr[str,4] = ["safetensors", "onnx", "transformers...
llama_model_loader: - kv 12: general.languages arr[str,1] = ["en"]
llama_model_loader: - kv 13: llama.block_count u32 = 24
llama_model_loader: - kv 14: llama.context_length u32 = 8192
llama_model_loader: - kv 15: llama.embedding_length u32 = 2048
llama_model_loader: - kv 16: llama.feed_forward_length u32 = 8192
llama_model_loader: - kv 17: llama.attention.head_count u32 = 32
llama_model_loader: - kv 18: llama.attention.head_count_kv u32 = 32
llama_model_loader: - kv 19: llama.rope.freq_base f32 = 130000.000000
llama_model_loader: - kv 20: llama.attention.layer_norm_rms_epsilon f32 = 0.000010
llama_model_loader: - kv 21: general.file_type u32 = 7
llama_model_loader: - kv 22: llama.vocab_size u32 = 49152
llama_model_loader: - kv 23: llama.rope.dimension_count u32 = 64
llama_model_loader: - kv 24: tokenizer.ggml.model str = gpt2
llama_model_loader: - kv 25: tokenizer.ggml.pre str = smollm
llama_model_loader: - kv 26: tokenizer.ggml.tokens arr[str,49152] = ["<|endoftext|>", "<|im_start|>", "<|...
llama_model_loader: - kv 27: tokenizer.ggml.token_type arr[i32,49152] = [3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, ...
llama_model_loader: - kv 28: tokenizer.ggml.merges arr[str,48900] = ["Ġ t", "Ġ a", "i n", "h e", "Ġ Ġ...
llama_model_loader: - kv 29: tokenizer.ggml.bos_token_id u32 = 1
llama_model_loader: - kv 30: tokenizer.ggml.eos_token_id u32 = 2
llama_model_loader: - kv 31: tokenizer.ggml.unknown_token_id u32 = 0
llama_model_loader: - kv 32: tokenizer.ggml.padding_token_id u32 = 2
llama_model_loader: - kv 33: tokenizer.chat_template str = {% for message in messages %}{% if lo...
llama_model_loader: - kv 34: tokenizer.ggml.add_space_prefix bool = false
llama_model_loader: - kv 35: tokenizer.ggml.add_bos_token bool = false
llama_model_loader: - kv 36: general.quantization_version u32 = 2
llama_model_loader: - type f32: 49 tensors
llama_model_loader: - type q8_0: 169 tensors
llm_load_vocab: special tokens cache size = 17
llm_load_vocab: token to piece cache size = 0.3170 MB
llm_load_print_meta: format = GGUF V3 (latest)
llm_load_print_meta: arch = llama
llm_load_print_meta: vocab type = BPE
llm_load_print_meta: n_vocab = 49152
llm_load_print_meta: n_merges = 48900
llm_load_print_meta: vocab_only = 0
llm_load_print_meta: n_ctx_train = 8192
llm_load_print_meta: n_embd = 2048
llm_load_print_meta: n_layer = 24
llm_load_print_meta: n_head = 32
llm_load_print_meta: n_head_kv = 32
llm_load_print_meta: n_rot = 64
llm_load_print_meta: n_swa = 0
llm_load_print_meta: n_embd_head_k = 64
llm_load_print_meta: n_embd_head_v = 64
llm_load_print_meta: n_gqa = 1
llm_load_print_meta: n_embd_k_gqa = 2048
llm_load_print_meta: n_embd_v_gqa = 2048
llm_load_print_meta: f_norm_eps = 0.0e+00
llm_load_print_meta: f_norm_rms_eps = 1.0e-05
llm_load_print_meta: f_clamp_kqv = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale = 0.0e+00
llm_load_print_meta: n_ff = 8192
llm_load_print_meta: n_expert = 0
llm_load_print_meta: n_expert_used = 0
llm_load_print_meta: causal attn = 1
llm_load_print_meta: pooling type = 0
llm_load_print_meta: rope type = 0
llm_load_print_meta: rope scaling = linear
llm_load_print_meta: freq_base_train = 130000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_ctx_orig_yarn = 8192
llm_load_print_meta: rope_finetuned = unknown
llm_load_print_meta: ssm_d_conv = 0
llm_load_print_meta: ssm_d_inner = 0
llm_load_print_meta: ssm_d_state = 0
llm_load_print_meta: ssm_dt_rank = 0
llm_load_print_meta: ssm_dt_b_c_rms = 0
llm_load_print_meta: model type = ?B
llm_load_print_meta: model ftype = Q8_0
llm_load_print_meta: model params = 1.71 B
llm_load_print_meta: model size = 1.69 GiB (8.50 BPW)
llm_load_print_meta: general.name = SmolLM2 1.7B Instruct
llm_load_print_meta: BOS token = 1 '<|im_start|>'
llm_load_print_meta: EOS token = 2 '<|im_end|>'
llm_load_print_meta: EOT token = 0 '<|endoftext|>'
llm_load_print_meta: UNK token = 0 '<|endoftext|>'
llm_load_print_meta: PAD token = 2 '<|im_end|>'
llm_load_print_meta: LF token = 143 'Ä'
llm_load_print_meta: EOG token = 0 '<|endoftext|>'
llm_load_print_meta: EOG token = 2 '<|im_end|>'
llm_load_print_meta: max token length = 162
llm_load_tensors: CPU_Mapped model buffer size = 1734.38 MiB
................................................................................................
llama_new_context_with_model: n_seq_max = 1
llama_new_context_with_model: n_ctx = 4096
llama_new_context_with_model: n_ctx_per_seq = 4096
llama_new_context_with_model: n_batch = 2048
llama_new_context_with_model: n_ubatch = 512
llama_new_context_with_model: flash_attn = 0
llama_new_context_with_model: freq_base = 130000.0
llama_new_context_with_model: freq_scale = 1
llama_new_context_with_model: n_ctx_per_seq (4096) < n_ctx_train (8192) -- the full capacity of the model will not be utilized
llama_kv_cache_init: CPU KV buffer size = 768.00 MiB
llama_new_context_with_model: KV self size = 768.00 MiB, K (f16): 384.00 MiB, V (f16): 384.00 MiB
llama_new_context_with_model: CPU output buffer size = 0.19 MiB
llama_new_context_with_model: CPU compute buffer size = 280.01 MiB
llama_new_context_with_model: graph nodes = 774
llama_new_context_with_model: graph splits = 1
common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
srv init: initializing slots, n_slots = 1
slot init: id 0 | task -1 | new slot n_ctx_slot = 4096
main: model loaded
main: chat template, built_in: 1, chat_example: '<|im_start|>system
You are a helpful assistant<|im_end|>
<|im_start|>user
Hello<|im_end|>
<|im_start|>assistant
Hi there<|im_end|>
<|im_start|>user
How are you?<|im_end|>
<|im_start|>assistant
'
main: server is listening on http://127.0.0.1:8080 - starting the main loop
srv update_slots: all slots are idle

And now we can access the web UI on http://127.0.0.1:8080 or whatever host/port combo you’ve set.

From this point, we can freely chat with the LLM using the web UI, or you can use the OpenAI-compatible API that llama-server provides. I won’t dig into the API itself here, i’ve written a Python library for it if you’re interested in using it (i’m trying to keep it up-to-date with llama.cpp master, but it might not be all the time). I recommend looking into the llama-server source code and README for more details about endpoints.

Let’s see what we can do with web UI. On the left, we have list of conversations. Those are stored in browser’s localStorage (as the disclaimer on the bottom-left graciously explains), which means they are persistent even if you restart the browser. Keep in mind that changing the host/port of the server will “clear” those. Current conversation is passed to the LLM as context, and the context size is limited by server settings (we will learn how to tweak it in a second). I recommend making new conversations often, and keeping their context focused on the subject for optimal performance.

On the top-right, we have (from left to right) “remove conversation”, “download conversation” (in JSON format), “configuration” and “Theme” buttons. In the configuration window, we can tweak generation settings for our LLM. Those are currently global, not per-conversation. All of those settings are briefly described below.

llama.cpp server settings

Web UI provides only a small subset of configuration options we have available, and only those related to LLM samplers. For the full set, we need to call llama-server --help. With those options we can drastically improve (or worsen) the behavior of our model and performance of text generation, so it’s worth knowing them. I won’t explain all of the options listed there, because it would be mostly redundant, but i’ll probably explain most of them in one way of another here. I’ll try to explain all interesting options though.

One thing that’s also worth mentioning is that most of those parameters are read from the environment. This is also the case for most other llama.cpp executables, and the parameter names (and environment variables) are the same for them. Names of those variables are provided in llama-server --help output, i’ll add them to each described option here.

Let’s start with common params section:

--threads/--threads-batch (LLAMA_ARG_THREADS) - amount of CPU threads used by LLM. Default value is -1, which tells llama.cpp to detect the amount of cores in the system. This behavior is probably good enough for most of people, so unless you have exotic hardware setup and you know what you’re doing - leave it on default. If you do have an exotic setup, you may also want to look at other NUMA and offloading-related flags.
--ctx-size (LLAMA_ARG_CTX_SIZE) - size of the prompt context. In other words, the amount of tokens that the LLM can remember at once. Increasing the context size also increases the memory requirements for the LLM. Every model has a context size limit, when this argument is set to 0, llama.cpp tries to use it.
--predict (LLAMA_ARG_N_PREDICT) - number of tokens to predict. When LLM generates text, it stops either after generating end-of-message token (when it decides that the generated sentence is over), or after hitting this limit. Default is -1, which makes the LLM generate text ad infinitum. If we want to limit it to context size, we can set it to -2.
--batch-size/--ubatch-size (LLAMA_ARG_BATCH/LLAMA_ARG_UBATCH) - amount of tokens fed to the LLM in single processing step. Optimal value of those arguments depends on your hardware, model, and context size - i encourage experimentation, but defaults are probably good enough for start.
--flash-attn (LLAMA_ARG_FLASH_ATTN) - Flash attention is an optimization that’s supported by most recent models. Read the linked article for details, in short - enabling it should improve the generation performance for some models. llama.cpp will simply throw a warning when a model that doesn’t support flash attention is loaded, so i keep it on at all times without any issues.
--mlock (LLAMA_ARG_MLOCK) - this option is called exactly like Linux function that it uses underneath. On Windows, it uses VirtualLock. If you have enough virtual memory (RAM or VRAM) to load the whole model into, you can use this parameter to prevent OS from swapping it to the hard drive. Enabling it can increase the performance of text generation, but may slow everything else down in return if you hit the virtual memory limit of your machine.
--no-mmap (LLAMA_ARG_NO_MMAP) - by default, llama.cpp will map the model to memory (using mmap on Linux and CreateFileMappingA on Windows). Using this switch will disable this behavior.
--gpu-layers (LLAMA_ARG_N_GPU_LAYERS) - if GPU offloading is available, this parameter will set the maximum amount of LLM layers to offload to GPU. Number and size of layers is dependent on the used model. Usually, if we want to load the whole model to GPU, we can set this parameter to some unreasonably large number like 999. For partial offloading, you must experiment yourself. llama.cpp must be built with GPU support, otherwise this option will have no effect. If you have multiple GPUs, you may also want to look at --split-mode and --main-gpu arguments.
--model (LLAMA_ARG_MODEL) - path to the GGUF model file.

Most of the options from sampling params section are described in detail below. Server-specific arguments are:

--no-context-shift (LLAMA_ARG_NO_CONTEXT_SHIFT) - by default, when context is filled up, it will be shifted (“oldest” tokens are discarded in favour of freshly generated ones). This parameter disables that behavior, and it will make the generation stop instead.
--cont-batching (LLAMA_ARG_CONT_BATCHING) - continuous batching allows processing prompts in parallel with text generation. This usually improves performance and is enabled by default. You can disable it with --no-cont-batching (LLAMA_ARG_NO_CONT_BATCHING) parameter.
--alias (LLAMA_ARG_ALIAS) - Alias for the model name, used by the REST API. Set to model name by default.
--host (LLAMA_ARG_HOST) and --port (LLAMA_ARG_PORT) - host and port for llama.cpp server.
--slots (LLAMA_ARG_ENDPOINT_SLOTS) - enables /slots endpoint of llama.cpp server.
--props (LLAMA_ARG_ENDPOINT_PROPS) - enables /props endpoint of llama.cpp server.

other llama.cpp tools

Webserver is not the only thing llama.cpp provides. There’s few other useful tools hidden in built binaries.

`llama-bench`

llama-bench allows us to benchmark the prompt processing and text generation speed of our llama.cpp build for a selected model. To run an example benchmark, we can simply run the executable with path to selected model.

llama-bench --model selected_model.gguf

llama-bench will try to use optimal llama.cpp configuration for your hardware. Default settings will try to use full GPU offloading (99 layers) and mmap. I recommend enabling flash attention manually (with --flash-attn flag, unfortunately llama-bench does not read the environmental variables) Tweaking prompt length (--n-prompt) and batch sizes (--batch-size/--ubatch-size) may affect the result of prompt processing benchmark. Tweaking number of tokens to generate (--n-gen) may affect the result of text generation benchmark. You can also set the number of repetitions with --repetitions argument.

Results for SmolLM2 1.7B Instruct quantized to Q8 w/ flash attention on my setup (CPU only, Ryzen 5900X, DDR4 RAM @3200MHz):

> llama-bench --flash-attn 1 --model ./SmolLM2.q8.gguf

| model | size | params | backend | threads | fa | test | t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | -: | ------------: | -------------------: |
| llama ?B Q8_0 | 1.69 GiB | 1.71 B | CPU | 12 | 1 | pp512 | 162.54 ± 1.70 |
| llama ?B Q8_0 | 1.69 GiB | 1.71 B | CPU | 12 | 1 | tg128 | 22.50 ± 0.05 |

build: dc223440 (4215)

The table is mostly self-explanatory, except two last columns. test contains the benchmark identifier, made from two parts. First two letters define the bench type (pp for prompt processing, tg for text generation). The number defines the prompt size (for prompt processing benchmark) or amount of generated tokens (for text generation benchmark). t/s column is the result in tokens processed/generated per second.

There’s also a mystery -pg argument, which can be used to perform mixed prompt processing+text generation test. For example:

> llama-bench --flash-attn 1 --model ./SmolLM2.q8.gguf -pg 1024,256

| model | size | params | backend | threads | fa | test | t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | -: | ------------: | -------------------: |
| llama ?B Q8_0 | 1.69 GiB | 1.71 B | CPU | 12 | 1 | pp512 | 165.50 ± 1.95 |
| llama ?B Q8_0 | 1.69 GiB | 1.71 B | CPU | 12 | 1 | tg128 | 22.44 ± 0.01 |
| llama ?B Q8_0 | 1.69 GiB | 1.71 B | CPU | 12 | 1 | pp1024+tg256 | 63.51 ± 4.24 |

build: dc223440 (4215)

This is probably the most realistic benchmark, because as long as you have continuous batching enabled you’ll use the model like that.

`llama-cli`

This is a simple CLI interface for the LLM. It allows you to generate a completion for specified prompt, or chat with the LLM.

It shares most arguments with llama-server, except some specific ones:

--prompt - can also be used with llama-server, but here it’s bit more useful. Sets the starting/system prompt for the LLM. Prompt can also be loaded from file by specifying it’s path using --file or --binary-file argument.
--color - enables colored output, it’s disabled by default.
--no-context-shift (LLAMA_ARG_NO_CONTEXT_SHIFT) - does the same thing as in llama-server.
--reverse-prompt - when LLM generates a reverse prompt, it stops generation and returns the control over conversation to the user, allowing him to respond. Basically, this is list of stopping words/sentences.
--conversation - enables conversation mode by enabling interactive mode and not printing special tokens (like those appearing in chat template) This is probably how you want to use this program.
--interactive - enables interactive mode, allowing you to chat with the LLM. In this mode, the generation starts right away and you should set the --prompt to get any reasonable output. Alternatively, we can use --interactive-first to start chatting with control over chat right away.

Here are specific usage examples:

text completion

> llama-cli --flash-attn --model ./SmolLM2.q8.gguf --prompt "The highest mountain on earth"

build: 4215 (dc223440) with cc (GCC) 14.2.1 20240910 for x86_64-pc-linux-gnu
main: llama backend init
main: load the model and apply lora adapter, if any
llama_model_loader: loaded meta data with 37 key-value pairs and 218 tensors from ./SmolLM2.q8.gguf (version GGUF V3 (latest))
...
llm_load_tensors: CPU_Mapped model buffer size = 1734,38 MiB
................................................................................................
llama_new_context_with_model: n_seq_max = 1
llama_new_context_with_model: n_ctx = 4096
llama_new_context_with_model: n_ctx_per_seq = 4096
llama_new_context_with_model: n_batch = 2048
llama_new_context_with_model: n_ubatch = 512
llama_new_context_with_model: flash_attn = 1
llama_new_context_with_model: freq_base = 130000,0
llama_new_context_with_model: freq_scale = 1
llama_new_context_with_model: n_ctx_per_seq (4096) < n_ctx_train (8192) -- the full capacity of the model will not be utilized
llama_kv_cache_init: CPU KV buffer size = 768,00 MiB
llama_new_context_with_model: KV self size = 768,00 MiB, K (f16): 384,00 MiB, V (f16): 384,00 MiB
llama_new_context_with_model: CPU output buffer size = 0,19 MiB
llama_new_context_with_model: CPU compute buffer size = 104,00 MiB
llama_new_context_with_model: graph nodes = 679
llama_new_context_with_model: graph splits = 1
common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
main: llama threadpool init, n_threads = 12

system_info: n_threads = 12 (n_threads_batch = 12) / 24 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | LLAMAFILE = 1 | AARCH64_REPACK = 1 |

sampler seed: 2734556630
sampler params:
 repeat_last_n = 64, repeat_penalty = 1,000, frequency_penalty = 0,000, presence_penalty = 0,000
 dry_multiplier = 0,000, dry_base = 1,750, dry_allowed_length = 2, dry_penalty_last_n = -1
 top_k = 40, top_p = 0,950, min_p = 0,050, xtc_probability = 0,000, xtc_threshold = 0,100, typical_p = 1,000, temp = 0,800
 mirostat = 0, mirostat_lr = 0,100, mirostat_ent = 5,000
sampler chain: logits -> logit-bias -> penalties -> dry -> top-k -> typical -> top-p -> min-p -> xtc -> temp-ext -> dist
generate: n_ctx = 4096, n_batch = 2048, n_predict = -1, n_keep = 0

The highest mountain on earth is Mount Everest, which stands at an astonishing 8,848.86 meters (29,031.7 feet) above sea level. Located in the Mahalangur Sharhungtrigangla Range in the Himalayas, it's a marvel of nature that draws adventurers and thrill-seekers from around the globe.

Standing at the base camp, the mountain appears as a majestic giant, its rugged slopes and snow-capped peaks a testament to its formidable presence. The climb to the summit is a grueling challenge that requires immense physical and mental fortitude, as climbers must navigate steep inclines, unpredictable weather, and crevasses.

The ascent begins at Base Camp, a bustling hub of activity, where climbers gather to share stories, exchange tips, and prepare for the climb ahead. From Base Camp, climbers make their way to the South Col, a precarious route that offers breathtaking views of the surrounding landscape. The final push to the summit involves a grueling ascent up the steep and treacherous Lhotse Face, followed by a scramble up the near-vertical wall of the Western Cwm.

Upon reaching the summit, climbers are rewarded with an unforgettable sight: the majestic Himalayan range unfolding before them, with the sun casting a golden glow on the snow. The sense of accomplishment and awe is indescribable, and the experience is etched in the memories of those who have conquered this mighty mountain.

The climb to Everest is not just about reaching the summit; it's an adventure that requires patience, perseverance, and a deep respect for the mountain. Climbers must be prepared to face extreme weather conditions, altitude sickness, and the ever-present risk of accidents or crevasses. Despite these challenges, the allure of Everest remains a powerful draw, inspiring countless individuals to push their limits and push beyond them. [end of text]


llama_perf_sampler_print: sampling time = 12,58 ms / 385 runs ( 0,03 ms per token, 30604,13 tokens per second)
llama_perf_context_print: load time = 318,81 ms
llama_perf_context_print: prompt eval time = 59,26 ms / 5 tokens ( 11,85 ms per token, 84,38 tokens per second)
llama_perf_context_print: eval time = 17797,98 ms / 379 runs ( 46,96 ms per token, 21,29 tokens per second)
llama_perf_context_print: total time = 17891,23 ms / 384 tokens

chat mode

> llama-cli --flash-attn --model ./SmolLM2.q8.gguf --prompt "You are a helpful assistant" --conversation

build: 4215 (dc223440) with cc (GCC) 14.2.1 20240910 for x86_64-pc-linux-gnu
main: llama backend init
main: load the model and apply lora adapter, if any
llama_model_loader: loaded meta data with 37 key-value pairs and 218 tensors from ./SmolLM2.q8.gguf (version GGUF V3 (latest))
...
llm_load_tensors: CPU_Mapped model buffer size = 1734,38 MiB
................................................................................................
llama_new_context_with_model: n_seq_max = 1
llama_new_context_with_model: n_ctx = 4096
llama_new_context_with_model: n_ctx_per_seq = 4096
llama_new_context_with_model: n_batch = 2048
llama_new_context_with_model: n_ubatch = 512
llama_new_context_with_model: flash_attn = 1
llama_new_context_with_model: freq_base = 130000,0
llama_new_context_with_model: freq_scale = 1
llama_new_context_with_model: n_ctx_per_seq (4096) < n_ctx_train (8192) -- the full capacity of the model will not be utilized
llama_kv_cache_init: CPU KV buffer size = 768,00 MiB
llama_new_context_with_model: KV self size = 768,00 MiB, K (f16): 384,00 MiB, V (f16): 384,00 MiB
llama_new_context_with_model: CPU output buffer size = 0,19 MiB
llama_new_context_with_model: CPU compute buffer size = 104,00 MiB
llama_new_context_with_model: graph nodes = 679
llama_new_context_with_model: graph splits = 1
common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
main: llama threadpool init, n_threads = 12
main: chat template example:
<|im_start|>system
You are a helpful assistant<|im_end|>
<|im_start|>user
Hello<|im_end|>
<|im_start|>assistant
Hi there<|im_end|>
<|im_start|>user
How are you?<|im_end|>
<|im_start|>assistant


system_info: n_threads = 12 (n_threads_batch = 12) / 24 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | LLAMAFILE = 1 | AARCH64_REPACK = 1 |

main: interactive mode on.
sampler seed: 968968654
sampler params:
 repeat_last_n = 64, repeat_penalty = 1,000, frequency_penalty = 0,000, presence_penalty = 0,000
 dry_multiplier = 0,000, dry_base = 1,750, dry_allowed_length = 2, dry_penalty_last_n = -1
 top_k = 40, top_p = 0,950, min_p = 0,050, xtc_probability = 0,000, xtc_threshold = 0,100, typical_p = 1,000, temp = 0,800
 mirostat = 0, mirostat_lr = 0,100, mirostat_ent = 5,000
sampler chain: logits -> logit-bias -> penalties -> dry -> top-k -> typical -> top-p -> min-p -> xtc -> temp-ext -> dist
generate: n_ctx = 4096, n_batch = 2048, n_predict = -1, n_keep = 0

== Running in interactive mode. ==
 - Press Ctrl+C to interject at any time.
 - Press Return to return control to the AI.
 - To return control without starting a new line, end your input with '/'.
 - If you want to submit another line, end your input with '\'.

system
You are a helpful assistant

> hi
Hello! How can I help you today?

>
llama_perf_sampler_print: sampling time = 0,27 ms / 22 runs ( 0,01 ms per token, 80291,97 tokens per second)
llama_perf_context_print: load time = 317,46 ms
llama_perf_context_print: prompt eval time = 2043,02 ms / 22 tokens ( 92,86 ms per token, 10,77 tokens per second)
llama_perf_context_print: eval time = 407,66 ms / 9 runs ( 45,30 ms per token, 22,08 tokens per second)
llama_perf_context_print: total time = 5302,60 ms / 31 tokens
Interrupted by user

building the llama, but better

All right, now that we know how to use llama.cpp and tweak runtime parameters, let’s learn how to tweak build configuration. We already set some generic settings in chapter about building the llama.cpp but we haven’t touched any backend-related ones yet.

Back-end for llama.cpp is provided via ggml library (created by the same author!). This library contains the code for math operations used to run LLMs, and it supports many hardware accelerations that we can enable to get the maximum LLM performance on our hardware. Let’s start with clearing up the llama.cpp repository (and, optionally, making sure that we have the latest commit):

cd llama.cpp
git clean -xdf
git pull
git submodule update --recursive

Now, we need to generate the build files for a custom backend. As of writing this, the list of backends supported by ggml library is following:

Metal - acceleration for Apple Silicon
Accelerate - BLAS (Basic Linear Algebra Subprograms) acceleration for Mac PCs, enabled by default.
OpenBLAS - BLAS acceleration for CPUs
BLIS - relatively recently released high-performance BLAS framework
SYCL - acceleration for Intel GPUs (Data Center Max series, Flex series, Arc series, Built-in GPUs and iGPUs)
Intel oneMKL - acceleration for Intel CPUs
CUDA - acceleration for Nvidia GPUs
MUSA - acceleration for Moore Threads GPUs
hipBLAS - BLAS acceleration for AMD GPUs
Vulkan - generic acceleration for GPUs
CANN - acceleration for Ascend NPU

As we can see, there’s something for everyone. It’s good to note here that llama.cpp also supports Android! My backend selection recommendation is following:

Users without GPUs should try Intel oneMKL in case of Intel CPUs, or BLIS/OpenBLAS.
Users with Nvidia GPUs should use CUDA or Vulkan
Users with AMD GPUs should use Vulkan or ROCm (order important here, ROCm was bugged last time i’ve used it)
Users with Intel GPUs should use SYCL or Vulkan

As we can see, Vulkan is the most generic option for GPU acceleration and i believe it’s the simplest to build for, so i’ll explain in detail how to do that. Doesn’t matter whether you’re Nvidia, AMD or Intel graphics user - Vulkan implementation should work on most (if not all) of their GPUs, as long as they’re reasonably modern. The build process for every backend is very similar - install the necessary dependencies, generate the llama.cpp build files with proper flag to enable the specific backend, and build it.

Oh, and don’t worry about Python and it’s dependencies. The performance of model conversion scripts is not limited by pytorch, so there’s no point in installing CUDA/ROCm versions.

Before generating the build file, we need to install Vulkan SDK.

On Windows, it’s easiest to do via MSYS. That’s why i’ve recommended using it at the beginning. I have tried installing it directly on Windows, but encountered issues that i haven’t seen when using MSYS - so, obviously, MSYS is a better option. Run this command in MSYS (make sure to use UCRT runtime) to install the required dependencies:

pacman -S git \
 mingw-w64-ucrt-x86_64-gcc \
 mingw-w64-ucrt-x86_64-cmake \
 mingw-w64-ucrt-x86_64-vulkan-devel \
 mingw-w64-ucrt-x86_64-shaderc

If you really don’t want to use MSYS, i recommend following the docs

On Linux, i recommend installing Vulkan SDK using the package manager. If it’s not in package manager of your distro, i assume you know what you’re doing and how to install it manually.

Afterwards, we can generate the build files. To enable Vulkan back-end, we need to set GGML_VULKAN variable to ON. Replace /your/install/dir with custom installation directory, if you want one - otherwise remove this variable:

cmake -S . -B build -G Ninja -DGGML_VULKAN=ON -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/your/install/dir -DLLAMA_BUILD_TESTS=OFF -DLLAMA_BUILD_EXAMPLES=ON -DLLAMA_BUILD_SERVER=ON

and build/install the binaries (replace X with amount of cores in your system):

cmake --build build --config Release -j X
cmake --install build --config Release

Don’t mind the warnings you’ll see, i get them too. Now our llama.cpp binaries should be able to use our GPU. We can test it by running llama-server or llama-cli with --list-devices argument:

> llama-cli --list-devices

ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon RX 7900 XT (AMD open-source driver) | uma: 0 | fp16: 1 | warp size: 64
Available devices:
 Vulkan0: AMD Radeon RX 7900 XT (20464 MiB, 20464 MiB free)

Running that command previously would print an empty list:

> llama-cli --list-devices

Available devices:

Remember the llama-bench results i’ve got previously on CPU build? This is them now:

> llama-bench --flash-attn 1 --model ./SmolLM2.q8.gguf -pg 1024,256

ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon RX 7900 XT (AMD open-source driver) | uma: 0 | fp16: 1 | warp size: 64
| model | size | params | backend | ngl | fa | test | t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -: | ------------: | -------------------: |
ggml_vulkan: Compiling shaders..............................Done!
| llama ?B Q8_0 | 1.69 GiB | 1.71 B | Vulkan | 99 | 1 | pp512 | 880.55 ± 5.30 |
| llama ?B Q8_0 | 1.69 GiB | 1.71 B | Vulkan | 99 | 1 | tg128 | 89.78 ± 1.66 |
| llama ?B Q8_0 | 1.69 GiB | 1.71 B | Vulkan | 99 | 1 | pp1024+tg256 | 115.25 ± 0.83 |

Now, that’s some good shit right here. Prompt processing speed has increased from ~165 to ~880 tokens per second (5.3x faster). Text generation - from ~22 to ~90 tokens per second (4x faster). Mixed text processing went up from ~63 to ~115 tokens per second (1.8x faster). All of this due to the fact that i’ve switched to correct backend.

need better CPU support?

Vulkan back-end is a good generic way of running the LLMs on GPU. But what if you don’t have one, and still want to get the maximum performance out of your hardware? Fortunately, ggml library has support for optional CPU instruction sets that provide faster math operations for LLMs, like AVX.

All the configuration options for ggml library are defined in ggml/CMakeLists.txt file of llama.cpp repo. You set them just like you did when building Vulkan, -DVARIABLE_NAME=VALUE when calling cmake for project generation. I’m gonna list CPU-related ones here, and explain what they do and when you should use them (if i know that). This list may become outdated, ping me somewhere when ggml gets updated and i don’t notice the change! You should still double-check and validate this list with latest CMakeLists.txt of ggml.

To quickly check which instruction sets your CPU supports, i recommend HWInfo

GGML_CPU - enables CPU back-end. This is enabled by default and usually should not be changed, even when GPU back-end is enabled.
GGML_CPU_HBM - enables memkind allocator, mostly for systems with CPU HBM memory.
GGML_CPU_AARCH64 - enables support for AARCH64 (Armv8-A or newer) architecture.
GGML_AVX - enables AVX support. If you’re not cross-compiling, should be enabled by default - most modern CPUs support it.
GGML_AVX_VNNI - enables AVX-VNNI support. This is a sub-set of AVX instructions introduced in Intel’s Alder Lake (Q4 2021) and AMD Zen 5 (Q3 2024) CPUs or newer. Enable it if your CPU supports it.
GGML_AVX2 - similar to GGML_AVX, also enabled by default if not cross-compiling.
GGML_AVX512 - enables AVX512F support. Check the linked article for the list of supported CPUs. Enable if supported by your CPU.
GGML_AVX512_VBMI/VNNI/BF16 - those three options control the optimizations for specific AVX512 subsets. Article linked above also provides info about support of those subsets, so enable it accordingly to your hardware.
GGML_LASX/LSX - enables support for LoongArch (Loongson CPUs)
GGML_RVV - enables support for RISC-V Vector Extension, affects only supported RISC-V CPUs.
GGML_SVE - enables support for ARM Scalable Vector Extensions. Available on some ARM CPUs, like Armv9 Cortex-A510.

Following options are not supported under MSVC (another reason to not use it):

GGML_FMA - enables optimizations for FMA instruction set, every modern CPU should support them and it’s enabled by default if not cross-compiling.
GGML_F16C - enables optimizations for F16C instruction set, similarly to FMA - every modern CPU supports it, enabled by default when not cross-compiling.
GGML_AMX_TILE/INT8/BF16 - enables support for Intel’s Advanced Matrix Extensions. This is available only on recent Xeon CPUs.

what about CUDA/ROCm/BLAS?

I’ll write about ROCm in another blog post i’m preparing (with some benchmarks and comparisons). For CUDA and BLAS vendors, i’ll have to refer you to the docs. They are clear enough, and if you were able to build llama.cpp for CPU and you have your build environment set up correctly (with all requirements installed) - it should be equally easy to build it with custom backend support.

LLM configuration options explained

This will be a relatively long and very informational part full of boring explanations. But - it’s a good knowledge to have when playing with LLMs.

how does LLM generate text?

Prompt

Everything starts with a prompt. Prompt can be a simple raw string that we want the LLM to complete for us, or it can be an elaborate construction that allows the LLM to chat or use external tools. Whatever we put in it, it’s usually in human-readable format with special “tags” (usually similar to XML tags) used for separating the parts of the prompt.

We’ve already seen an example of a prompt used for chat completion, provided by llama-server:
```
<|im_start|>system
You are a helpful assistant<|im_end|>
<|im_start|>user
Hello<|im_end|>
<|im_start|>assistant
Hi there<|im_end|>
<|im_start|>user
How are you?<|im_end|>
<|im_start|>assistant
```
(if you’re wondering what are those funny <| and |> symbols - those are ligatures from Fira Code font made out of |, > and < characters)
Tokenization

The LLM does not understand the human language like we do. We use words and punctuation marks to form sentences - LLMs use tokens that can be understood as an equivalent to those. First step in text generation is breaking the language barrier by performing prompt tokenization. Tokenization is a process of translating input text (in human-readable format) into an array of tokens that can be processed by an LLM. Tokens are simple numeric values, and with a vocabulary they can be easily mapped to their string representations (at least in case of BPE models, don’t know about others). In fact, that vocabulary is available in SmolLM2 repository, in tokenizer.json file! That file also contains some metadata for special tokens that have special meaning for the LLM. Some of those tokens represent meta things, like start and end of a message. Other can allow the LLM to chat with the user by providing tags for separating parts of conversation (system prompt, user messages, LLM responses). I’ve also seen tool calling capabilities in LLM templates, which in theory should allow the LLM to use external tools, but i haven’t tested them yet (check out Qwen2.5 and CodeQwen2.5 for example models with those functions).

We can use llama-server API to tokenize some text and see how it looks after being translated. Hope you’ve got curl.
```
curl -X POST -H "Content-Type: application/json" -d '{"content": "hello world! this is an example message!"}' http://127.0.0.1:8080/tokenize
```
For SmolLM2, the response should be following:
```
{"tokens":[28120,905,17,451,314,1183,3714,17]}
```
Which we can very roughly translate to:
- 28120 - hello
- 905 - world
- 17 - !
- 451 - this
- 314 - is
- 354 - an
- 1183 - example
- 3714 - message
- 17 - !
We can pass this JSON back to /detokenize endpoint to get our original text back:
```
curl -X POST -H "Content-Type: application/json" -d '{"tokens": [28120,905,17,451,314,354,1183,3714,17]}' http://127.0.0.1:8080/detokenize
```
```
{"content":"hello world! this is an example message!"}
```
Dank Magick (feeding the beast)

I honestly don’t know what exactly happens in this step, but i’ll try my best to explain it in simple and very approximate terms. The input is the tokenized prompt. This prompt is fed to the LLM, and the digestion process takes a lot of processing time due to the insane amount of matrix operations that must be performed to satisfy the digital beast. After the prompt is digested, the LLM starts “talking” to us. LLM “talks” by outputting the probability distribution for all the tokens in it’s dictionary. This distribution is a list of pairs containing the token value and it’s probability to be used as the next one. Now, in theory we could use a single token with the highest probability and be done with generation, but that’s not the best way of doing things.
Token sampling

This is probably the most interesting step for us, because we can control it’s every single parameter. As usual, i advise caution when working with raw output from demons - digital or not, it may result in unexpected stuff happening when handled incorrectly. Samplers are used to choose the “best” token from the raw LLM output. They work by trimming down all the available tokens to a single one. Note that not every sampler will do that directly (for example, Top-K picks K tokens), but in the end of tokenization, the result is a single token.
Detokenization

Generated tokens must be converted back to human-readable form, so a detokenization must take place. This is the last step. Hooray, we tamed the digital beast and forced it to talk. I have previously feared the consequences this could bring upon the humanity, but here we are, 373 1457 260 970 1041 3935.

list of LLM configuration options and samplers available in llama.cpp

System Message - Usually, conversations with LLMs start with a “system” message that tells the LLM how to behave. This is probably the easiest-to-use tool that can drastically change the behavior of a model. My recommendation is to put as much useful informations and precise behavior descriptions for your application as possible, to maximize the quality of LLM output. You may think that giving the digital demon maximum amount of knowledge may lead to bad things happening, but our reality haven’t collapsed yet so i think we’re good for now.
Temperature - per llama.cpp docs “Controls the randomness of the generated text by affecting the probability distribution of the output tokens. Higher = more random, lower = more focused”. I don’t have anything to add here, it controls the “creativity” of an LLM. High values result in more random and “creative” output, but overcooking the beast may result in hallucinations, and - in certain scenarios - screams that will destroy your sanity. Keep it in 0.2-2.0 range for a start, and keep it positive and non-zero.
Dynamic temperature - Dynamic temperature sampling is an addition to temperature sampler. The linked article describes it in detail, and it’s pretty short so i strongly recommend reading it - i can’t really do a better job explaining it. There’s also the reddit post with more explanations from the algorithm’s author. However, in case the article goes down - the short explanation of this algorithm is: it tweaks the temperature of generated tokens based on their entropy. Entropy here can be understood as inverse of LLMs confidence in generated tokens. Lower entropy means that the LLM is more confident in it’s predictions, and therefore the temperature of tokens with low entropy should also be low. High entropy works the other way around. Effectively, this sampling can encourage creativity while preventing hallucinations at higher temperatures. I strongly recommend testing it out, as it’s usually disabled by default. It may require some additional tweaks to other samplers when enabled, to produce optimal results. The parameters of dynamic temperature sampler are:
- Dynatemp range - the range of dynamic temperature to be added/subtracted
- Dynatemp exponent - changing the exponent changes the dynamic temperature in following way (figure shamelessly stolen from previously linked reentry article):
Top-K - Top-K sampling is a fancy name for “keep only K most probable tokens” algorithm. Higher values can result in more diverse text, because there’s more tokens to choose from when generating responses.
Top-P - Top-P sampling, also called nucleus sampling, per llama.cpp docs “Limits the tokens to those that together have a cumulative probability of at least p”. In human language, it means that the Top-P sampler takes a list of tokens and their probabilities as an input (note that the sum of their cumulative probabilities is by definition equal to 1), and returns tokens with highest probabilities from that list until the sum of their cumulative probabilities is greater or equal to p. Or, in other words, p value changes the % of tokens returned by the Top-P sampler. For example, when p is equal to 0.7, the sampler will return 70% of input tokens with highest probabilities. There’s a pretty good article about temperature, Top-K and Top-P sampling that i’ve found and can recommend if you wanna know more.
Min-P - Min-P sampling, per llama.cpp docs “Limits tokens based on the minimum probability for a token to be considered, relative to the probability of the most likely token”. There’s a paper explaining this algorithm (it contains loads of citations for other LLM-related stuff too, good read). Figure 1 from this paper nicely shows what each of the sampling algorithms does to probability distribution of tokens:
Exclude Top Choices (XTC) - This is a funky one, because it works a bit differently from most other samplers. Quoting the author, Instead of pruning the least likely tokens, under certain circumstances, it removes the most likely tokens from consideration. Detailed description can be found in the PR with implementation. I recommend reading it, because i really can’t come up with anything better in few sentences, it’s a really good explanation. I can, however, steal this image from the linked PR to show you more-or-less what XTC does: The parameters for XTC sampler are:
- XTC threshold - probability cutoff threshold for top tokens, in (0, 1) range.
- XTC probability - probability of XTC sampling being applied in [0, 1] range, where 0 = XTC disabled, 1 = XTC always enabled.
Locally typical sampling (typical-P) - per llama.cpp docs “Sorts and limits tokens based on the difference between log-probability and entropy”. I… honestly don’t know how exactly it works. I tried reading the linked paper, but i lack the mental capacity to understand it enough to describe it back. Some people on Reddit also have the same issue, so i recommend going there and reading the comments. I haven’t used that sampling much, so i can’t really say anything about it from experience either, so - moving on.
DRY - This sampler is used to prevent unwanted token repetition. Simplifying, it tries to detect repeating token sequences in generated text and reduces the probabilities of tokens that will create repetitions. As usual, i recommend reading the linked PR for detailed explanation, and as usual i’ve stolen a figure from it that shows what DRY does: I’ll also quote a short explanation of this sampler: The penalty for a token is calculated as multiplier * base ^ (n - allowed_length), where n is the length of the sequence before that token that matches the end of the input, and multiplier, base, and allowed_length are configurable parameters. If the length of the matching sequence is less than allowed_length, no penalty is applied. The parameters for DRY sampler are:
- DRY multiplier - see explanation above
- DRY base - see explanation above
- DRY allowed length - see explanation above. Quoting llama.cpp docs: Tokens that extend repetition beyond this receive exponentially increasing penalty.
- DRY penalty last N - how many tokens should be scanned for repetition. -1 = whole context, 0 = disabled.
- DRY sequence breakers - characters that are used as separators for parts of sentences considered for DRY. Defaults for llama.cpp are ('\n', ':', '"', '*').
Mirostat - is a funky sampling algorithm that overrides Top-K, Top-P and Typical-P samplers. It’s an alternative sampler that produces text with controlled perplexity (entropy), which means that we can control how certain the model should be in it’s predictions. This comes without side-effects of generating repeated text (as it happens in low perplexity scenarios) or incoherent output (as it happens in high perplexity scenarios). The configuration parameters for Mirostat are:
- Mirostat version - 0 = disabled, 1 = Mirostat, 2 = Mirostat 2.0.
- Mirostat learning rate (η, eta) - specifies how fast the model converges to desired perplexity.
- Mirostat target entropy (τ, tau) - the desired perplexity. Depending on the model, it should not be too high, otherwise you may degrade it’s performance.
Max tokens - i think that’s pretty self-explanatory. -1 makes the LLM generate until it decides it’s end of the sentence (by returning end-of-sentence token), or the context is full.
Repetition penalty - Repetition penalty algorithm (not to be mistaken with DRY) simply reduces the chance that tokens that are already in the generated text will be used again. Usually the repetition penalty algorithm is restricted to N last tokens of the context. In case of llama.cpp (i’ll simplify a bit), it works like that: first, it creates a frequency map occurrences for last N tokens. Then, the current logit bias for each token is divided by repeat_penalty value. By default it’s usually set to 1.0, so to enable repetition penalty it should be set to >1. Finally, frequency and presence penalties are applied based on the frequency map. The penalty for each token is equal to (token_count * frequency_penalty) + (presence_penalty if token_count > 0). The penalty is represented as logit bias, which can be in [-100, 100] range. Negative values reduce the probability of token appearing in output, while positive increase it. The configuration parameters for repetition penalty are:
- Repeat last N - Amount of tokens from the end of the context to consider for repetition penalty.
- Repeat penalty - repeat_penalty argument described above, if equal to 1.0 then the repetition penalty is disabled.
- Presence penalty - presence_penalty argument from the equation above.
- Frequency penalty - frequency_penalty argument from the equation above.

Additional literature:

Your settings are (probably) hurting your model

In Other sampler settings we can find sampling queue configuration. As i’ve mentioned earlier, the samplers are applied in a chain. Here, we can configure the order of their application, and select which are used. The setting uses short names for samplers, the mapping is following:

d - DRY
k - Top-K
y - Typical-P
p - Top-P
m - Min-P
x - Exclude Top Choices (XTC)
t - Temperature

Some samplers and settings i’ve listed above may be missing from web UI configuration (like Mirostat), but they all can be configured via environmental variables, CLI arguments for llama.cpp binaries, or llama.cpp server API.

final thoughts

That is a long post, damn. I have started writing this post at the end of October. It’s almost December now. During that time, multiple new models have been released - including SmolLM2, fun fact - i have originally planned to use Llama 3.2 3B. The speed at which the LLM community moves and releases new stuff is absolutely incredible, but thankfully llama.cpp is relatively stable now. I hope the knowledge i’ve gathered in this post will be useful and inspiring to the readers, and will allow them to play with LLMs freely in their homes. That’s it, i’m tired. I’m releasing that shit into the wild.

Suggestions for next posts are welcome, for now i intend to make some scripts for automated benchmarks w/ llama-bench and gather some data. I’ll try to keep this post up-to-date and maybe add some stuff if it’s requested. Questions are welcome too, preferably in the comments section. Have a nice one.

bonus: where to find models, and some recommendations

My favorite site for finding models and comparing them is LLM Explorer. It’s basically a search engine for models. The UI is not exactly great but it’s good enough, it has the most comprehensive list of LLMs i’ve seen, and lots of search options.

As for my recommendations, some relatively recent models i’ve tried that made a positive impression upon me are:

Google Gemma 2 9B SimPO - a fine-tune of Google Gemma model. Gemma models are pretty interesting, and their responses are noticeably different from other models.
Meta Llama 3.1/3.2 - i recommend trying out Llama 3.1 8B Instruct, as it’s the default go-to model for most LLM applications. There’s also many finetunes and abliterated versions that don’t have any built-in restrictions available publicly.
Microsoft Phi 3.5 - a series of models from Microsoft. Most of them are small, but there’s also big MoE (Mixture of Experts) version available.
Qwen/CodeQwen 2.5 - series of models from Alibaba, currently one of the best open-source models available. At the moment of writing this, CodeQwen 14B is my daily driver model.

post-mortem

I’ve shared this post on Reddit and after getting the feedback, i managed to fix multiple issues with this post. Thanks! I’m currently at the process of refactors, so many small things may change, and many other small things may be added in following days. I’ll update this section when i’m finished.

03.12.2024 - Updated the part about LLM text generation. I have made some mistakes there, so i’ve updated my knowledge and the post. Thanks /r/AbaGuy17, /r/Oscylator and others for pointing stuff up.
10.12.2024 - Updated the back-end part, added info about CPU optimizations.
25.12.2024 - Merry christmas, i just realized that Python 3.13 doesn’t have working PyTorch package, and i’ve been running Py3.12 all along! Sorry, fixed that!

Making C/C++ project template in Meson - part 2.5

Thu, 29 Feb 2024 00:00:00 +0000

Fixing CppUTest wrap and contributing to WrapDB

As per my previous post, I have fixed the CppUTest wrap and it’s now available on WrapDB! Since this post series is strictly related to Meson, I want to talk a bit more about creating wraps and contributing there. If you don’t care, and just want to continue to the guide part - jump to the next section.

I have started the process of creating this wrap by making a very basic example project with a single binary that uses CppUTest. Then, I plugged in CppUTest as a normal subproject, and began rewriting the meson.build files, along with the options. Eventually I’ve managed to get to a point where it compiled successfully on my setup, linked to an app, and I was able to run it. Thinking that’s good enough, I’ve started worrying about wrapping it properly.

There are few different kinds of wraps that Meson supports. If the project supports Meson, CMake or Cargo it can be wrapped directly, either as an archive or a specific revision of code on a supported repository. Otherwise, it’s possible to provide an “overlay” patch that adds/replaces the files of the library in order to provide Meson support - that’s what file-wraps are.

I’ve considered it before, but to be clear - CppUTest’s primary build system is CMake, but I don’t have much pleasant memories related to CMake and I’d prefer to avoid interacting with it.

I already had the files necessary to create a file-wrap, so all I needed to do was modifying cpputest.wrap… Right? Wrong! This is where I’ve spent an hour or two, thinking that I can host this on my Github repo instead of contributing to WrapDB. And this is my first major complaint about Meson!

First thing I wanted to try was putting the wrap patch files on my Github repository, and pointing the wrap to it. I’ve figured that this way should be supported, because it enables hosting wraps on private repositories, for example in company environments. So i did that, created a release to have an archive, pointed the wrap to it, and… It did not work, because Meson kept putting the patch files in different directory than source files.

I was thinking that directory option in .wrap file defines the directory both archives will be extracted to, but that’s not the case! This is the option that defines where the source archive will be extracted to, but not the patch archive. And the output directory for patch was dependent on… The name of my repository. Duh. Effectively, I think that I’d have to name my repository with the wrap cpputest to make it working that way, and I’d rather not do that… So I’ve abandoned this idea and proceeded to read how to contribute to WrapDB. Alas, I believe that this way of providing files for file-wraps should be supported.

The contribution itself was rather pleasant experience - after moving the wrap files to the WrapDB fork, I’ve made a pull request and waited for CI to pass. I was happy to see the CI checked the code by running a Python script with no external dependencies, so I could run the checks locally. I was even more happy to see that this script actually downloaded and built every library WrapDB supports, and CI was running it on Windows, Linux and MacOS runners with Clang, GCC and MSVC. That is a nice thing, but as we experienced, it’s not quite enough to guarantee quality - even practically unusable wraps can build with no issues. I was less happy when I’ve enabled the CI on my fork and it ate 1/4th of my monthly free GitHub runner limit after a single full run. Oh well. Good thing they do incremental builds on PRs.

Of course, my PR it did not pass the CI first time - it forced me to add some functionality to the wrap (platform auto-detection), but after that it was good to go and got merged. The whole process took only a few hours, the PR was reviewed, the CI did it’s job - no complains there.

Adding unit tests support

Now we can proceed with the actual subject of this post.

Remove everything from the subprojects/ directory, open a terminal and run meson wrap install cpputest. This is the preferred method of installing wraps, and I forgot to mention it earlier. I don’t quite understand why it’s not able to create subprojects/ directory by itself by default, but it’s a non-issue.

I assume that you’ve followed the previous part, and already added tests directory to main meson.build, and created example test in tests/calc/. If not, go back and do that. If you already did that, then the project should compile successfully. The only thing I’d like to change is cpputest_dependency to just cpputest, because it looks better.

tests/meson.build:

cpputest_project = subproject('cpputest', required: true)
cpputest = cpputest_project.get_variable('cpputest_dep')

subdir('calc')
subdir('greeter')

tests/calc/meson.build:

calc_test_exec = executable('calc_test', 'test.cpp', dependencies: cpputest)
test('calc test', calc_test_exec)

Now, we should also be able to run our test via meson test command.

> meson test -C builddir
ninja: Entering directory `\builddir'
ninja: no work to do.
1/1 calc test FAIL 0.01s exit status 1
>>> MALLOC_PERTURB_=245 ASAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1 UBSAN
_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1:print_stacktrace=1 \builddir\tests\calc\calc_test.exe
Ok: 0
Expected Fail: 0
Fail: 1
Unexpected Pass: 0
Skipped: 0
Timeout: 0
Full log written to \builddir\meson-logs\testlog.txt

It will fail, because in the previous part we’ve written a test that should fail, so it’s all good. Your MALLOC_PERTURB_ may be different, as it’s randomly chosen every test run. It’s used for some magic involving malloc that helps detecting memory-related issues.

So, it seems like we have a working unit test setup. Let’s write some meaningful tests for both modules and we’ll check if we can call them separately. We also have to finish up our Meson script for calc test, because we’re not linking to calc yet, and we should! And we also have to add libs_includes to includes for our test exec.

tests/calc/meson.build

calc_test_exec = executable(
 'calc_test',
 sources: 'test.cpp',
 dependencies: cpputest,
 link_with: calc_lib,
 include_directories: libs_includes,
)

test('calc test', calc_test_exec)

And our test may look like this:

tests/calc/test.cpp

#include <CppUTest/CommandLineTestRunner.h>
#include <CppUTest/TestHarness.h>

#include <calc/calc.hpp>

TEST_GROUP(TemperatureCalcTests){};

TEST(TemperatureCalcTests, convertsCelsiusToFahrenheit) {
 auto const givenCelsius = 37.7778;
 auto const expectedFahrenheit = 100.0;
 auto const gotFahrenheit = celsius_to_fahrenheit(givenCelsius);
 DOUBLES_EQUAL(expectedFahrenheit, gotFahrenheit, 0.001);
}


TEST(TemperatureCalcTests, convertsFahrenheitToCelsius) {
 auto const givenFahrenheit = 212;
 auto const expectedCelsius = 100.0;
 auto const gotCelsius = fahrenheit_to_celsius(givenFahrenheit);
 DOUBLES_EQUAL(expectedCelsius, gotCelsius, 0.001);
}

int main(int ac, char** av) {
 return CommandLineTestRunner::RunAllTests(ac, av);
}

We should create similar meson and source file for greeter.

tests/greeter/meson.build

greeter_test_exec = executable(
 'greeter_test',
 sources: 'test.cpp',
 dependencies: cpputest,
 link_with: greeter_lib,
 include_directories: libs_includes,
)

test('greeter test', greeter_test_exec)

tests/greeter/test.cpp

#include <CppUTest/CommandLineTestRunner.h>
#include <CppUTest/TestHarness.h>

#include <greeter/greeter.hpp>
#include <string>

TEST_GROUP(GreeterTests){};

TEST(GreeterTests, greetsUser) {
 auto const user = std::string("user");
 auto const expectedMessage = std::string("Hello user");
 auto const actualMessage = greet(user);
 CHECK_EQUAL(expectedMessage, actualMessage);
}

int main(int ac, char** av) {
 return CommandLineTestRunner::RunAllTests(ac, av);
}

And now, after running meson test -C builddir, we should get something like this:

PS .> meson test -C builddir
ninja: Entering directory `.\builddir'
ninja: no work to do.
1/2 calc test OK 0.01s
2/2 greeter test OK 0.01s
Ok: 2
Expected Fail: 0
Fail: 0
Unexpected Pass: 0
Skipped: 0
Timeout: 0
Full log written to .\builddir\meson-logs\testlog.txt

And here we go, we finally have working unit tests in our project.

Summary

Oh well, this part took way too long to publish. I blame Helldivers II for that. To be fair, I finished writing it in a manner of two weeks, I just didn’t have energy to finish it and push it. Still, I’m gonna try to move it forward faster because my backlog of stuff to do is incredibly long, and this template could help. For anyone interested, there’s a tag in the repository with it’s current state. In the next part, i will show you how to configure Doxygen with Meson to generate documentation of our modules and applications functions.

Making C/C++ project template in Meson - part 2

Wed, 07 Feb 2024 00:00:00 +0000

Intro

In previous part, we’ve created the base of our project by defining it’s structure and making our first modules and executable. In this one, we’re gonna add unit tests support.

But before that…

Fixing our mistakes

So, uhh, let’s look at our project. Specifically, at our hello_world.cpp. More specifically, at includes. It’s not that something is wrong with them right now, but think about it - what would happen if we’d put an utils.hpp file in both libraries? That is actually a question that can be answered using the “fuck around and find out” method, so let’s find out.

This is the content of our utils.hpp. Adjust the returned string appropriately for the other module.

#pragma once
#include <string>

inline std::string what_am_i() { return std::string("i am calc"); }

Now, our new hello_world.cpp will become this:

#include <calc.hpp>
#include <greeter.hpp>
#include <utils.hpp>
#include <iostream>

int main() {
 std::cout << greet(what_am_i()) << std::endl;
 std::cout << "12.5*F == " << fahrenheit_to_celsius(12.5) << "*C" << std::endl;
 std::cout << "12.5*C == " << celsius_to_fahrenheit(12.5) << "*F" << std::endl;
}

Aaand… In my case, it prints Hello i am calc, which is caused by the fact that calc_includes are before greeter_includes in my hello_world/meson.build’s executable() call. After swapping them, Hello i am greeter is printed.

Now that we know what happens, we can easily deduce that it’s not supposed to work like that - because we don’t have a reasonable way to access the other utils.hpp. The simplest solution is to move the include directory one level up, to lib directory. And in retrospect, that’s how it should be done from the beginning, so excuse my blunder there.

In order to fix our mistake, we have to revisit some meson.build files. First, let’s create an include directory object in lib/meson.build.

libs_includes = include_directories('.')

subdir('calc')
subdir('greeter')

I’ve added it before the subdir() calls just in case some modules would depend on each other. This is not something we want to do often, because for every dependency we will probably have to create a mock in order to properly test the modules that use them, but it’s sometimes necessary.

After that, remove the include_directories() calls from greeter and calc modules meson.build, and fix the executable() call arguments in hello_world/meson.build.

hello_world = executable(
 'hello_world',
 'hello_world.cpp',
 link_with: [calc_lib, greeter_lib],
 include_directories: [libs_includes],
)

And fix the includes in hello_world.cpp. We can also remove the utils.hpp and restore the original hello_world.cpp.

#include <calc/calc.hpp>
#include <greeter/greeter.hpp>
#include <iostream>

int main() {
 std::cout << greet("random developer") << std::endl;
 std::cout << "12.5*F == " << fahrenheit_to_celsius(12.5) << "*C" << std::endl;
 std::cout << "12.5*C == " << celsius_to_fahrenheit(12.5) << "*F" << std::endl;
}

Also; let’s rename our lib/ directory to libs/, this is a very small change but my brain automatically tries to write libs/ instead of lib/ because there are multiple libraries there, so I think it reflects this directory’s content better. Remember to also change the argument of subdir() in the root meson.build.

Verify if the project still builds and the executable still works. Remove the builddir first, as we made pretty big change and I honestly don’t expect Meson to correctly re-generate it.

Harnessing the power of unit tests

I want to assume that You have at least minimal experience with testing your code, but i know some people who don’t, so i will not. But I’ll also try not to over explain, there are other resources for that. If you already know a thing or two about unit testing, you can ignore the rest of this chapter. Or maybe don’t, maybe you’ll learn something new. I dunno.

When You write some code, it’s usually a good habit to test it. After all, how do you know if the code working correctly? You run it, and see if it does what it’s supposed to. The simplest way of doing that is writing some small programs that use the code in question in various ways, and check if the outputs of this code are correct. For example, let’s assume you’ve written a function that calculates the amount of damage an attack does in an RPG game. To test it, You can create some attack scenarios with various items, statistics, and whatever else can influence damage. Then, calculate the results manually beforehand, and write the code that will perform those calculations using that function and verify if they match your calculations. Tests that verify those small pieces of code, like functions or classes, are usually called unit tests. The most basic unit test can be something like this:

#include <assert.h>

#include <DamageCalculation.h>

#include "test_utils.h"

int main() {
 Weapon const weapon = create_test_weapon();
 Enemy const enemy = create_test_enemy();

 assert(calculate_damage(&weapon, &enemy) == 3.14);
}

Although I’d replace that magic number with some very approximate manual calculation, if possible. Tests should not be cryptic - this is, but only because it’s an unspecified example.

Fortunately, the humanity have (mostly) progressed past the need for assert, and invented test harnesses that are (usually) less painful to use. Trust me on that. Even C guys didn’t like rawdogging assert, and jumped on Unity (not that Unity) and other similar “wrappers”. In any case, using a test harness is easier than making one ourselves (or not using one at all), so we are going to use one. I’d prefer if it also were multi-platform and supported both C and C++, and fortunately we have some options in the open-source market for that.

Managing external dependencies with Meson

Before we choose a test harness, however, let’s check how Meson handles external dependencies. Fortunately, documentation got us covered. And it seems that Meson supports a lot of popular libraries in very specific ways, which is very, very nice. However, the dependency() function seems to work only for locally available packages, which would usually mean that we have to manually download and set up the libraries for development each time we set up a new project environment - but there’s a nice catch! Meson also provides Wrap dependency system. I recommend reading the linked documentation, but if you want a TL;DR: this is a package manager. Sort of. You put a valid .wrap file in subprojects/ directory and voila, Meson can download and build the dependency, allowing us to use it in our project - automagically.

After looking at available wraps, and considering the fact that I’d like to use that template for ARM projects, I have decided I’m going with CppUTest. If you want to try another test harness, feel free to do that and experiment - but keep in mind that the setup process may vary a bit, depending on the library needs. CppUTest is relatively simple, which is nice if you’re looking for something without much complexity, but it won’t provide as many features as some other harnessed - like Catch2 or Google Test. I like simple, and it should probably make porting everything to ARM easier, so I’m going with CppUTest.

Let’s create subprojects/ directory and run meson wrap install cpputest (or your preferred test harness). Installation should proceed automatically, and subprojects/cpputest.wrap file should appear. Then, to use it, we have to declare a subproject() in one of our meson.build files. Let’s do that in our tests/meson.build. We’ll also add our tests subdirectories, since we’re already editing that file. Add them below the subproject() to make sure the test harness is available there.

cpputest_project = subproject('cpputest', required: true)
cpputest_dependency = cpputest_project.get_variable('cpputest_dep')

subdir('calc')
subdir('greeter')

Now, we also have to declare tests directory as a subdir() in main meson.build. I’ve put it after apps, because integration tests may require built apps as dependencies to run.

subdir('libs')
subdir('apps')
subdir('tests')

And finally, we can check if this works by deleting builddir, and running meson setup builddir and meson compile -C builddir. You should see some new logs after running setup:

Executing subproject cpputest
cpputest| Project name: cpputest
cpputest| Project version: 4.0
cpputest| C++ compiler for the host machine: ccache c++ (gcc 13.2.0 "c++ (MinGW-W64 x86_64-ucrt-posix-seh, built by Brecht Sanders) 13.2.0")
cpputest| C++ linker for the host machine: c++ ld.bfd 2.41
cpputest| Build targets in project: 5
cpputest| Subproject cpputest finished.
Build targets in project: 5
project_template 0.1
Subprojects
cpputest: YES

That tells us Meson found the wrap and should’ve downloaded it. Compilation should also take significantly longer, because our test harness must be compiled for the first time. If that’s the case, our first step is done. Now, we have to actually write a test. Create test.cpp file in tests/calc/ and put this there:

#include <CppUTest/CommandLineTestRunner.h>
#include <CppUTest/TestHarness.h>

TEST_GROUP(FirstTestGroup){};

TEST(FirstTestGroup, FirstTest) {
 FAIL("Fail me!");
}

int main(int ac, char** av) {
 return CommandLineTestRunner::RunAllTests(ac, av);
}

Now, let’s make it compile. Open tests/calc/meson.build and put it there:

calc_test_exec = executable('calc_test', 'test.cpp', dependencies: cpputest_dependency)
test('calc test', calc_test_exec)

In theory, this should build just fine. However, in practice…

[44/44] Linking target tests/calc/calc_test.exe
FAILED: tests/calc/calc_test.exe
"c++" -o tests/calc/calc_test.exe tests/calc/calc_test.exe.p/test.cpp.obj "-Wl,--allow-shlib-undefined" "-Wl,--start-group" "subprojects/cpputest-4.0/src/CppUTestExt/libCppUTestExt.a" "-Wl,--subsystem,console" "-lkernel32" "-luser32" "-lgdi32" "-lwinspool" "-lshell32" "-lole32" "-loleaut32" "-luuid" "-lcomdlg32" "-ladvapi32" "-Wl,--end-group"
C:/gcc/bin/../lib/gcc/x86_64-w64-mingw32/13.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: tests/calc/calc_test.exe.p/test.cpp.obj: in function `TEST_FirstTestGroup_FirstTest_Test::testBody()':
F:\Projects\C_C++\meson_c_cpp_project_template\builddir/../tests/calc/test.cpp:7:(.text+0x11): undefined reference to `UtestShell::getCurrent()'

In practice, I get a massive linking error. Gee, I wonder why.

After few minutes of investigation, I’ve reached the conclusion: the “official” CppUTest wrap is very bad. And by that, I mean it’s completely broken. If you’ve used a different test harness, and it works (i know for a fact that Catch2 should, last i tried at least…) - congratulations, you can skip the next part! For the rest of you, don’t worry - we’re fix that issue. It’ll just take a bit longer for you, and much longer for me.

Side quest - fixing CppUTest!

Let’s look at it’s build files, as everything is stored in subprojects/cpputest-4.0. First thing that I’ve noticed was spelling error - the author used extinctions instead of extensions (in multiple places, so this was… Intentional?) Then, I’ve noticed something worse.

cpputest_dep = declare_dependency(
 link_with : cpputest_lib,
 version : meson.project_version(),
 include_directories : cpputest_dirs)

if get_option('extinctions')
 cpputest_dep = declare_dependency(
 link_with : cpputest_ext_lib,
 version : meson.project_version(),
 include_directories : cpputest_dirs)
endif

You see this, right? Obviously, a typo, someone forgot to add _ext, but one that completely breaks the wrap. Extensions are enabled by default, by the way. See meson_options.txt in CppUTest directory:

option('extinctions',
 type : 'boolean',
 value : true,
 description : 'Use the CppUTest extension library'
)

So, what happens when we fix that? We can do it pretty easily, just declare a second dependency, we don’t even have to link the extensions because at this point we’re not using them. Rename cpputest_dep for extensions to cpputest_ext_dep and let’s add the second dependency to cpputest.wrap:

if get_option('extinctions')
 cpputest_ext_dep = declare_dependency(
 link_with : cpputest_ext_lib,
 version : meson.project_version(),
 include_directories : cpputest_dirs)
endif

[provide]
cpputest = cpputest_dep
cpputest_ext = cpputest_ext_dep

And after clean compilation, unfortunately, this doesn’t solve my issue. I get multiple undefined references to platform-related functions, because apparently someone forgot to add platform-specific implementations that CppUTest provides as dependencies…

Well, the fix is easy - I just have to make a proper wrap myself. To be continued, after I make it working.

Yeah, I could jump the ship and just use different library, but where’s the fun in that? I’ve already vibe checked that one. Also; there is no tag in the repository for now, I’m gonna make one after making that wrap and finishing up this part in next post(s).

Making C/C++ project template in Meson - part 1

Sat, 27 Jan 2024 00:00:00 +0000

Intro

One of the things that I dislike about C and C++ in particular is lack of standardized build environment, package manager, and all that stuff. And although CMake is de-facto “standard” over here, and even tries to double as a “package manager”, I find it hard to like this tool. It’s not a bad tool, I’ve used it few times, I’ve seen some non-trivial projects made with CMake, but I did not like what I’ve seen.

I’ve also been missing a proper C/C++ project template for a very long time. I was not aware I’m missing one, until i got a hang of the codebase of one of the projects I’ve been working on in my work, and experienced the true power of scripting and automation. I’ve been tinkering with many build systems over the years, but I’ve never found one that really “clicked” for me. And since I still cannot lift the curse of C/C++ from my life, it seems the only alternative I have is to pick something and force myself through. And I’m taking You for a ride with me.

So we’re going with Meson Build system, which was recommended to me by a colleague few years ago. Since then I’ve read the docs and also experimented with it a bit, but I haven’t really made a proper project based on it yet, as of writing this sentence. This is about to change.

I will try to make this series of blog posts as “followable” as possible, so You should be able to reproduce most of my work by yourself, and understand the choices I make. Remember that the best tool for the job is sometimes the one you make yourself, not the one you blindly copy&paste without deeper understanding. ~~But please, don’t reinvent the wheel if that’s not necessary.~~ ~~Use the right tools for the job, and whatnot.~~

Goals

Our main goal here is to make a generic, compiler/platform-independent, easy-to-expand project template in Meson for small-to-medium size C/C++ projects. I also want it to have some specific features that I believe are “must have” in any self-respecting project. In the end, we will use that template to write some kind of application and maybe play with the project’s structure afterwards, until I’m satisfied with the results. After that, I have a plan to use this template as a basis for ARM Cortex-M project template, but we’ll cross that bridge when we’ll get to it.

The first thing that I’m interested in is unit and integration test support. I don’t want to make assumptions about test harnesses though, so we’ll just have to make sure that the project’s structure is testable and Meson can build the tests. Unit tests have built-in support in Meson, so let’s go with that. There’s also some support for code coverage, and we will surely explore that too. Integration tests are much more project-dependent, so the only requirements is that Meson should recognize them as targets, but running them might be done via external script or program (executed by Meson). Meson is our build system, not necessarily the runner, nor the harness, so let’s not expect it to be able to handle generic integration tests by itself. Some external tooling/scripting may still be required (especially when we’ll get to the embedded part…), but everything should eventually be held together by Meson.

The second thing that I wanna see there is support for C/C++ LSP. My primary choice there is clangd, so we’ll have to generate compile_commands.json using Meson. Fortunately, it does that automatically.

The third thing is support for code checks and code formatting. External tools can be plugged to Meson via custom build targets, and we will configure at least one code checker and autoformatter like that. Our project should provide commands to perform project-wide code and formatting check (which is very useful in git prehooks).

And the last, but not the least - documentation. I want to be able to generate docs for my code via Meson. Probably using Doxygen, as it’s relatively easy to set up. This also includes coverage reports, and we will most likely use lcov/gcov for that.

Did I mention that I’ve spent most of last year writing Rust code? Did you know that Cargo provides most of the stuff that I’ve described here out-of-the-box? Now you do.

Preparations

Before we start, let’s check our prerequisites, because there’s gonna be some.

Meson - on Linux, should be in your package manager. If it’s not, i assume you know what to do. On Windows, i recommend grabbing an installer from Github. Just make sure it’s in your PATH variable - check if opening a terminal and running meson --version returns expected version string. I’m currently running 1.3.1. Technically Meson is available via winget, but their repo currently provides outdated version, so i recommend installing it manually on Windows.
Ninja - I’ll describe it’s purpose in a bit. It should be bundled with Meson on Windows, and should be installed automatically on Linux when installing Meson via package manager. Should be in your PATH too. In any case, verify if it’s installed correctly by running ninja --version. I’m currently running 1.11.1. You can also install it manually if needed by downloading a zip from Github and extracting it into a directory that’s in PATH variable.
C/C++ toolchain - TL;DR install latest GCC. In theory, we don’t have to choose any specific toolchain, because we’re using Meson and Ninja which support the popular ones. However, I’ve mentioned that I want to have coverage report generation, and the only coverage tools that I’m familiar with are lcov/gcov, and this pretty much forces me to use GCC. If you don’t care about code coverage reports, pick your treat. If you somehow got here as a complete newbie and you don’t have any C/C++ toolchain installed, either grab GCC from your package manager if you’re running Linux (look for package called build-essentials or base-devel or something like that), or - if you’re running Windows - download latest WinLibs package and add it’s bin subdirectory to PATH. Or install Microsoft’s Visual C++ toolchain via Visual Studio Build Tools (you may need to do that even after installing WinLibs, if you want to use clang, but I recommend not using MSVC for pure C unless you don’t value your sanity).
LLVM tools - mostly clangd, but we’re also gonna use clang-tidy at some point. If you’re running Linux, again - find those tools in your package manager. If you’re running Windows and WinLibs, you already got them. If you don’t use WinLibs, install latest LLVM release (winget install llvm, should be “recent enough”, or get it from Github), it should contain everything we’ll need. Again, if manually installed - make sure it’s in PATH by running clangd --version. I’m currently using 17.0.5.
Doxygen - we’re not gonna be using it any time soon, but let’s make sure it’s available. Linux - install from repo, Windows - it’s already in WinLibs, if not using WinLibs (or you want the latest version), install manually - winget install doxygen or grab installer from here. Run doxygen --version to verify, I’m using 1.10.0. Note - Winget may not add it to PATH, it’s installed in Program Files/doxygen by default. Add bin subdirectory to PATH manually if that’s the case.

Oh, and there’s a small issue of lcov not having official Windows support. There are some unofficial releases, but we’re not gonna use them - instead, we’re gonna lock the coverage report generation feature to Linux and GCC only. I don’t want to completely lock this template to GCC because of that feature, so we will have to separate this part appropriately.

We can always use containers to get this running with all features on Windows.

Also; i assume that we’re gonna be using Ninja as our building “back-end”, but it doesn’t really matter because Meson should handle whatever “back-end” you’d like to use. Just make sure to check the “whatever “back-end” you’d like to use” output when I’m talking about Ninja output. This is an “I assume you know what you’re doing” warning.

Hello, world!

Assuming that everything is installed, we are ready to get going. I’ve made a repository for this project, I’ll try to make a tag after each part so You can easily follow.

Let’s start by initializing a new Meson project in an empty directory. The arguments don’t really matter here - we’re gonna be building our meson.build from scratch anyway, but right now we want to verify if our environment is set up correctly, so let’s go with those for now:

meson init --name project_template --language cpp --type executable

This should generate two files - meson.build and project_template.cpp. meson.build should contain project definition with some reasonable defaults, executable, and a test:

project('project_template', 'cpp',
 version : '0.1',
 default_options : ['warning_level=3',
 'cpp_std=c++14'])

exe = executable('project_template', 'project_template.cpp',
 install : true)

test('basic', exe)

Source file should contain a very simple test program:

#include <iostream>

#define PROJECT_NAME "project_template"

int main(int argc, char **argv) {
 if(argc != 1) {
 std::cout << argv[0] << "takes no arguments.\n";
 return 1;
 }
 std::cout << "This is project " << PROJECT_NAME << ".\n";
 return 0;
}

Let’s try building it. But first, we need to generate actual build scripts via Meson, as it’s (similarly to CMake) a meta-build system. Which means that it’s only a generator for an actual build system (like Ninja, or GNU Make) that performs the heavy lifting. If you’re not used to this approach, it may make no sense at first - why use two tools instead of one, right? Unfortunately, building stuff is not easy in C/C++, especially when you’re working on a big project with multiple dependencies, that must be supported on many different platforms. As far as I know, the earliest successful application of this approach is GNU Autotools project with it’s configure-make-make install routine. Apparently, it’s good to have an additional layer of abstraction over the target build system. I have already mentioned some reasons for that (multi-platform, yadda yadda yadda), also - meta-build systems are usually easier to use and have more features. Anyway, to generate a build directory with all the stuff required for compilation, run this from project’s directory

meson setup builddir

The argument builddir is the name of our build directory. If you’ve set up your compiler and Ninja correctly, you should get some info about compiler executables and your environment, and builddir directory should appear with some Meson and Ninja files.

PS> meson setup builddir
The Meson build system
Version: 1.3.1
Source dir: F:\Projects\C_C++\meson_c_cpp_project_template
Build dir: F:\Projects\C_C++\meson_c_cpp_project_template\builddir
Build type: native build
Project name: project_template
Project version: 0.1
C++ compiler for the host machine: ccache c++ (gcc 13.2.0 "c++ (MinGW-W64 x86_64-ucrt-posix-seh, built by Brecht Sanders) 13.2.0")
C++ linker for the host machine: c++ ld.bfd 2.41
Host machine cpu family: x86_64
Host machine cpu: x86_64
Build targets in project: 1
Found ninja-1.11.1.git.kitware.jobserver-1 at C:\gcc\bin\ninja.EXE

Now, we can tell Meson to compile the project:

meson compile -C builddir

We could also use ninja directly, but I’ll stick to using Meson whenever possible instead, for compatibility reasons. That’s actually a core part of this template - we want Meson to do stuff for us, we don’t want to care what happens under the hood.

We should get some output from Ninja.

PS> meson compile -C builddir
ninja: Entering directory `F:/Projects/C_C++/meson_c_cpp_project_template/builddir'
[2/2] Linking target project_template.exe

And an executable

PS .> cd .\builddir\
PS .\builddir> .\project_template.exe
This is project project_template.
PS .\builddir> .\project_template.exe hello world
F:\Projects\C_C++\meson_c_cpp_project_template\builddir\project_template.exe takes no arguments.

And also a test!

meson test

Oh, if you haven’t noticed yet - the -C argument changes the working directory of Meson, so it’s necessary only when you’re outside of it. I assume that we cd’d into it in previous step, so we don’t need it now. I’ll also stop adding it to future example invocations, just remember that it exists and that Meson’s working directory must be the one that was generated via meson setup for most commands.

Let’s look at the output:

PS .\builddir>meson test
ninja: Entering directory `F:\Projects\C_C++\meson_c_cpp_project_template\builddir'
ninja: no work to do.
1/1 basic OK 0.01s
Ok: 1
Expected Fail: 0
Fail: 0
Unexpected Pass: 0
Skipped: 0
Timeout: 0
Full log written to F:\Projects\C_C++\meson_c_cpp_project_template\builddir\meson-logs\testlog.txt

Yeah, that checks out, we have one “test” and it does nothing and returns 0, so it passes. If you have encountered any issues until now, take a break and investigate. Check if everything is where it’s supposed to be. You can also run meson commands with --help flag to see what options you have available, just as an exercise. Continue, when you’re ready.

Structuring our project

Now it’s time to think about the structure of our project. We must be aware of the fact that there’s no one “best” project structure that would suit every kind of project. However, due to the fact that I’d rather not work on a completely abstract project, we’ll have to make some assumptions about it’s structure, at least for now.

Let’s gather what we already assumed about our project. We assumed that it’s supposed to be testable. Unit tests are the bare minimum that we want to support. We also assumed that we’d like our code to be checked by external tools, but we can ignore that, because those tools should work no matter how we structure our project. It’s just the matter of invoking them properly. Same thing with Doxygen. As for clangd, Meson provides compile_commands.json, therefore it will work as long as the structure is valid.

Considering the usual good design practices, we’ll probably want to split our codebase into modules according to their responsibility. No, not those modules, let’s ignore the existence of C++20 for now. We can store each module in a separate directory, with it’s own meson.build file that would define how it’s built. Modules can be built as static libraries and linked to executables. That way, it’s trivial to test them - we can just link the same binary that’s used with our program executable to the test executable, and validate it’s behavior.

This is how it’s done in some projects that I’ve been working on in my current job, and if it’s good enough choice for space-grade projects, it’s sure as hell good enough choice for me. But really, I’ve seen this approach in action and it should work well.

Let’s also assume that our project can contain more than one executable, because it should be simple to setup for that. We’re gonna put them in apps directory. Modules will go to lib, and tests into tests.

Let’s create some dummy libs, tests for them, and an application to tie it together. Also; let’s add meson.build to each directory, empty - for now. This is how it might look like:

meson_c_cpp_project_template
â”‚ meson.build
â”‚
â”œâ”€â”€â”€apps
â”‚ â””â”€â”€â”€hello_world
â”‚ hello_world.cpp
â”‚ meson.build
â”‚
â”œâ”€â”€â”€lib
â”‚ â”œâ”€â”€â”€calc
â”‚ â”‚ calc.cpp
â”‚ â”‚ calc.hpp
â”‚ â”‚ meson.build
â”‚ â”‚
â”‚ â””â”€â”€â”€greeter
â”‚ greeter.cpp
â”‚ greeter.hpp
â”‚ meson.build
â”‚
â””â”€â”€â”€tests
â”œâ”€â”€â”€calc
â”‚ calc_test.cpp
â”‚ meson.build
â”‚
â””â”€â”€â”€greeter
greeter_test.cpp
meson.build

Adding libraries

Let’s ignore the fact that we don’t have a test harness yet, and focus on making the libraries build. But first, we have to put some code into them:

calc.hpp:

#pragma once

double celsius_to_fahrenheit(double celsius);
double fahrenheit_to_celsius(double fahrenheit);

calc.cpp:

#include "calc.hpp"

double celsius_to_fahrenheit(double celsius) {
 return (celsius * (9.0 / 5.0)) + 32.0;
}

double fahrenheit_to_celsius(double fahrenheit) {
 return (fahrenheit - 32.0) * (5.0 / 9.0);
}

greeter.hpp:

#pragma once
#include <string>

std::string greet(std::string const& name);

greeter.cpp:

#include "greeter.hpp"

std::string greet(std::string const& name) {
 return std::string("Hello ") + name;
}

And now, let’s fill meson.build files. In order to build a library, we have to use library() function. Duh.

calc = library('calc', 'calc.cpp')

The first argument is name of the library, followed by 0 or more sources in next arguments. There are some options that we can set here using keyword arguments, but we’re gonna ignore them for now because we don’t need any. We also save the result of this function to calc variable, which we’re gonna use later. We need to do the same thing for the other module, and the next step is modifying our top-level meson.build to recognize this dependency. Let’s remove the executable() and test() calls generated by Meson, and call subdir instead, to include lib directory.

project('project_template', 'cpp',
 version : '0.1',
 default_options : ['warning_level=3',
 'cpp_std=c++14'])

subdir('lib')

Then, we’re gonna create meson.build in lib, that will include all the modules we currently have.

subdir('calc')
subdir('greeter')

And now, we should be able to build them. I recommend removing builddir directory after every big change, although I’m pretty sure that there is a way to do it in more elegant fashion (re-setup?), and re-generating the build files. Oh, yeah, it’s meson setup --reconfigure builddir. There you go. And I got instantly reminded why I’m not using it - it doesn’t really reconfigure everything, so it’s better to stick to the ol’ reliable rm builddir technique, at least until our project’s structure stabilizes a bit.

PS> meson setup builddir
The Meson build system
Version: 1.3.1
Source dir: F:\Projects\C_C++\meson_c_cpp_project_template
Build dir: F:\Projects\C_C++\meson_c_cpp_project_template\builddir
Build type: native build
Project name: project_template
Project version: 0.1
C++ compiler for the host machine: ccache c++ (gcc 13.2.0 "c++ (MinGW-W64 x86_64-ucrt-posix-seh, built by Brecht
Sanders) 13.2.0")
C++ linker for the host machine: c++ ld.bfd 2.41
Host machine cpu family: x86_64
Host machine cpu: x86_64
Build targets in project: 2
Found ninja-1.11.1.git.kitware.jobserver-1 at C:\gcc\bin\ninja.EXE
PS> meson compile -C builddir
INFO: autodetecting backend as ninja
INFO: calculating backend command to run: C:\gcc\bin\ninja.EXE -C F:/Projects/C_C++/meson_c_cpp_project_template/
builddir
ninja: Entering directory `F:/Projects/C_C++/meson_c_cpp_project_template/builddir'
[4/4] Linking target lib/greeter/libgreeter.dll

And it seems that it works. It detected 2 build targets in project, exactly what we wanted to see. Let’s look at builddir to confirm it.

meson_c_cpp_project_template\builddir\lib
â”œâ”€â”€â”€calc
â”‚ â”‚ libcalc.dll
â”‚ â”‚ libcalc.dll.a
â”‚ â”‚
â”‚ â””â”€â”€â”€libcalc.dll.p
â”‚ calc.cpp.obj
â”‚
â””â”€â”€â”€greeter
â”‚ libgreeter.dll
â”‚ libgreeter.dll.a
â”‚
â””â”€â”€â”€libgreeter.dll.p
greeter.cpp.obj

Okay, but we got DLLs (or .so’s, if you’re running Linux). We don’t want dynamic libs, we want static libs. Fortunately, there’s an easy fix, straight from the docs - library() respects the default_library project option, so we just have to change it in our project’s settings before building the libs.

project('project_template', 'cpp',
 version : '0.1',
 default_options : {
 'warning_level': '3',
 'cpp_std': 'c++14',
 'c_std': 'c11',
 'default_library': 'static',
 }
)

I took an opportunity to refactor that ugly array list into a proper dict, and set the default C standard to C11. You can find the list of available default options here, if you’re interested. Let’s rebuild the project (again, --reconfigure may not detect this change, so rm builddir), and as we can see - we’ve got static libs now!

meson_c_cpp_project_template\builddir\lib
â”œâ”€â”€â”€calc
â”‚ â”‚ libcalc.a
â”‚ â”‚
â”‚ â””â”€â”€â”€libcalc.a.p
â”‚ calc.cpp.obj
â”‚
â””â”€â”€â”€greeter
â”‚ libgreeter.a
â”‚
â””â”€â”€â”€libgreeter.a.p
greeter.cpp.obj

Adding an executable

Next, let’s use those libs with our executable. Example hello_world.cpp may look like this:

#include <calc.hpp>
#include <greeter.hpp>
#include <iostream>

int main() {
 std::cout << greet("random developer") << std::endl;
 std::cout << "12.5*F == " << fahrenheit_to_celsius(12.5) << "*C" << std::endl;
 std::cout << "12.5*C == " << celsius_to_fahrenheit(12.5) << "*F" << std::endl;
}

Of course, the LSP will not be able to detect those headers properly yet, so ignore all the errors. Let the toolchain speak the truth. This is the content of hello_world’s directory meson.build

hello_world = executable(
 'hello_world',
 'hello_world.cpp',
 link_with: [calc, greeter]
)

Notice that we’ve used previously defined calc and greeter variables. In order to do that, lib subdirectory must be evaluated before apps, and that’s why subdir('apps') call must be put below subdir('lib') in our main meson.build file. subdir basically evaluates the meson.build from the directory provided via the argument, and retains the environment after that. Pretty useful, but we have to be careful not to misuse that feature, otherwise our Meson scripts may become very fragile and complicated to maintain.

subdir('lib')
subdir('apps')

We also have to create meson.build for apps directory.

subdir('hello_world')

And yes, we probably could just subdir('apps/hello_world'), but I wanna keep everything as local as possible. You may add meson.build to every directory of the project now, because we’re gonna need them.

meson setup builddir tells me that there are 3 targets now. That’s what we want. But after running meson compile -C builddir, it seems that we have a problem.

../apps/hello_world/hello_world.cpp:1:10: fatal error: calc.hpp: No such file or directory
1 | #include <calc.hpp>
| ^~~~~~~~~~
compilation terminated.

Yeah, that checks out. We’ve linked the libs, but we never told executable() where are their include files. To do that, we need to use include_directories argument of executable(), which expects a list of objects created using include_directories() function. And include_directories() accepts string with relative paths, so we can just add them to the meson.build of our libraries and use them just like the objects returned from library().

calc_lib = library('calc', 'calc.cpp')
calc_includes = include_directories('.')

I’ve also added the suffix to lib. We need to do the same for greeter, and fix our hello_world.

hello_world = executable(
 'hello_world',
 'hello_world.cpp',
 link_with: [calc_lib, greeter_lib],
 include_directories: [calc_includes, greeter_includes],
)

And voila, we should have a green build now! Let’s see if the executable actually works.

PS> .\builddir\apps\hello_world\hello_world.exe
Hello random developer
12.5*F == -10.8333*C
12.5*C == 54.5*F

Yeah, the output checks out. And that is where I’d like to finish this part, because it’s already long enough, but there’s just one tiny thing that annoys me…

Feeding the LSP

I assume that you’re using an editor/IDE with LSP support, and will be using clangd. If that’s not the case, i strongly recommend getting one. I personally use neovim, but I cannot really recommend it to everyone, so Visual Studio Code with clangd plugin should be more than enough. You may want to look at my older guide to see how to setup that combination. You can find it here. Meson generates compile_commands.json by default, automatically on every build (i think). However, clangd will not be aware of that, because this file will be stored in builddir instead of our project’s root directory, so we have two options:

Tell clangd that compile_commands.json is in builddir/
Copy/link compile_commands.json into root directory of our project

Fortunately, we can go with the first solution, because clangd supports configuration, and has an option for telling it where the compilation database is. Let’s create .clangd file in the root of our project, and put it there:

CompileFlags:
 CompilationDatabase: builddir/

You can also add some additional config for the LSP.

aaaand… It looks like it’s working for me, so that’s it, we’re done here. I’ve made a tag and release on this project’s repository if you want to go the easy route and just git clone everything we’ve done here. See you next time.

When should you choose C++ as your starting language?

Tue, 14 Nov 2023 00:00:00 +0000

When I’m lurking through the internet, I often see posts asking about “what language should I pick as a beginner???”. As someone who struggled a lot with this choice, and ultimately picked C++ (for reasons that made no real sense—but, of course, I didn’t know that back then), I think I can say a few words about this specific choice and how it can affect the learning process of an individual.

It’s a trap!

TL;DR of this post - it’s usually not a good idea to pick C++ as your starting language. After working with C++ for a long time, and tasting many different programming languages, I feel like C++ is a convoluted mess taped together using subpar-quality duct tape, somehow still holding on, maybe even going in a relatively good direction with recent changes, but certainly not good enough for a beginner to learn the programming principles on at its current state.

Why would you even want to do that?

Excellent question! In most cases, I hear these specific arguments, trying very hard to justify picking C++ as a starter:

It’s very fast!—very common misconception. Languages are not inherently fast nor slow. Sure, some languages can be parsed or interpreted, faster than others, but it does not imply that a program written in language A will always be faster than a program written in language B, or vice versa. Surely, that can be a thing - we absolutely can write a program in C++ that will be faster than an equivalent program in Python or Java, but it works both ways! A good practice is asking yourself “why?” and “when?”. Why a program written in language A may be faster/slower than an equivalent program written in language B? And at what scale does it become meaningful to performance? But that’s not something that a newbie should care about. Generally speaking, you should NOT care about “language performance” as a beginner, as it’s one of the last things you’ll have to worry about when learning programming. In the beginning, the real performance will lie mostly in data structures and algorithms used in your code, not the language choice.
Learning C++ teaches you low-level concepts, like pointers and manual memory management! It can, but there’s a biiiiiig but. You don’t have to know these low-level concepts to write software and learn programming. Every day thousands of programmers write perfectly fine and working code without even knowing what a pointer is, or how to manually manage the memory. It’s not something that you absolutely must know in order to write working code. It’s not something that may ever be useful for you. I certainly agree with people saying that knowing how pointers and manual memory management work can be useful in many situations, but for a beginner—it’s certainly not a must-know, as even in C++ you usually don’t want to manually manage raw pointers and memory allocations.
Game programming is done in C++—usually paired with the “much performance” argument—is also invalid, as there are plenty of very popular game engines that provide a much more newbie-friendly approach to game development. Check out Godot as one of the best examples, since it has both its own toolset that uses its own GDScript language, but also can be used with other languages and tools if you prefer that approach or you already know them and don’t want to learn new stuff. I know there are plenty of people doing gamedev in C++, and they do have a point in that—being relatively close to the hardware it’s not the worst choice of a language, but I still strongly believe that it’s an overkill if you want to learn “programming” and “game development”, as it carries a heavy bag of “having to learn C++” before learning the stuff you really wanna learn. Don’t forget that, if you’re thinking that it’ll be a breeze and something you’ll do “on the side”, you may lack some crucial knowledge. Of course, if you wanna do it “raw” and really refuse to use existing game engines, you absolutely can do it in any language that has some kind of multimedia API… so, basically, the most available and popular languages for sane people (brainfuck btfo). If you don’t believe me, this game was written in Pascal, from scratch, in 2017. And it runs great!
I know some C, so C++ is just an extension, so I’ll learn it. It is not, and thinking that way will force you onto a path from which it’s very hard to return, as I have witnessed multiple times with my own eyes. Beware of thinking about C++ as an addition to C, as it’s just as foolish as thinking about a dragon being an addition to his precious mountain of gold. You WILL get burned. You WILL witness horrors beyond your comprehension. That last part is also true when talking about C++ in general. Compare the following articles with their C counterparts (the link is at the bottom of the reference page) and rethink your life choices.

That list may become longer with time.

So, what are the alternatives?

Another excellent question! Loving the audience today.

Obviously, I will have to answer according to my own opinions and beliefs, however, I will also try to explain why I think like I think I think. I think that will make it easier to, at least, continue your research. Remember - t̴h̴e̶ ̷r̷e̵s̷e̴a̸r̸c̴h̸ ̸n̷e̷v̸e̷r̷ ̷e̵n̵d̴s̸.

Python: Some people will tell you that Python sucks. Some people will tell you that Python is not a good language for beginners. Some people are wrong and cannot comprehend the simplicity and beauty of whitespace-defined syntax. I, however, will tell you that I love Python because it rarely disappoints me. I need to write a simple tool that does some stuff automatically, and I want it to work everywhere? Python is my guy. Its long support, big community, package manager, and existing tools are making it very hard for me not to recommend this language as a starter. The syntax is sure very specific, some might even say exotic, but after working with it for a while it annoys me only sometimes when I have to move a block of code to a different place and manually fix every indent. And it even has type hints and easy-to-use tools that’ll make sure you adhere to typing rules, which some people may find very useful. For a beginner though, you’ll most likely want to focus on the vast PyPI repository to write some useful stuff for yourself. Its documentation is also relatively decent and even has a tutorial to get you started, but if you’re completely new to the IT and/or programming world I’d recommend looking for a first-time tutorial more suited to your level of knowledge, as the “official” one is not very newbie-oriented.
JavaScript (or maybe even TypeScript?): It’s a very simple gateway to the vast world of web applications, but not only, as you can plug JavaScript almost anywhere nowadays… I’m personally not a fan of this language, but from what I’ve seen - people who start from JavaScript have a certain “freedom” of being able to use this language anywhere they want, which is nice if you don’t know what you exactly want to do yet, but usually comes with additional baggage of tools for creating user interfaces - HTML and CSS being the usual ones, but most likely also paired with some fancy frameworks and other funky stuff that—according to many web developers—are needed to write functional pages, for which more elegant technologies from simpler times may be more than enough). I’m getting out of topic though, I endorse JavaScript as a starter way more than I endorse C++, but be aware of many pitfalls this language comes with. Which can be said about basically any language, but the Big Dev won’t tell you about that!
Kotlin: If you want to learn Java for whatever god-forsaken reason you may have, don’t learn Java—learn Kotlin instead! If you wanna make Android apps and you’d rather use official tools than some webdev mumbo-jumbo, Kotlin is your guy! Its Java legacy makes it a reasonably good starting language, as you have both a nice, modern language with nice, modern syntax and useful features that are simply a joy to work with, and a vast repository of libraries, some written long before Kotlin even existed. I haven’t worked much with this language, but I enjoyed almost every moment of it.
C: Yeah, let’s put it on the list, why not. You wanna have a very close relationship with your hardware? Here’s your language. Don’t tell me I didn’t warn you though. In all fairness, C is certainly lacking—real generics, for example, but that doesn’t stop people from using it even right now. As a language to get the hang of programming and “how does stuff work under the hood” it’s a great choice, as a language to write working applications without having to create a whole universe beforehand… not so much, unless you work on something that does not require an entire universe to run, but I will get to it soon. I promise.

This list is by no means comprehensive. There are a lot of languages that are deemed “reasonable for beginners” that are not on this list, just because I either never worked with them, or worked so long ago that I cannot assume that my knowledge is still applicable to its current state (e.g. C#, which I’d like to say something about, but I really can’t).

Notice that my focus here is to recommend a language that will be relatively easy to learn and easy to use, with a big community and lots of resources to learn from—C++ is lacking in many of these aspects, which makes learning modern C++ the “correct” way surprisingly hard. And, to be fair, it makes doing anything in C++ unreasonably convoluted and painful. Ever tried to set up a C++ project from scratch with automatic unit and integration tests? I did. And failed. Not because it’s impossible, but because I simply rather choose a language that supports that stuff out-of-the-box (unit testing, at least). And I’ve seen projects that do have this stuff in C++, and I… uhh… I’d rather avoid having to jump through the hoops of CMake and similar tools in order to have stuff that’s relatively easy to do in other languages usually without having to learn how to use 3rd party tools. And there’s C, as a direct C++ alternative, and I’ll explain why it’s here—because it’s certainly not for the same reason as other languages on this list.

When it actually makes sense to go C++ head-first

As with any rules, there are exceptions, and this includes my “No C++ for beginners allowed” rule.

The most obvious exception for me is targeting embedded programming as the thing you want to do. And it’s also the reason why C is on my list - because C is much simpler than C++, it might be a better idea to start with that instead. But if you wanna use Arduino, like most beginners nowadays, sticking only to C makes little sense as there are powerful and relatively easy-to-use features of C++ that you already have access to (and might have to use, depending on the libraries you work with). Sure, you may not have the standard library and its fancy features (and I’d argue that it’s a good thing), but you still have the power of templates, lambda expressions, constexpr, and many more cool features. So yeah, if you wanna tinker with hardware then learning C++ might be unavoidable for you.

But only might—if you ignore the existence of Arduino framework—not the hardware, the hardware is its own thing and it’s just a “brand” name - you may just stick to pure C and write the code from scratch directly for the microcontroller you have, or using vendor-provided tools (like STM32CubeIDE for ST microcontrollers). However, this is substantially harder as it usually requires you to actually read the documentation of hardware you’re working on. It’s really not something for faint-hearted developers that would rather stick to copying and pasting random pieces of code from the internet or ChatGPT, with the hope that it’ll solve all their issues.

Another obvious scenario is “you have to learn it because of school/university/work/side-project I’m doing with some C++ developers”. Or you may just be very, very, very fiercely and weirdly focused on learning C++. For that, I have no cure. I just have a tip: try to stick to modern sources and don’t get sucked by outdated, pre-C++11 tutorials and books that have questionable reputation. There are many. There are curated lists of sources from which you can start your journey.

And I can’t really think about any other scenarios where C++ would really make sense. Some time ago I’d say that OS-dev is a good niche for that, but you can do OS-dev in C as well, and there’s Rust for the cool kids that sit on the back of the bus—much saner choice than C++ if you ask me. Borrow checker may go hard, but it’s there and will never leave you. Do you want the same commitment from C or C++? Configure it yourself. And it won’t be half as good as what Rust does for your code. Graphics programming and game development? Yea, sure, maybe. Maybe if you are new in this world, still with hopes for a better tomorrow, still young and innocent, it will sound like a good choice - and you may even walk through this path relatively unscathed. But then again, you may not. I’d still look at alternatives first, just because lots of people do it in C++, doesn’t mean that you have to do it in C++ too.

I may sound like a big C++ hater, but to be honest, it earned it. I’ve spent years with that language, and half of this time I’ve spent fighting with it. Do I regret it? I regret not jumping this ship earlier, but I’d say that lots of things I’ve learned via C++ I’ve found useful at some point. Could I learn those things while focusing on different languages? Some of them, yes, but not all of them. Would I go a different path if I could start my journey again? Fuck yes, I’d jump straight to Python or webdev, but considering my areas of interests and expertise I’d end up with C++ at some point anyway.

I’m also not saying that C++ should never be learned or used. It’s got its things going, and with a certain amount of experience in general programming and with setting up software projects, it’s possible to work with it. But you need that experience and knowledge first, and head-slamming into a C++ wall may not help you get that knowledge in comparison with other available methods.

The future of this blog

Sat, 28 Oct 2023 00:00:00 +0000

The flashback

First, quick recap of my last year (or two), to provide some context for this post. This will also be kind-of-a-vent, completely out of topic of this blog, so if you don’t care about the drama in my life feel free to scroll right to the end for conclusions.

I’m currently 25 years old. I’ve had a pretty good relationship with my parents, although their relationship with each other was pretty rocky, especially in the last few years.

Last year I’ve learned that my Mom had cancer. Hidden in gallbladder.

When I learned that, I’ve been working at ST, doing my Master’s degree, and also working on a telemedicine-related side-project at the same time. The amount of work and stress, along with the situation at my family’s house, made me quit my work at ST so I could focus on finishing my degree and helping Mom and my brother. We had a lot of stuff to do.

At first, everything seemed relatively okay, as the gallbladder was completely removed, and after the first chemo everything seemed clean and we had hope that it was something she can live with for, at least, few more years.

Unfortunately, the tumors returned few months later - this time on the liver, and they grew very fast and couldn’t be removed surgically - at least not completely.

Mom died 16th January, 2023. She knew it was coming, she prepared us for it best she could, and I will be forever thankful for everything she has done for us.

Since then, I’ve been trying to move on. I’ve been working at N7 for some time and it’s been a good distraction so far - and I generally like this job and the people there, so I’m not planning to quit any time soon. I’ve dropped my science-club related duties, and focused on work and staying mostly sane, most of the time at least.

For the last few weeks, I had a popup in my mind, constantly appearing and telling me to do some stuff and write about it. I’ve also had an urge to do some makeovers in my life and become a productive member of society. Thanks to the privilege I have, as someone who’s being paid for creating new spells (writing code) using my elaborate sorcerer’s setup (split ortholinear keyboard with RGB strip), and who doesn’t have to wonder if he’ll have a place to live next month, I’ve decided that one of the productive things I can do is writing blog posts to force myself to learn and experience new things, because otherwise I don’t have enough motivation to even pick up the stick that I’m going to be poking the stuff I want to experience with.

Making stuff have helped others cope with loss of the loved ones, so i guess it’s worth trying.

Conclusions?

I’m planning to be much more active here.

I’ve had lots of ideas for projects and blog posts recently, but due to the fact that my time is finite and I only have few hours per day to do this stuff (in contrast to more than dozen I had, back when I was broke and had no work - hefty price to pay), I need to prioritize and choose something to pour my attention into, one at a time.

From the top of my (unordered) stack, I can list some of those:

HTMX - I’ve bought a mug, so might as well learn it. Since I have almost no experience in web or frontend development, it should be a fun ride. The plan is to make a dashboard for my RaspberryPi to control the LED strip from a browser.
C/C++ project setup in Meson - i, uhh, probably will have to go back to C/C++ at some point so I’d rather be ready. I poked Meson with a stick already, and I’ve even made some working stuff, but it’s not quite finished yet. When I will finish it, I will describe it here.
Embedded-related stuff - Most of my time I’ve spent programming in the last few years were spent on embedded programming. So I guess I can pretend I’m qualified to talk about it now. I think that a series on implementing (and using) abstraction over microcontroller’s hardware could be nice, considering I’ve been developing BSP/HAL libraries for most part of the last year.

But first, i’ll probably do a cleanup of my NeoVim config, as it broke recently (again). Maybe i’ll write a post on that too.

I’ve also been considering making some video content, as i personally find this form more appealing, but looking at YouTube and their latest ideas constantly makes me reconsider that. The true reason i haven’t done any video content, of course, is lack of motivation, but we don’t talk about it here, it’s easier to blame The Corporation.

That’s basically it for now. I’m slowly getting up. I need to create and get used to a new daily routine. And everything will be fine, eventually. Hopefully, at some point in the future, this blog will not be a graveyard of few outdated and mediocre-quality posts, but rather an actual source of knowledge for people looking for it.

VSCode CubeMX Project Setup

Thu, 16 Sep 2021 00:00:00 +0000

Up until now, i’ve used STM32CubeIDE with STM32CubeMX as my primary development tools for STM32 microcontrollers. CubeIDE has lots of useful features available out-of-the-box. However, it also suffers from pretty feelable bloat, caused by the fact that it’s heavily modified Eclipse IDE with lots of plugins that pile up pretty quickly. It also lacks some features that i’m used to having in more… “civilized” IDE’s (dark theme which is not hurting the eyes with contrast by default? better Git integration? and probably lots of other, smaller thingies). So, i’ve decided to try something new and the first obvious choice was Visual Studio Code.

The second choice would be CLion, which has pretty decent CubeMX project support out-of-the-box, but it is sometimes a little bit buggy with debugging the projects… or at least it was few months ago, when i tried it. Maybe it got fixed.

So, after few hours of tinkering and lurking around for guides and plugins, i’ve found my setup. Most of the config i’ve taken from two great videos by Embedded Geek (first one is here, second one - here), but i’ve changed some things since i’m not using the exact same VSCode config as he does (which, actually, makes things easier and less painful to configure for me). I also made sure that it should work on both Windows and Linux (and probably MacOS).

If you are very lazy, you can use stm32-for-vscode plugin instead of this guide. It should do most of the work described there automatically. However, it uses default IntelliSense engine, and as i prefer clangd and knowing how to configure the project from scratch (so when something goes wrong, i know how to fix it), i still created this guide. I’ve used it in the past, but after trying it again and comparing with setup made using this guide i found out it doesn’t offer better experience in the long run

Anyway, let’s get to it.

Prerequisites

From hardware side, you need to have an STM32 board, and a debugger that’s compatible with it. For this guide, i’m using Nucleo-G474RE, with STM32G474RET6 microcontroller, and STLink v3 onboard. The exact choice of MCU and debugger doesn’t matter, as this setup will work with any STM32 MCU and every debugger supported by OpenOCD. Also, you’ll need an USB cable to connect the board to your PC. If you’re using ST-Link, update it’s firmware before proceeding. You can do it with this official tool. Free ST account is required to download it.

From software side, these tools are required:

STM32CubeMX - STM32 MCU helper and code generator, simplifies the project generation and MCU initialization to maximum - downloadable for free from official ST site here. Requires free ST account to download. PSA for HAL-haters: you don’t have to use HAL even if you generate the code using CubeMX - read about LL libraries, you can use them too, and they are as close to the metal as it’s sanely possible. I’ll put a quick info about it at the end of this guide, as a bonus.
ARM-GCC toolchain (arm-none-eabi-gcc to be specific, along with all the other GCC tools, and GDB). Linux users should have latest version in the repositories - make sure you also install arm-none-eabi-newlib and arm-none-eabi-binutils if these are separate packages. Windows users can either use their preferred package manager, or download the latest toolchain manually from official ARM site - MacOS and Linux versions are available there too, in case you don’t have one in repositories. Just make sure it’s in your OS PATH variable before proceeding - open terminal and try running arm-none-eabi-gcc --version. If it fails, add the toolchain’s bin directory to PATH.
OpenOCD - we’ll use it to debug and flash the firmware on our microcontroller. Get it from your package manager, or via one of the unofficial distributions listed here. Or you can also build it from source, which is fairly easy. Make sure it’s in PATH too - check if openocd --version prints the info, if not - add the directory with it’s executable to PATH.
Visual Studio Code - obviously. If you are new to this editor, i strongly suggest to check out my previous guide about configuring it for C/C++ projects with CMake. We won’t use CMake here (although it is possible to do that with a little bit of help from stm32-cmake repository), but i will use clangd, as i prefer this language server over the default one, and it’s easier to set-up. So, if you want to stay with default language server, keep in mind that you will have to configure include paths manually, which is painful and i won’t describe it here. If you have no idea what i’m writing about here - just follow this guide and install all the plugins listed below. I strongly recommend to use these plugins for VSCode:
- C/C++ - for general C and C++ language support - required;
- Cortex-Debug - for ARM Cortex-M debugging - required;
- clangd - alternative language server that usually works better than default one, and has easier and more robust config. Also features configurable code auto-formatting. If you don’t know what language server does, or you haven’t played with default one yet - install and use clangd, you can find basic setup instructions here, and i’ll guide you through configuration for this project. If you know what you’re doing - have fun.
- ARM - for ARM assembly language support. Useful when lurking around *.s files - optional;
- LinkerScript - for linker script language support. Useful when lurking around *.ld files - optional;
- And also plugins from my list of recommended C/C++ plugins, which you can find here.
ST-Link (or your preferred debugger) drivers - if you already have STM32CubeIDE or STM32CubeProgrammer installed on your system, you should already have them. If that’s not the case, then either install STM32CubeProgrammer to get all the drivers that you’d need for most STM32-compatible debuggers, or download and install the drivers for your specific debugger and OS manually. Just make sure that your debugger is detected correctly by your operating system before proceeding.

And that should be just about it. If you have everything set-up, can get to work on our project.

Creating a project

Open up STM32CubeMX and make a new project. I’ll assume that you already have basic knowledge of STM32CubeMX usage, so you can create a project for your board. After that, set up your peripherals (i’ll just use onboard LED which is set up by default to test stuff out), make sure the debug is enabled (System Core -> SYS -> Debug), clocks are configured correctly (Clock Configuration tab on the top of the window), and go to Project Manager tab. Fill up the project name and location, and set Toolchain/IDE to Makefile. Save the project after that.

Next, i recommend going to Code Generator tab on the left and selecting all the checkboxes in Generated files section. I do it as force of habit, to have a bit better organized project.

When you’re done with CubeMX config, Generate the code and open the directory with the project in Visual Studio Code.

Setting up the project in VSCode

Building the project

Now, time for the fun part. First, we’ll make sure we can build the project. Open integrated terminal (Ctrl+` to open existing one, or Ctrl+Shift+` to create a new one) and run make. You should see some output - the commands ran by Make, and size of output binary - and a new directory called build should appear. The project directory should look like this now:

If the build failed, read the error message, check if you have all required prerequisites, and look around the internet for support - most of basic issues have been solved by someone already, and solutions are out there.

The build directory should contain a lot of object (*.o) , listing (*.lst), and *.d files, along with few binaries with the same name as your project, and also a *.map file, which contains the memory map of whole program.

The only file here that’s currently interesting for us is the *.elf file - this is the actual compiled binary we’ll flash to MCU using OpenOCD. *.bin and *.hex files are compiled binary files too, created out of the *.elf file, but they are usually used in different scenarios (for example, *.bin file can easily be used by custom bootloaders, as it’s basically raw compiled code, stripped out of unnecessary stuff).

Configuring the language server (clangd)

If you’re able to build project successfully, then it’s time to set up our language server. Clangd requires compile_commands.json file to work properly - this file contains translated output from your build system, that tells Clangd how the project is built. Thanks to that, Clangd can automatically detect the configuration you’re using and you doesn’t have to manually set anything up, unlike in default language server, where you’d have to set the include directories, flags and toolchain manually in JSON file.

There are few ways to generate compile_commands.json. As i’ve described in my previous blog post, CMake generates this file automatically. However, we are not using CMake, so we have to use external tool for that. I’ve know of two, that i managed to get working without issues: Bear, which supposedly works only on Linux, but apparently there is a version available in winget repository, and compiledb which should work anywhere, and is written in Python, so you need it installed to use it. Note: install compiledb IN USER DIRECTORY (python3 -m pip install --user compiledb), because it may not work when installed globally due to permission issues! If you encounter permission-related issues, make sure all of it’s dependencies are installed in user directories too! And make sure it’s in your PATH variable! Same goes for Bear.

The usage of these tools is simple: you run them with your build command, so they take it’s output and convert it to Clangd’s compile_commands.json. In case of Bear, you also have to make clean-build (run make clean before running it), which makes it troublesome to integrate with our setup, so i’m going to use compiledb here.

To build the project and generate the compile_commands.json, run compiledb make -j8.

Tip: You can tell make to build using multiple threads with -j argument. This will significantly decrease compilation time. For example: make -j8 will build the program using 8 threads. Modify this argument according to your CPU capabilities (amount of cores and thread per core).

Compiledb will run make and pass all the arguments to it, -j8 in our case. After running this command from the project root, the compile_commands.json should appear there. Open any *.c or *.h file, or re-open, if you opened them already, and check if the language server works - you should now see no errors related to missing includes or unrecognized function, see the tooltips for code on hover, and have all the other language-server-related features working.

Configuring standard library visibility

After playing with clangd a bit more in this project, i found out that even though project builds successfully when i use standard library, clangd doesn’t recognize it’s headers for some reason. I suppose it’s because the target platform is not the native one, so it doesn’t know where to look for headers. I’ve fixed it by adding include path with stdlib headers to C_INCLUDES variable in Makefile, and clean-building the project.

# C includes
C_INCLUDES = \
-ICore/Inc \
-IDrivers/STM32G4xx_HAL_Driver/Inc \
-IDrivers/STM32G4xx_HAL_Driver/Inc/Legacy \
-IDrivers/CMSIS/Device/ST/STM32G4xx/Include \
-IDrivers/CMSIS/Include \
-I/usr/arm-none-eabi/include

Flashing the binary

We’ll create a rule in Makefile to flash our program to MCU. You could do this as VSCode task too, if you’d like, with slight modifications. Open the Makefile, and somewhere on the bottom, before EOF marker, add a new rule that will run OpenOCD:

flash: all
 openocd -f interface/stlink.cfg -f target/stm32g4x.cfg -c "program $(BUILD_DIR)/$(TARGET).elf verify reset exit"

If you are not using ST-Link, change the interface config file to proper one. Same applies to MCU config file - switch it to one that’s for MCU you’re working on. All the configuration files are stored in OpenOCD installation directory. If you’re on Linux, they should be stored in /usr/share/openocd/ by default. The rest of this command tells OpenOCD what to do - program the specific ELF file to MCU flash memory, verify the memory integrity, reset the MCU and exit. It also requires all rule, so the project will build itself automatically and upload the ELF file to MCU every time you run make flash now. This does NOT disappear during CubeMX project re-generation, so you don’t have to worry about copy&pasting it every time you do this. To delete all the build files (to do a clean build for example), use make clean command.

Configuring VSCode tasks

To get some degree of automation, we’ll create VSCode tasks for building, cleaning and flashing the project. Open command prompt in VSCode (Ctrl+Shift+P), and look for Tasks: Configure Default Build Task command (or Task: Configure Task).

Then, select Create tasks.json file from template, and select Others template.

After that, you should see this tasks.json file, which happened to be added to .vscode directory in project root. If you’re using VCS (Git or Mercurial for example), you should be able to store it in repository without any issues, so don’t wildcard out the entire .vscode directory in your .gitignore (or any other VCS-related ignorefile) to keep the tasks.

Copy this task and modify label and command accordingly, to create a build, clean and flash task. After that, run Tasks: Configure Default Build Task command again, and select your build task. Remember that we’re using Bear/compiledb to generate compilation database for Clangd! Don’t call make for building the project directly, to always have updated and correct database! Now, your tasks.json file should look similar to this (change the thread amount accordingly to your CPU capabilities):

{
 // See https://go.microsoft.com/fwlink/?LinkId=733558
 // for the documentation about the tasks.json format
 "version": "2.0.0",
 "tasks": [
 {
 "label": "Build",
 "type": "shell",
 "command": "compiledb",
 "args": [
 "make",
 "-j8"
 ],
 "problemMatcher": [],
 "group": {
 "kind": "build",
 "isDefault": true
 }
 },
 {
 "label": "Build & Flash",
 "type": "shell",
 "command": "compiledb",
 "args": [
 "make",
 "flash",
 "-j8"
 ]
 },
 {
 "label": "Clean",
 "type": "shell",
 "command": "make",
 "args": [
 "clean"
 ]
 }
 ]
}

Test the tasks out by running Clean task (Tasks: Run Task command -> select Clean task -> don’t scan the output, if you’re asked for that), and using Tasks: Run Build Task shortcut - Ctrl+Shift+B. The program should first clean, then build itself, and you should see make output in integrated console window.

Make a backup of this tasks.json file, because it should be exactly the same for every project configured this way. Also, reminder: instead of putting openocd call in Makefile, you can do it directly in VSCode task if you want - just split the arguments correctly (the command after -c should be a single argument), and set the Build task as Flash task dependency with dependsOn list option. You can also create a flash-only task like that. It’s up to your preference.

Configuring the debugger

Time to configure the debugging task. Go to the debug tab, and click on create a launch.json file. Then, from the list, select Cortex Debug option.

You should now have this file created, with default task for Cortex Debug plugin. We’ll have to change few things to make it working with OpenOCD, our debugger and MCU.

First, we’ll have to change the path to executable, so it points to the elf file with our program, which is in ${workspaceRoot}/build/ directory. Then, we have to change servertype to openocd. It’s not the only debugger that Cortex Debug can use, you can see the whole list by calling autocompletion in empty field. Most of them should work with STM32, but i’ll focus on OpenOCD.

Then, we have to configure some OpenOCD-related stuff. We have to specify device, and put the microcontroller name in it (without Tx suffix, which corresponds to the temperature version of chip: for example, i’m using STM32G474RET6, but in device i have to put only STM32G474RE, because all the temperature versions have exactly the same silicon inside). We also have to specify the configuration files for OpenOCD - the same ones that we specified for flashing. You can do it via configFiles list.

You should also specify preLaunchTask, so VSCode will always build and flash the binary before starting debug session. It’s important, because if you’d forget about it and start debugging session with different program on MCU than on your PC, debugger - at some point - will start showing you complete junk and unreasonable code flow, and if you are not aware of that, you will probably spend a lot of time thinking what’s wrong with your code or MCU.

The launch.json file content should now look like this, except the device and target config files should conform to MCU and debugger you’re using, preLaunchTask should have the same name as your flash task, and executable should point to your ELF file.

{
 // Use IntelliSense to learn about possible attributes.
 // Hover to view descriptions of existing attributes.
 // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
 "version": "0.2.0",
 "configurations": [
 {
 "name": "Cortex Debug",
 "cwd": "${workspaceRoot}",
 "executable": "${workspaceRoot}/build/blog-vscode-cubemx-example.elf",
 "request": "launch",
 "type": "cortex-debug",
 "servertype": "openocd",
 "device": "STM32G474RE",
 "configFiles": [
 "interface/stlink.cfg",
 "target/stm32g4x.cfg"
 ],
 "preLaunchTask": "Build & Flash",
 }
 ]
}

If everything has been configured correctly, you should be able to debug your program now. Add a breakpoint in the first line of main function (click left to line number, red dot should appear) and run the debugging session (F5 by default, or click a button on top of Debug menu).

VSCode should switch to debug perspective, and you should be able to see variables/objects (expand Global section), call stack, and add variables/objects to the watch. You should also be able to step through the program - either via controls on top, or keyboard shortcuts (F5 to let the program run until the next breakpoint, F10 to step over current line, F11 to step into current line, Shift+F11 to step out of current scope, Ctrl+Shift+F5 to restart debugging session, and Shift+F5 to stop debugging session). There’s also a button to reset the MCU (first from left).

Configuring the peripheral view

Finally, to see all the peripherals of our MCU and their registers content, we can download and specify SVD (or rather, to be precise, CMSIS-SVD - System View Description) file. This step is optional, but being able to see the peripherals is useful, so i recommend doing this anyway. For ST MCU’s, you can get SVD files from the MCU page on st.com. For example, in case of my MCU i go here, into CAD Resources tab, and i download STM32G4 System view description (ST account not required).

After that, i unpack it. Inside unpacked archive, there should be a directory wih *.svd files - find one compatible with your MCU (in my case it’s named STM32G474xx.svd) and copy it to project root directory. Now, open launch.json and add new option in debug configuration:

 "svdFile": "${workspaceRoot}/STM32G474xx.svd",

Of course, change the name of the file to correct one. To check if it’s working, start debug session (F5) and look for Cortex Peripherals section in debug view. Open it and check if you can see the peripherals, along with their addresses, registers and register fields content.

And that’s all. If you did all the steps, and verified they’re working, congratulations - you now have an alternative for CubeIDE for STM32 development with CubeMX, that isn’t very painful to configure and has most of the features you’ll need. I’ve heard that Cortex Debug also allows to configure ITM output, but i haven’t tried that yet, so i’ll probably describe it in different blog post in the future.

Now, for the bonus part…

STM32 HAL vs LL libraries, and how to change between them in CubeMX

If you’re using CubeMX for STM32 projects, you are (most probably) inherently using HAL libraries to work with the MCU. HAL is an Hardware Abstraction Layer - set of libraries which abstract operations on the hardware, that are easy to use for someone who doesn’t want to dig through the manual to do even simplest things. Also worth mentioning, that STM32 HAL is mostly the same under all STM32 MCUs, therefore moving project that relies on HAL between different STM32 MCUs in the same family, or even between different families of STM32 MCUs, is fairly easy. However, it do have some issues - mainly, the memory bloat that’s sometimes really noticeable (especially on low-end MCUs with very little memory - some of them even can’t use HAL at all, because it takes more memory than they have for simple project with just the initialization code). It can also be slower than dedicated solutions, due to handling almost everything that can happen every time.

The solution of these issues doesn’t have to be throwing away CubeMX, and going back to the registers and CMSIS level, writing everything manually. There is an alternative - Low Layer libraries. These exist for all the peripherals that don’t require upper-level stack (like USB or Ethernet), and are significantly lighter than HAL. However, they are also more complicated to use, as they don’t cover the functionality as HAL libraries do. LL offers a low-level API that’s just above the registers. In other words, LL is an almost unnoticeable (performance and memory-wise) layer that abstracts the registers away, giving you fairly well documented and more readable API instead, that doesn’t take away much control from you.

To learn about HAL and LL differences in detail, i suggest going to CubeMX package site for your STM32 MCU (in my case: this one), into Documentation section, and reading User Manual called Description of STM32XY HAL and low-layer drivers, where XY symbolize you series. You can find full documentation of both HAL and LL libraries there, along with their philosophy.

There is an easy way of changing between HAL and LL drivers in CubeMX: open the project and go to Project Manager -> Advanced Settings.

You can do three things there:

Change between HAL and LL drivers for every peripheral, and even every instance of peripheral;
Configure the initialization order of used peripherals (very useful for issues with DMA, when it’s wrongly initialized after the peripheral that’s using it);
Enable or disable callback registration per peripheral (as an alternative to global interrupt callbacks).

While you still might not want to use LL, it’s a good idea to take a look there if you look for memory to free. For example, if you are not using any peripheral-related functions in code, yet you need that peripheral initialized (which can be a case for RCC for example), you can switch it from HAL to LL to save few bytes of memory without touching the code yourself. But remember: HAL and LL functions ARE NOT inter-compatible. You CANNOT use both HAL and LL for the same peripheral at the same time (without proper precautions), as HAL requires a handle that’s state reflects the current state of peripheral. If you start using non-HAL code along with HAL, you HAVE TO make sure the state in HAL handle will not be invalid when HAL functions are being called (in interrupts that use HAL handlers too!), otherwise HAL code can break the program.

Summary

And that would be it. Thank you for reading, and if you have questions or issues, feel free to contact me or drop an issue/pull request in the blog’s repository.

Visual Studio Code - C/C++ Setup

Tue, 08 Jun 2021 00:00:00 +0000

Visual Studio Code is a great open-source editor with plenty of useful plugins for insane amount of languages and frameworks.

However, as C and C++ environment is pretty janky for today’s standards, so is the configuration. So i made this guide to streamline the process and make it easy for somebody new in C, C++ or VSCode to setup a reasonably working dev environment with some useful quality-of-life tools. It might not be IntelliJ-level of quality, but hey - it’s free.

In this guide, i will tell you:

What tools you’ll need to start developing C and C++ apps in VSCode
What extensions you might want to install to ease up the code writing process, and how to configure them
How do you create a C/C++ project in VSCode (with CMake) and integrate it with VSCode

Prerequisites

VSCode

The obvious prerequisite is Visual Studio Code. Install it from the official site: https://code.visualstudio.com/, or from a repository if you’re using a package manager.

For Windows: either download the installer from official site and run it, or install VSCode via scoop: scoop install vscode. btw; i strongly recommend scoop - great package manager

For Linux: if you have it in your package manager repository, install it from there. Otherwise, use the installer from official site.

For MacOS: same as for Linux. Probably. I don’t use MacOS so i can’t really tell.

Fun fact; there are two versions of VSCode you can find on the internet and in package managers - OSS version, and Non-OSS. The OSS version is basically the VSCode you’d get by downloading it from official repository and building it yourself. Non-OSS version is the one from Microsoft distribution (for example, their official site), and the only difference between them is that Non-OSS version contains some Microsoft proprietary code, while OSS doesn’t. There is some functional difference (IIRC, OSS version lacks proprietary features like Settings Sync or Remote WSL/SSH/Containers), but both are fully compatible in terms of plugins and configuration, so you usually don’t need to worry about the exact version you’ve installed.

C/C++ Toolchain

There are many C/C++ toolchains available, and i’m not gonna enforce one, because this guide is mostly toolchain-independent (in the end we’re gonna use build system, but most of the examples are for GCC). However, if you are new to C/C++, i’d recommend starting with GCC (or MinGW, if you’re on Windows) as it’s easy to install and use out of the box.

If you already have your preferred toolchain installed and configured, feel free to go to the next step. If not, here’s the guide:

For Windows:

My usual way of installing C++ toolchain on Windows is via WinLibs package. To install it:

Download latest UCRT runtime version for Win64 (i assume you’re running at least Win10 64bit, otherwise read the instructions on WinLibs page).
Extract it somewhere - preferably directly to a hard drive, for example C:\, but it can be anywhere. I recommend avoiding paths with spaces and non-ASCII characters though, they may cause headaches in the future.
Add /bin subdirectory to system or user PATH variable.
Open terminal and run gcc --version command. You should see similar message to the one below. You may also want to run where.exe gcc to make sure you’re using correct GCC executable, in case you’d have other GCC installation in the system that you forgot about - usually, WinLibs provides most recent one, so this is the one you should be using.

If you’d rather use different toolchain than GCC, i recommend Visual C++, which you can install via Build Tools for Visual Studio 2022. You can download it from here. To install C++ toolchain, run the installer and select the option “Desktop development with C++”. Note that you don’t need both toolchains, one is enough, and that build tools do not include Visual Studio - if you want Visual Studio, you can install it as separate product using the same installer. Since this guide is about Visual Studio Code, which is a different product, i’m not going to provide instructions for that.

For Linux:

Usually, you should have GCC in your repository. On Ubuntu, Debian and similar distributions (Mint, Kubuntu, Lubuntu, PopOS!, Zorin, and so on), you have build-essential package with most tools needed to build C/C++ programs.

On Arch Linux and similar distributions (Manjaro), you have base-devel package.

On other distributions, search for similar package or install latest gcc and g++ packages from your repository.

Test it the same way as on Windows - open terminal and try gcc --version, see what happens.

You also have to install gdb (GCC debugger) separately, as it may not come with the base development packages, and you definitely do want to have it and use it.

For MacOS: probably same thing as on Linux, look for GCC (or any other preferred toolchain) in package manager and install it from there. Verify the installation the same way as on Linux.

VSCode Essential Plugins

If you already have working C/C++ toolchain, time to run VSCode and install some plugins. Run VSCode and go to the Extensions menu.

Now, for some general C++ plugins:

C/C++ - that’s the core extension we’ll need. It contains the fundamental stuff to work with C/C++ in VSCode.
Better C++ Syntax - it’s always nice to have better syntax coloring, so i strongly recommend that one. You might want to use one of the themes from this extension description to get full experience.
C/C++ Snippets - pretty useful extension that adds automatic generation of snippets in C/C++ code - instead of writing loops, structures and class definitions by hand, you can generate them with autocompletion support.
C++ Intellisense - pretty good plugin with some intelligent autocompletion features.
C++ Helper - simple extension which adds automatic function definition generation feature.
C-mantic - very useful plugin that adds auto-generation of function definitions, getters, setters and more. An alternative to C++ Helper - pick whatever seems more ergonomic for you.

And i’d also recommend these:

Bracket Pair Colorizer 2 - very useful extension which colorize the matching bracket pairs, increasing code clarity. Strongly recommended.
GitLens - if you want to work with Git repositories, that’s the extension you’re looking for. You need git installed to use it!
Material Icon Theme - better looking than default ones

With these plugins, you will have a pretty decent bare-bones environment to work with C and C++. You’ll have autocompletion, some refactoring features, some code generation and a pretty decent syntax highlighting. The screenshot below is outdated - use the list above for updated recommended plugins

I have one more plugin to show you, an alternative language server with many useful features, but i’ll leave it as a bonus at the end of this guide, because it needs a bit more configuration. Make sure to check it out!

Creating a project - VSCode and CMake

We’ll start with something simple. As i’ve mentioned before, i’m not gonna teach you how to make a raw VSCode project, which builds the app from scratch and without any other tools, because that’s simply painful, not really scalable, and not worth the trouble.

Instead, i’m gonna teach you how to use a build system.

But what’s a build system?

Well, build system is a tool that tells the toolchain (compiler and his friends) how to create a program out of all the source (and resource) files you’ve created. And sometimes does other things, but that’s out of this tutorial scope.

Without it, you’d have to enter the toolchain commands manually each time you’d want to build the application. That’s fine for small apps with one, two or maybe five files. But it gets messy when your program starts to grow.

So, for example, assuming you use gcc, to build your C program manually, you’d have to enter something like this:

gcc [list of your source files] [some fancy flags for your compiler] [maybe some flags telling the compiler where the libraries are] -o program.exe

An actual example would be:

gcc main.c lib.c lib2.c -O2 -Wall -Wextra -L./some/lib -lmylib -lsomeotherlib -o program.exe

That doesn’t look so bad, right? You could even put this command in some shell script and easily run it every time you’d like to build the code. But then, you’d have to change this command every time your project structure changes - so, every time you add a new file, or library, or change some directory name, you gotta edit this script. And yes, i know that wildcards exist, but for the sake of this example i’m gonna ignore them.

Another issue is that if you’d want to give this code to your friend or teacher, he would either have to use the same shell as you, and gcc, or write his own build script (or project) for the compiler/shell he’s using, and that’s not very user-friendly (or, rather, programmer-friendly) solution.

There are also some other issues with manual building, but the point is: manual building is not comfortable or scalable on a larger scale. So, we’ll use a build system to do it for us and make everyone’s life easier!

Bonus note: Tool i’m going to talk about next - CMake - isn’t technically a build system. It’s a meta-build system. The difference between those is that build system runs the toolchain commands directly, while meta-build system generates the build system files. Basically, meta-build systems are more flexible, and sometimes easier to use, therefore we’re gonna use one.

Bonus note number two: originally, i was going to show how to use Premake and CMake, but due to some pretty bad issues with Premake i’ve decided to stay with CMake. I’ll probably make a Premake guide in the future.

Okay, how do i use it?

Well, let’s start with installation. You can either install it from your package manager, just like VSCode, or download from official site. Make sure to add CMake to your PATH variable if you’re using an official installer!.

Important note: if you’re using your package manager, check the CMake version after installation (cmake --version command). I’m gonna use some stuff that was added in CMake 3.12, but from what i can see, some older Linux distributions (like Ubuntu 18.04) still have CMake 3.10 in their repositories, so if that’s the case i strongly recommend installing newer version manually. If that’s not possible, i’ll tell what things come from CMake 3.12 and how to make a workaround.

Done? Great. Now a little bit of theory.

How does CMake work?

CMake is a meta-build system. As i’ve mentioned earlier, it means that when we run it, it should give us a project for a build system of our choice, which we can use to build our application. To tell CMake how it should generate the project, a CMakeLists.txt file is used. This file includes the project configuration, written in CMake’s scripting language.

Some toolchains - like GCC, MinGW or Visual C++ - come with their own build systems. In case of GCC/MinGW, it’s GNU Make. In case of Visual C++, it’s MSBuild. CMake can generate the necessary files for these build systems, and then we can use them to build the whole project with a single command. Pretty convenient.

Creating a simple project

Let’s open VSCode (or restart, if you had it opened while installing CMake) and add some extensions.

These two plugins will enable CMakeLists.txt syntax highlighting and CMake integration for VSCode. And this integration is a very powerful and helpful tool, as we’ll see in a bit. You can also install a cmake-format extension, if you have Python installed and follow the plugin’s installation guide

Now, make a folder for our new project and create a simple main.cpp (or main.c, if you want to code in C):

C++ version:

#include <iostream>

int main() {
 std::cout << "Hello, world!" << std::endl;
}

C version:

#include <stdio.h>

int main() {
 printf("Hello, world!\n");
}

You can check if your toolchain works correctly by building the app manually. If you’re using GCC or MinGW, open up integrated VSCode terminal with Ctrl+` command (or Ctrl+Shift+` to create a new one), and run:

C++: g++ main.cpp -o main

C: gcc main.c -o main

And then run the app with ./main command. You should see your Hello World printed in terminal.

If that doesn’t work, make sure you have correctly added your toolchain to PATH variable.

Now, we can proceed with CMake. Delete the compiled program main you just created and tested, and create a CMakeLists.txt file in the same directory as your code file. We’ll start with bare-bones template and then we’ll expand it a little.

Put this code into CMakeLists.txt:

cmake_minimum_required(VERSION 3.12)

# Change CXX to C, if you're making a C program
project(HelloWorld LANGUAGES CXX)

# Change main.cpp to main.c, if you're making a C program
add_executable(${PROJECT_NAME} main.cpp)

And then open up the VSCode command list with Ctrl+Shift+P shortcut, and look for CMake: Configure option. Run it.

Next, you should see a list of detected toolchains installed in your system. Pick one.

After that, VSCode will run CMake and configure the project for the first time. You should see similar output in your VSCode output window:

From now on, VSCode will automatically run CMake every time you change CMakeLists.txt to re-generate the project files. You should also see a new menu on left-side toolbar

Building your program

So, we have a project now. How do we build it, and how do we run our program?

Thankfully, CMake plugin for VSCode got us covered. To build the program, either use F7 shortcut, look for the CMake: Build command in command list, or press the Build button either on VSCode bottom bar, or in CMake menu.

You should see similar output in VSCode output window:

Running your program

Now, let’s run it. Press the Run button on the bottom VSCode bar:

And the app should run without any issues. The output should be in terminal window.

Is that it?

Of course not - there’s always more!

But yeah, we just created a very simple project with CMake + VSCode, built it and ran it. Now, let’s talk what actually happened and how can we expand our project.

CMake 101 - how does it work?

CMake, as i’ve mentioned multiple times, is a tool that generates the project files used to build it. However, it can also be used for some other things - like application packaging, managing tests, configuring the project, and so on. At the moment CMake is an industry standard, which means that most of the C and C++ projects have CMake support, therefore it’s a very versatile tool. Not the best, but versatile.

You may notice that after configuring the project a new directory appeared in your project folder, called build. Let’s peek into it.

You are usually not supposed to touch and modify these files, as CMake manages them, but i’ll still explain the function of some of the more important ones:

CMakeCache.txt - inside that file, you can find a list of CMake variables that are used in project generation process. You can find the toolchain paths, compiler flags, user and many different configuration variables there.
compile_commands.json - that file contains a list of commands used to build your program. This list can be used by some code analyzing tools (and i’ll describe one at the end of this guide).
HelloWorld.exe - hey, that’s our program!
Makefile - this file is used by GNU Make to build our program. Basically, that’s the final CMake output. If you’re using a different toochain with different build system, like Visual C++, you will get a different file (usually, a whole Visual Studio solution, so maybe even a whole directory).

CMakeLists - what’s inside?

Let’s analyze our CMakeLists.txt

cmake_minimum_required(VERSION 3.12)

# Change CXX to C, if you're making a C program
project(HelloWorld LANGUAGES CXX)

# Change main.cpp to main.c, if you're making a C program
add_executable(${PROJECT_NAME} main.cpp)

cmake_minimum_required defines the minimal version of CMake required to generate this project. We’ll use 3.12 because i’m gonna show you some stuff that was added in this version.
project defines our project. We can also add information about version, project description, homepage and used languages - which we do in that case.
add_executable tells CMake to generate a code that will build an executable file with specified name from specified source files. ${PROJECT_NAME} is a CMake variable containing the project name defined by project command.

It’s important that add_executable requires a full list of all the source files making up the program. So, in theory, you would have to add every single source file to the list manually, like that: add_executable(${PROJECT_NAME} main.cpp a.cpp b.cpp [...]), but fortunately that’s not necessary. We can use file command to generate that list for us.

Expanding our project - adding more files

Let’s add some more files for our project. First, create two new directories - include and src in root project directory:

Now, move the main.cpp (or main.c) to src directory, and make a new file called lib.cpp (or lib.c) there. Put this code inside:

C++ version:

#include "lib.hpp"
#include <iostream>

void f(int x) {
 std::cout << "Hello, i'm a library!\nx = " << x << std::endl;
}

C version:

#include "lib.h"
#include <stdio.h>

void f(int x) {
 printf("Hello, i'm a library!\nx = %d\n", x);
}

Then, add a lib.hpp (or lib.h) file inside include directory. Put this inside:

#pragma once

void f(int x);

The project should look like this now:

Let’s update the CMakeLists.txt.

We’ll use two new commands now: file and include_directories. file command will generate the list of source files, while include_directories will tell our toolchain where to look for header files.

cmake_minimum_required(VERSION 3.12)

# Change CXX to C, if you're making a C program
project(HelloWorld LANGUAGES CXX)

# Change *.cpp to *.c, if you're making a C program
file(GLOB PROJECT_SOURCE_FILES CONFIGURE_DEPENDS src/*.cpp)
include_directories(include/)

# Change main.cpp to main.c, if you're making a C program
add_executable(${PROJECT_NAME} ${PROJECT_SOURCE_FILES})

file(GLOB PROJECT_SOURCE_FILES CONFIGURE_DEPENDS src/*.cpp) - this command generates the list of files from src dir with .cpp extension, and stores it in PROJECT_SOURCE_FILES variable. The CONFIGURE_DEPENDS flag was added in CMake 3.12, and thanks to it, the list of files will be automatically re-generated every time we add a new file to the project. The command will work without it, but then you would have to manually regenerate CMake project after adding new files.
include_directories tells the toolchain where to look for header files.

The PROJECT_SOURCE_FILES variable should have a list of all of our code files, and we’ll pass it to add_executable instead of adding a new file manually. Of course, this variable can be called anyhow you want - it’s not a special name or anything.

Now, let’s configure our project again (VSCode should do this automatically, but i wanna show you how to do it in case it doesn’t). Right-click on CMakeLists.txt and select “Configure All Projects”

Sometimes, when something goes wrong and CMake or VSCode starts behaving strange or getting buggy, it’s good to clean up and reconfigure the project (“Clean Reconfigure All Projects” and “Clean Rebuild All Projects”). This can sometimes happen when playing with CMakeLists.

Let’s add some code to our main.cpp (or main.c) to test if additional files are added properly to our project:

C++ version:

#include <iostream>
#include "lib.hpp"

int main() {
 std::cout << "Hello, world" << std::endl;
 f(42);
}

C version:

#include <stdio.h>
#include "lib.h"

int main() {
 printf("Hello, world!\n");
 f(42);
}

Now, we should be able to build our program

And run it

At this point, everything should work automatically. You should be able to add new source files to src dir, and new headers to include, and CMake should handle them without having to touch CMakeLists.txt manually. Adding new directories with source/include files is done the same way - just file the source files, add include path with include_directories and voila.

Debugging our code

Time for a last step in our setup - using a debugger. Fortunately, CMake integration does most of the work for us here.

Let’s add a breakpoint in the first line of our main function - click on the left of line number, you should see a red dot:

And start a debugging session by pressing a little bug icon on the bottom of VSCode window, or by using CMake: Debug command

VSCode should now switch into debugging perspective and program should stop at breakpoint.

And that’s basically it. Don’t worry about lack of configuration for run/debug in VSCode - CMake integration does everything for us, so there’s no need to make them. Just make sure you run/debug the program via CMake commands in VSCode, instead of it’s own. Setting up the run/debug configs manually can be pretty annoying at the times, so that’s a subject for another guide.

There’s one subject i haven’t touched yet, and it’s adding external libraries. Since this guide is already long enough, i’ll make another one for that. Now, it’s the time for some bonus content.

Bonus: clangd setup

clangd is a language server for C++. It provides functionalities like code completion, code linting (showing warning and errors in real-time), simple refactoring, and so on. There is a C++ language server in VSCode already, as we installed C/C++ extension, but clangd is better in some ways, and it’s more multi-platform (default VSCode C++ language server doesn’t work on ARM yet, so if you wanna code on RaspberryPi via SSH - clangd is your best friend).

In order to work, clangd requires compile_commands.json file to know how your code is compiled and with what flags. Fortunately for us, CMake generates that file automatically, so no further configuration is required on our side!

Right after the plugin installation, you should see a popup like this one:

In order for clangd to work, you have to disable the default Intellisense server (press Disable IntelliSense), and download clangd binary (which VSCode does for us). After it’s downloaded, you should see this popup:

Reload the window, open up a C/C++ source file, and check if clangd info is displayed on the bottom of the screen

If that’s the case, congratulations, it should work now. Check if there’s no include errors, and if the autocompletion works (you can manually trigger autocompletion with Ctrl+Space shortcut). There should also be a .cache directory in your project folder now, with clangd’s cache files - make sure to add it to .gitignore before making a commit :P

Full list of clangd features can be found here: https://clangd.llvm.org/features.html. I strongly recommend checking it out.

Clang-Format setup

clangd embeds clang-format support, which is a code formatting tool - one of the best in C and C++ environment. clang-format allows you to manually or automatically format the code (for example, on-save, format-while-typing is not yet supported with clangd) according to your preferences, which you can set by creating .clang-format file in your workspace root, with configuration options for clang-format.

The list of configuration options is pretty long and can be found here: https://clang.llvm.org/docs/ClangFormatStyleOptions.html. I’m not gonna describe them all of course, i’ll just show an setup example.

Let’s create a .clang-format file in our workspace root dir, and put this inside:

BasedOnStyle: Chromium
IndentWidth: 2
Standard: c++17
BreakBeforeBraces: Linux

If clangd is the only formatter you have installed, it should be treated as default. If not, make sure it is by opening command list (Ctrl+Shift+P), looking for Format Document With... option, and setting default formatter to clangd.

Now, open a main file and either right-click and pick Format document option, or use a shortcut (Shift+Alt+F on Windows by default). The code should now look like this:

If it had been formatted differently, check the clangd output in VSCode window. All the issues and errors with .clang-format file should be there.

You can set autoformatting in VSCode settings - open settings window with Ctrl+, or File -> Preferences -> Settings, and look for format on options

Summary

And that would be it. I hope it’s not too long.

If there are any issues or questions about the guide itself or the setup process, feel free to contact me!