Strix Halo AI Toolboxes

The Project

In August 2025, I got my hands on a Strix Halo machine. I needed to run local inference for some Cyber Security work where Cloud LLMs were not an option.

I quickly realized the software ecosystem wasn't ready. Stuff wasn't working. So I started digging, learning, and fixing things. I shared my findings in a video. People found it useful, so I've continued to maintain these toolboxes to help others unlock the potential of their hardware.

Thanks to support from the Strix Halo Home Lab community, Framework, and AMD, I've continued to maintain these "Toolboxes" to help others reproduce this setup and run AI workloads on Strix Halo hardware.

// WHOAMI

Donato Capitella

Software Engineer and Ethical Hacker. I enjoy understanding systems by breaking them down and documenting the process.

YouTube Channel LinkedIn Profile LLM Chronicles

Support my work:

☕ Buy me a coffee

What is Strix Halo?

// RYZEN AI MAX+

"Strix Halo" (Ryzen AI MAX+) is AMD's high-performance mobile processor platform. Its key feature for AI workloads is Unified Memory, allowing the iGPU to access up to 128GB of system RAM, significantly increasing the model size capacity compared to traditional consumer GPUs.

Architecture Zen 5 + RDNA 3.5

GPU ID gfx1151

Max Unified Memory 128 GB

-> Official Product Page

Active Toolboxes

// MAINTAINED CONTAINERS

These are containerized environments built on Toolbx (Docker/Podman). This approach allows you to easily get the specific runtime needed for Strix Halo, keep the host system clean, and instantly switch between different ROCm or software versions without dependency conflicts.

Llama.cpp Toolboxes

Setup for LLM inference. Supports clustering via RDMA and Vulkan/ROCm backends.

View Repo ->

ComfyUI Toolboxes

Environment for Image & Video generation. Validated for LTX2, Wan 2.2, HunyuanVideo, and Qwen.

View Repo ->

vLLM Toolboxes

Serving server setup. Includes custom RCCL patches for high-speed clustering.

View Repo ->

LLM Fine-tuning

Training environment. QLoRA and Full Fine-Tuning support for Gemma 3, Qwen 3, and generic models.

View Repo ->

DwarfStar

A small native inference engine optimized first for DeepSeek V4 Flash.

View Repo ->

Llama Cockpit

// TUI CONTAINER MANAGER

Llama Cockpit is a Terminal User Interface (TUI) that makes it easier to manage llama.cpp toolboxes and GGUF weights. It also includes a server mode that doesn't require toolbox or distrobox, running natively via docker/podman to ensure compatibility with any Linux distribution.

GitHub Repo ->

$ pipx install git+https://github.com/kyuz0/llama-toolboxes-cockpit.git

Tutorials & Guides

// YOUTUBE VIDEOS

Performance Benchmarks

// HARDWARE CAPABILITIES

Llama.cpp Benchmarks

Token generation speeds (tokens/sec) across various GGUF models.

View Benchmarks ->

vLLM Benchmarks

Peak multi-user throughput (tokens/sec) and RDMA/RoCE clustering performance.

View Benchmarks ->

ComfyUI Benchmarks

Generation speeds (seconds/it) for HunyuanVideo, Wan 2.2, and Qwen image workflows.

View Benchmarks ->

DwarfStar Benchmarks

Inference performance metrics (tokens/sec) for DeepSeek V4 Flash models.

View Benchmarks ->

SWE-bench Verified Mini

Coding capability and speed metrics on the SWE-bench Verified Mini dataset. Measures real-world execution on Strix Halo using accessible quantized models rather than full-precision server clusters.

View Benchmarks ->

Host Config

// TUNED FOR PERFORMANCE

This is the configuration I use on my Framework Desktop to maintain and benchmark all toolboxes.

My Rig - Sent to me by Framework

System Specifications

Model Framework Desktop

CPU Ryzen AI MAX+ 395 "Strix Halo"

Total RAM 128 GB DDR5

OS Fedora 43 (Linux 6.18.5)

Step 1

Configure and optimize the host

These system-level tweaks free up unified memory for larger models and improve inference performance. The goal is to make ~124 GB of system RAM available to the GPU on demand, disable IOMMU overhead, reduce background memory pressure, and apply a performance-tuned CPU governor profile.

Kernel parameters and unified memory

Add these parameters to GRUB_CMDLINE_LINUX in /etc/default/grub, then regenerate the GRUB config.

root@fedora:~

$ sudo vim /etc/default/grub

GRUB_CMDLINE_LINUX="... amd_iommu=off amdgpu.gttsize=126976 ttm.pages_limit=32505856"

$ sudo grub2-mkconfig -o /boot/grub2/grub.cfg

$ sudo reboot

Parameter	What it changes
amd_iommu=off	Disable AMD IOMMU: Disables the AMD IOMMU entirely, which can improve GPU memory access performance on Strix Halo unified memory setups. Performance Note: Benchmarking by Lars Urban (Issue #66) shows a 5-12% performance increase by setting `amd_iommu=off` instead of the previously recommended pass-through mode. Warning: Disabling IOMMU prevents the NPU from working and removes protection against DMA attacks from rogue or compromised external devices. Leave IOMMU enabled if you need the NPU or DMA isolation.
amdgpu.gttsize=126976	GTT Size (Graphics Translation Table): Explicitly sets the maximum unified memory addressable by the GPU to ~124GB (126976 MB), overriding default driver limits.
ttm.pages_limit=32505856	Pinned Memory Limit: Allows the TTM (Translation Table Manager) to pin up to ~124GB of pages in high-speed system RAM, ensuring the GPU has direct access without swapping.

Disable the desktop environment

Switch to multi-user.target (runlevel 3) to free RAM and GPU resources by disabling the desktop environment and its background processes. Reboot to apply.

root@fedora:~

$ sudo systemctl set-default multi-user.target

$ sudo reboot

# To restore the desktop later:

$ sudo systemctl set-default graphical.target && sudo reboot

Power and performance tuning

Apply the accelerator-performance profile via tuned to disable higher-latency CPU STOP states and maximize throughput.

root@fedora:~

$ sudo dnf install tuned

$ sudo systemctl enable --now tuned

$ sudo tuned-adm profile accelerator-performance

$ tuned-adm active

Current active profile: accelerator-performance

Kernel parameters and unified memory

Add these parameters to GRUB_CMDLINE_LINUX in /etc/default/grub, then regenerate the GRUB config.

root@ubuntu:~

$ sudo vim /etc/default/grub

GRUB_CMDLINE_LINUX="... amd_iommu=off amdgpu.gttsize=126976 ttm.pages_limit=32505856"

$ sudo update-grub

$ sudo reboot

Parameter	What it changes
amd_iommu=off	Disable AMD IOMMU: Disables the AMD IOMMU entirely, which can improve GPU memory access performance on Strix Halo unified memory setups. Performance Note: Benchmarking by Lars Urban (Issue #66) shows a 5-12% performance increase by setting `amd_iommu=off` instead of the previously recommended pass-through mode. Warning: Disabling IOMMU prevents the NPU from working and removes protection against DMA attacks from rogue or compromised external devices. Leave IOMMU enabled if you need the NPU or DMA isolation.
amdgpu.gttsize=126976	GTT Size (Graphics Translation Table): Explicitly sets the maximum unified memory addressable by the GPU to ~124GB (126976 MB), overriding default driver limits.
ttm.pages_limit=32505856	Pinned Memory Limit: Allows the TTM (Translation Table Manager) to pin up to ~124GB of pages in high-speed system RAM, ensuring the GPU has direct access without swapping.

Disable the desktop environment

Switch to multi-user.target (runlevel 3) to free RAM and GPU resources by disabling the desktop environment and its background processes. Reboot to apply.

root@ubuntu:~

$ sudo systemctl set-default multi-user.target

$ sudo reboot

# To restore the desktop later:

$ sudo systemctl set-default graphical.target && sudo reboot

Power and performance tuning

Apply the accelerator-performance profile via tuned to disable higher-latency CPU STOP states and maximize throughput.

root@ubuntu:~

$ sudo apt update && sudo apt install tuned -y

$ sudo systemctl enable --now tuned

$ sudo tuned-adm profile accelerator-performance

$ tuned-adm active

Current active profile: accelerator-performance

Configure unified memory with amd-ttm

Ryzen AI Halo ships with a Debian-based distribution using systemd-boot instead of GRUB. Use the amd-ttm tool to set unified memory — it replaces both amdgpu.gttsize and ttm.pages_limit kernel parameters.

user@halo:~

# Install pipx and amd-debug-tools

$ sudo apt install pipx

$ pipx ensurepath

$ pipx install amd-debug-tools

# Set unified memory to ~124 GB

$ sudo amd-ttm --set 124

🐧 Successfully set TTM pages limit to 32505856 pages (124.00 GB)

# Reboot when prompted

Disable IOMMU via systemd-boot

Set amd_iommu=off via the systemd-boot cmdline file. Do not edit boot entries in /boot/loader/entries/ directly — they are regenerated on each kernel update.

Warning: Disabling IOMMU prevents the NPU from working and removes protection against DMA attacks from rogue or compromised external devices. Leave IOMMU enabled if you need the NPU or DMA isolation.

root@halo:~

# Add amd_iommu=off without removing the existing kernel parameters

$ sudoedit /etc/kernel/cmdline

# Rebuild the systemd-boot entry for the running kernel

$ sudo kernel-install add "$(uname -r)" \

"/boot/vmlinuz-$(uname -r)" \

"/boot/initrd.img-$(uname -r)"

$ sudo reboot

Disable the desktop environment

Switch to multi-user.target (runlevel 3) to free RAM and GPU resources by disabling the desktop environment and its background processes.

root@halo:~

$ sudo systemctl set-default multi-user.target

$ sudo reboot

# To restore the desktop later:

$ sudo systemctl set-default graphical.target && sudo reboot

Power and performance tuning

Apply the accelerator-performance profile via tuned.

root@halo:~

$ sudo apt update && sudo apt install tuned -y

$ sudo systemctl enable --now tuned

$ sudo tuned-adm profile accelerator-performance

$ tuned-adm active

Current active profile: accelerator-performance

Step 2

Run the toolboxes

Once your host is optimized, Llama Cockpit is the recommended way to create, update, and run the llama.cpp environments. Manual container commands remain available below for users who want direct control.

Recommended

Use Llama Cockpit

Llama Cockpit handles the differences between operating systems and container setups. It can manage Toolbox or Distrobox, and its server mode can run directly through Docker or Podman. Use the TUI to choose a backend, create or refresh its environment, manage GGUF models, and start local or distributed llama.cpp workloads.

Learn about Llama Cockpit

user@host:~

# Install once, then launch the TUI

$ pipx install git+https://github.com/kyuz0/llama-toolboxes-cockpit.git

$ llama-cockpit

Manual container setup Use the raw Toolbox or Distrobox commands

Use this path if you prefer to create and manage the containers yourself. Select the host operating system below, then substitute the name and image URL from the toolbox repository you want to run.

Toolbox on Fedora

Fedora has native support for Toolbox, providing seamless container integration. The command below maps the GPU devices and sets the required permissions.

user@fedora:~

# Create your toolbox mapped to the host's GPU

$ toolbox create <TOOLBOX_NAME> \

--image <IMAGE_URL> \

-- --device /dev/dri --device /dev/kfd \

--group-add video --group-add render --security-opt seccomp=unconfined

Note: <TOOLBOX_NAME> and <IMAGE_URL> are placeholders. Check the specific toolbox repository for the correct values.

Distrobox on Ubuntu, Debian, and Ryzen AI Halo

Users on Ubuntu, Debian, or AMD's Ryzen AI Halo Debian-based distro should use Distrobox instead of the default toolbox package, which can have permission issues on these distributions.

user@ubuntu:~

# Step 1: Add your user to required GPU groups

$ sudo usermod -aG video,render $USER

# Step 2: Ensure the compute device is accessible (persists across reboots)

$ echo -e 'SUBSYSTEM=="kfd", KERNEL=="kfd", MODE="0666"\nSUBSYSTEM=="drm", KERNEL=="renderD*", MODE="0666"' | sudo tee /etc/udev/rules.d/70-kfd.rules

$ sudo udevadm control --reload-rules && sudo udevadm trigger

# Step 3: Create your distrobox mapped to the host's GPU

$ distrobox create <TOOLBOX_NAME> \

--image <IMAGE_URL> \

-- --device /dev/dri --device /dev/kfd \

--group-add video --group-add render --group-add sudo --security-opt seccomp=unconfined

Note: <TOOLBOX_NAME> and <IMAGE_URL> are placeholders. Check the specific toolbox repository for the correct values.

Note: This Distrobox configuration has been tested on Ubuntu 25.10 with Mainline Kernel 6.18.7-061807. To enable mainline kernels on Ubuntu, you can use the Ubuntu Mainline Kernel Installer.

Community & Support

Connect with other Strix Halo owners, share benchmarks, and get help.

This is a hobby project that takes a lot of time to maintain and test. If you find these toolboxes useful, consider supporting the work.

Strix Halo Home Lab Wiki Join the Discord ☕ Buy me a Coffee