selfhost.directory

The index of self-hostable projects — ranked, categorized and served with live updates.
Own your data and stop giving it to big tech.

InternScience logo

Can the Radeon RX 7900 XTX run MLExpert?

MLExpert·34.7B (3.5B active)·Radeon RX 7900 XTX·24 GB VRAM

Saved in this browser, used on every model page.
Yes, at low quality

Yes, but only at Q3_K_M (about 18 GB of 24 GB), which costs noticeable quality, at roughly 313 tokens/s.

best quant
Q3_K_M
needs
18 GB
usable
24 GB
est. speed
~313 tok/s

Estimates for an 8K context; offload assumes 32 GB of system RAM.Try other contexts on the MLExpert page →

Every quantisation of MLExpert on the Radeon RX 7900 XTX

QuantQualityMemoryFitSpeed
Q2_K Heavy loss15 GBfits~465 tok/s
Q3_K_M pickNoticeable loss18 GBfits~313 tok/s
Q4_K_M Small loss23 GBtight~252 tok/s
Q5_K_M Near-lossless28 GBoffload~74 tok/s
Q6_K Near-lossless31 GBoffload~42 tok/s
Q8_0 Lossless37 GBoffload~23 tok/s
FP16 / BF16 Lossless69 GBtoo big—

Speed is a bandwidth-based estimate for generating text (960 GB/s); prompt processing and long contexts are slower.

Measured speed on the Radeon RX 7900 XTX

No community reports for this pair yet.

Report your own speed →

Run MLExpert on the Radeon RX 7900 XTX

Using Q3_K_M, the best fit for Radeon RX 7900 XTX.

Ollama · Q3_K_M straight from Hugging Facedocs
ollama run hf.co/bartowski/InternScience_MLExpert-GGUF:Q3_K_M
Ollama · Docker with ROCm (AMD)docs
docker run -d --name ollama --device /dev/kfd --device /dev/dri \
  -v ollama:/root/.ollama -p 11434:11434 ollama/ollama:rocm
docker exec -it ollama ollama run hf.co/bartowski/InternScience_MLExpert-GGUF:Q3_K_M

The image bundles the ROCm runtime; the host only needs the amdgpu kernel driver.

llama.cpp · OpenAI-compatible server with a web UIdocs
llama-server -hf bartowski/InternScience_MLExpert-GGUF:Q3_K_M

Needs a HIP (ROCm) build of llama.cpp. Downloads the GGUF on first run.

llama.cpp · Docker with ROCm (AMD)docs
docker run -d --name llama --device /dev/kfd --device /dev/dri -p 8080:8080 \
  -v ~/.cache/llama.cpp:/root/.cache/llama.cpp \
  ghcr.io/ggml-org/llama.cpp:server-rocm \
  -hf bartowski/InternScience_MLExpert-GGUF:Q3_K_M --host 0.0.0.0 --port 8080 -ngl 99

Server and web UI on port 8080.

LM Studio · Download with the lms CLIdocs
lms get bartowski/InternScience_MLExpert-GGUF@Q3_K_M

Then load it from the LM Studio app or with `lms load`.

vLLM · Serve an OpenAI-compatible APIdocs
vllm serve InternScience/MLExpert

Needs vLLM built for ROCm; it supports Instinct MI200/MI300 and Radeon RX 7900 / 9000 cards.

vLLM · Docker Compose (AMD ROCm)docs
services:
  vllm:
    image: rocm/vllm:latest
    command: vllm serve InternScience/MLExpert --host 0.0.0.0 --port 8000
    ports:
      - "8000:8000"
    ipc: host
    devices:
      - /dev/kfd
      - /dev/dri
    group_add:
      - video
    security_opt:
      - seccomp=unconfined
    volumes:
      - ~/.cache/huggingface:/root/.cache/huggingface

AMD’s rocm/vllm image. API on port 8000.

SGLang · Serve an OpenAI-compatible APIdocs
sglang serve InternScience/MLExpert --host 0.0.0.0 --port 30000

Needs SGLang’s ROCm build (Instinct MI300-class GPUs).

Transformers · Chat with the pipeline APIdocs
from transformers import pipeline

pipe = pipeline("text-generation", model="InternScience/MLExpert", dtype="auto", device_map="auto")
messages = [{"role": "user", "content": "Explain self-hosting in one paragraph."}]
out = pipe(messages, max_new_tokens=512)
print(out[0]["generated_text"][-1]["content"])

Chat models that fit the Radeon RX 7900 XTX comfortably

All that fit →
ModelParamsBest quantMemorySpeedLicence
36BA3BQ4_K_M≈23 GBFits in memory~290 tok/sCommercial OK
32.8BQ4_K_M≈22 GBFits in memory~27 tok/sCommercial OK
30.5BA3BQ5_K_M≈22 GBFits in memory~247 tok/sCommercial OK
20.9BA4.2BQ8_0≈13 GBFits in memory~119 tok/sCommercial OK
32.8BQ4_K_M≈22 GBFits in memory~27 tok/sCommercial OK
33.4BA2.2BQ4_K_M≈22 GBFits in memory~389 tok/sCommercial OK

Other GPUs that run MLExpert

Frequently asked questions

Can the Radeon RX 7900 XTX run MLExpert?
Yes, but only at Q3_K_M (about 18 GB of 24 GB), which costs noticeable quality, at roughly 313 tokens/s.
How fast does MLExpert run on the Radeon RX 7900 XTX?
An estimate from memory bandwidth gives about 313 tokens/s at Q3_K_M.
Which quantisation of MLExpert should I use on the Radeon RX 7900 XTX?
Q3_K_M: the largest one that fits. Expected quality: noticeable loss compared with the full-precision weights.

Memory and speed are estimates from the model's size and the card's memory bandwidth; real numbers depend on the runtime, drivers and context length. Community figures are medians of user reports.