selfhost.directory

The index of self-hostable projects — ranked, categorized and served with live updates.
Own your data and stop giving it to big tech.

InternScience logo

Can the Instinct MI50 32GB run MLExpert?

MLExpert·34.7B (3.5B active)·Instinct MI50 32GB·32 GB VRAM

Saved in this browser, used on every model page.
Yes

Yes. The Instinct MI50 32GB runs MLExpert fully in VRAM at Q5_K_M (about 28 GB of 32 GB), at roughly 229 tokens/s.

best quant
Q5_K_M
needs
28 GB
usable
32 GB
est. speed
~229 tok/s

Estimates for an 8K context; offload assumes 32 GB of system RAM.Try other contexts on the MLExpert page →

Every quantisation of MLExpert on the Instinct MI50 32GB

QuantQualityMemoryFitSpeed
Q2_K Heavy loss15 GBfits~496 tok/s
Q3_K_M Noticeable loss18 GBfits~334 tok/s
Q4_K_M Small loss23 GBfits~269 tok/s
Q5_K_M pickNear-lossless28 GBfits~229 tok/s
Q6_K Near-lossless31 GBtight~198 tok/s
Q8_0 Lossless37 GBoffload~47 tok/s
FP16 / BF16 Lossless69 GBtoo big—

Speed is a bandwidth-based estimate for generating text (1024 GB/s); prompt processing and long contexts are slower.

Measured speed on the Instinct MI50 32GB

No community reports for this pair yet.

Report your own speed →

Run MLExpert on the Instinct MI50 32GB

Using Q5_K_M, the best fit for Instinct MI50 32GB.

Ollama · Q5_K_M straight from Hugging Facedocs
ollama run hf.co/bartowski/InternScience_MLExpert-GGUF:Q5_K_M

Ollama’s ROCm/CUDA builds don’t cover this GPU. Recent Ollama releases use Vulkan when its Vulkan backend and driver are installed (older ones needed OLLAMA_VULKAN=1); llama.cpp or LM Studio with Vulkan are the safer bet.

llama.cpp · OpenAI-compatible server with a web UIdocs
llama-server -hf bartowski/InternScience_MLExpert-GGUF:Q5_K_M

Needs a Vulkan build of llama.cpp. Downloads the GGUF on first run.

llama.cpp · Docker with Vulkandocs
docker run -d --name llama --device /dev/dri -p 8080:8080 \
  -v ~/.cache/llama.cpp:/root/.cache/llama.cpp \
  ghcr.io/ggml-org/llama.cpp:server-vulkan \
  -hf bartowski/InternScience_MLExpert-GGUF:Q5_K_M --host 0.0.0.0 --port 8080 -ngl 99

Server and web UI on port 8080.

LM Studio · Download with the lms CLIdocs
lms get bartowski/InternScience_MLExpert-GGUF@Q5_K_M

Then load it from the LM Studio app or with `lms load`.

Transformers · Chat with the pipeline APIdocs
from transformers import pipeline

pipe = pipeline("text-generation", model="InternScience/MLExpert", dtype="auto", device_map="auto")
messages = [{"role": "user", "content": "Explain self-hosting in one paragraph."}]
out = pipe(messages, max_new_tokens=512)
print(out[0]["generated_text"][-1]["content"])

Chat models that fit the Instinct MI50 32GB comfortably

All that fit →
ModelParamsBest quantMemorySpeedLicence
36BA3BQ5_K_M≈26 GBFits in memory~264 tok/sCommercial OK
31.2BA4.6BQ5_K_M≈29 GBFits in memory~172 tok/sCommercial OK
32.8BQ6_K≈29 GBFits in memory~21 tok/sCommercial OK
RecursiveSI logoflintRecursiveSI
39.8BA1.3BQ5_K_M≈29 GBFits in memory~633 tok/sCommercial OK
20.9BA4.2BQ8_0≈13 GBFits in memory~126 tok/sCommercial OK
30.5BA3BQ6_K≈26 GBFits in memory~228 tok/sCommercial OK

Other GPUs that run MLExpert

Frequently asked questions

Can the Instinct MI50 32GB run MLExpert?
Yes. The Instinct MI50 32GB runs MLExpert fully in VRAM at Q5_K_M (about 28 GB of 32 GB), at roughly 229 tokens/s.
How fast does MLExpert run on the Instinct MI50 32GB?
An estimate from memory bandwidth gives about 229 tokens/s at Q5_K_M.
Which quantisation of MLExpert should I use on the Instinct MI50 32GB?
Q5_K_M: the largest one that fits. Expected quality: near-lossless compared with the full-precision weights.

Memory and speed are estimates from the model's size and the card's memory bandwidth; real numbers depend on the runtime, drivers and context length. Community figures are medians of user reports.