selfhost.directory

The index of self-hostable projects — ranked, categorized and served with live updates.
Own your data and stop giving it to big tech.

InternScience logo

Can the NVIDIA H100 SXM 80GB run MLExpert?

MLExpert·34.7B (3.5B active)·NVIDIA H100 SXM 80GB·80 GB VRAM

Saved in this browser, used on every model page.
Yes

Yes. The NVIDIA H100 SXM 80GB runs MLExpert fully in VRAM at FP16 / BF16 (about 69 GB of 80 GB), at roughly 267 tokens/s.

best quant
FP16 / BF16
needs
69 GB
usable
80 GB
est. speed
~267 tok/s

Estimates for an 8K context; offload assumes 32 GB of system RAM.Try other contexts on the MLExpert page →

Every quantisation of MLExpert on the NVIDIA H100 SXM 80GB

QuantQualityMemoryFitSpeed
Q2_K Heavy loss15 GBfits~1622 tok/s
Q3_K_M Noticeable loss18 GBfits~1091 tok/s
Q4_K_M Small loss23 GBfits~880 tok/s
Q5_K_M Near-lossless28 GBfits~750 tok/s
Q6_K Near-lossless31 GBfits~647 tok/s
Q8_0 Lossless37 GBfits~502 tok/s
FP16 / BF16 pickLossless69 GBfits~267 tok/s

Speed is a bandwidth-based estimate for generating text (3350 GB/s); prompt processing and long contexts are slower.

Measured speed on the NVIDIA H100 SXM 80GB

No community reports for this pair yet.

Report your own speed →

Run MLExpert on the NVIDIA H100 SXM 80GB

Using FP16 / BF16, the best fit for NVIDIA H100 SXM 80GB.

Ollama · BF16 straight from Hugging Facedocs
ollama run hf.co/bartowski/InternScience_MLExpert-GGUF:BF16
llama.cpp · OpenAI-compatible server with a web UIdocs
llama-server -hf bartowski/InternScience_MLExpert-GGUF:BF16

Downloads the GGUF on first run. Installed through llama.app? The same flags work with `llama serve`.

LM Studio · Download with the lms CLIdocs
lms get bartowski/InternScience_MLExpert-GGUF@BF16

Then load it from the LM Studio app or with `lms load`.

vLLM · Serve an OpenAI-compatible APIdocs
vllm serve InternScience/MLExpert
vLLM · Docker Compose (NVIDIA)docs
services:
  vllm:
    image: vllm/vllm-openai:latest
    command: InternScience/MLExpert
    ports:
      - "8000:8000"
    ipc: host
    volumes:
      - ~/.cache/huggingface:/root/.cache/huggingface
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

API on port 8000.

SGLang · Serve an OpenAI-compatible APIdocs
sglang serve InternScience/MLExpert --host 0.0.0.0 --port 30000
SGLang · Docker Compose (NVIDIA)docs
services:
  sglang:
    image: lmsysorg/sglang:latest
    command: sglang serve InternScience/MLExpert --host 0.0.0.0 --port 30000
    ports:
      - "30000:30000"
    ipc: host
    volumes:
      - ~/.cache/huggingface:/root/.cache/huggingface
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

API on port 30000.

Transformers · Chat with the pipeline APIdocs
from transformers import pipeline

pipe = pipeline("text-generation", model="InternScience/MLExpert", dtype="auto", device_map="auto")
messages = [{"role": "user", "content": "Explain self-hosting in one paragraph."}]
out = pipe(messages, max_new_tokens=512)
print(out[0]["generated_text"][-1]["content"])

Chat models that fit the NVIDIA H100 SXM 80GB comfortably

All that fit →
ModelParamsBest quantMemorySpeedLicence
36BA3BQ8_0≈38 GBFits in memory~578 tok/sCommercial OK
31.2BA4.6BQ8_0≈39 GBFits in memory~376 tok/sCommercial OK
32.8BQ8_0≈37 GBFits in memory~53 tok/sCommercial OK
RecursiveSI logoflintRecursiveSI
39.8BA1.3BQ8_0≈43 GBFits in memory~1387 tok/sCommercial OK
20.9BA4.2BQ8_0≈13 GBFits in memory~414 tok/sCommercial OK
30.5BA3BQ8_0≈33 GBFits in memory~578 tok/sCommercial OK

Other GPUs that run MLExpert

Frequently asked questions

Can the NVIDIA H100 SXM 80GB run MLExpert?
Yes. The NVIDIA H100 SXM 80GB runs MLExpert fully in VRAM at FP16 / BF16 (about 69 GB of 80 GB), at roughly 267 tokens/s.
How fast does MLExpert run on the NVIDIA H100 SXM 80GB?
An estimate from memory bandwidth gives about 267 tokens/s at FP16 / BF16.
Which quantisation of MLExpert should I use on the NVIDIA H100 SXM 80GB?
FP16 / BF16: the largest one that fits. Expected quality: lossless compared with the full-precision weights.

Memory and speed are estimates from the model's size and the card's memory bandwidth; real numbers depend on the runtime, drivers and context length. Community figures are medians of user reports.