selfhost.directory

The index of self-hostable projects — ranked, categorized and served with live updates.
Own your data and stop giving it to big tech.

ByteDance-Seed logo

Can the Radeon RX 6700 XT run Cola2?

Cola2·8.8B·Radeon RX 6700 XT·12 GB VRAM

Saved in this browser, used on every model page.
Yes

Yes. The Radeon RX 6700 XT runs Cola2 fully in VRAM at Q8_0 (about 11 GB of 12 GB), at roughly 23 tokens/s.

best quant
Q8_0
needs
11 GB
usable
12 GB
est. speed
~23 tok/s

Estimates for an 8K context; offload assumes 32 GB of system RAM.Try other contexts on the Cola2 page →

Every quantisation of Cola2 on the Radeon RX 6700 XT

QuantQualityMemoryFitSpeed
Q2_K Heavy loss4.4 GBfits~73 tok/s
Q3_K_M Noticeable loss5.8 GBfits~49 tok/s
Q4_K_M Small loss6.8 GBfits~40 tok/s
Q5_K_M Near-lossless7.7 GBfits~34 tok/s
Q6_K Near-lossless8.6 GBfits~29 tok/s
Q8_0 pickLossless11 GBfits~23 tok/s
FP16 / BF16 Lossless19 GBoffload~4.1 tok/s

Speed is a bandwidth-based estimate for generating text (384 GB/s); prompt processing and long contexts are slower.

Measured speed on the Radeon RX 6700 XT

No community reports for this pair yet.

Report your own speed →

Run Cola2 on the Radeon RX 6700 XT

Transformers · Chat with the pipeline APIdocs
from transformers import pipeline

pipe = pipeline("text-generation", model="ByteDance-Seed/Cola2", dtype="auto", device_map="auto")
messages = [{"role": "user", "content": "Explain self-hosting in one paragraph."}]
out = pipe(messages, max_new_tokens=512)
print(out[0]["generated_text"][-1]["content"])

Chat models that fit the Radeon RX 6700 XT comfortably

All that fit →
ModelParamsBest quantMemorySpeedLicence
8.2BQ8_0≈10 GBFits in memory~24 tok/sCommercial OK
8BQ8_0≈9.8 GBFits in memory~25 tok/sConditions
7.6BQ8_0≈8.9 GBFits in memory~26 tok/sCommercial OK
8BQ8_0≈9.8 GBFits in memory~25 tok/sConditions
9.2BQ6_K≈11 GBFits in memory~28 tok/sConditions
ornith-ai logoOrnith-1.5-9Bornith-ai
9.7BQ8_0≈11 GBFits in memory~21 tok/sCommercial OK

Other GPUs that run Cola2

Frequently asked questions

Can the Radeon RX 6700 XT run Cola2?
Yes. The Radeon RX 6700 XT runs Cola2 fully in VRAM at Q8_0 (about 11 GB of 12 GB), at roughly 23 tokens/s.
How fast does Cola2 run on the Radeon RX 6700 XT?
An estimate from memory bandwidth gives about 23 tokens/s at Q8_0.
Which quantisation of Cola2 should I use on the Radeon RX 6700 XT?
Q8_0: the largest one that fits. Expected quality: lossless compared with the full-precision weights.

Memory and speed are estimates from the model's size and the card's memory bandwidth; real numbers depend on the runtime, drivers and context length. Community figures are medians of user reports.