Can the RTX 3060 12GB run Cola2?
Cola2·8.8B·GeForce RTX 3060 12GB·12 GB VRAM
Saved in this browser, used on every model page.
Yes
Yes. The RTX 3060 12GB runs Cola2 fully in VRAM at Q8_0 (about 11 GB of 12 GB), at roughly 21 tokens/s.
- best quant
- Q8_0
- needs
- 11 GB
- usable
- 12 GB
- est. speed
- ~21 tok/s
Estimates for an 8K context; offload assumes 32 GB of system RAM.Try other contexts on the Cola2 page →
Every quantisation of Cola2 on the RTX 3060 12GB
| Quant | Memory | Fit | Speed |
|---|---|---|---|
| Q2_K | 4.4 GB | fits | ~69 tok/s |
| Q3_K_M | 5.8 GB | fits | ~46 tok/s |
| Q4_K_M | 6.8 GB | fits | ~37 tok/s |
| Q5_K_M | 7.7 GB | fits | ~32 tok/s |
| Q6_K | 8.6 GB | fits | ~27 tok/s |
| Q8_0 pick | 11 GB | fits | ~21 tok/s |
| FP16 / BF16 | 19 GB | offload | ~4 tok/s |
Speed is a bandwidth-based estimate for generating text (360 GB/s); prompt processing and long contexts are slower.
Measured speed on the RTX 3060 12GB
No community reports for this pair yet.
Report your own speed →Run Cola2 on the RTX 3060 12GB
Transformers · Chat with the pipeline APIdocs
from transformers import pipeline
pipe = pipeline("text-generation", model="ByteDance-Seed/Cola2", dtype="auto", device_map="auto")
messages = [{"role": "user", "content": "Explain self-hosting in one paragraph."}]
out = pipe(messages, max_new_tokens=512)
print(out[0]["generated_text"][-1]["content"])Chat models that fit the RTX 3060 12GB comfortably
All that fit →| Model | Params | Best quant | Memory | Speed |
|---|---|---|---|---|
| 8.2B | Q8_0 | ≈10 GBFits in memory | ~23 tok/s | |
| 8B | Q8_0 | ≈9.8 GBFits in memory | ~23 tok/s | |
| 7.6B | Q8_0 | ≈8.9 GBFits in memory | ~24 tok/s | |
| 8B | Q8_0 | ≈9.8 GBFits in memory | ~23 tok/s | |
| 9.2B | Q6_K | ≈11 GBFits in memory | ~26 tok/s | |
| 9.7B | Q8_0 | ≈11 GBFits in memory | ~19 tok/s |
Other GPUs that run Cola2
Frequently asked questions
- Can the RTX 3060 12GB run Cola2?
- Yes. The RTX 3060 12GB runs Cola2 fully in VRAM at Q8_0 (about 11 GB of 12 GB), at roughly 21 tokens/s.
- How fast does Cola2 run on the RTX 3060 12GB?
- An estimate from memory bandwidth gives about 21 tokens/s at Q8_0.
- Which quantisation of Cola2 should I use on the RTX 3060 12GB?
- Q8_0: the largest one that fits. Expected quality: lossless compared with the full-precision weights.
Memory and speed are estimates from the model's size and the card's memory bandwidth; real numbers depend on the runtime, drivers and context length. Community figures are medians of user reports.