selfhost.directory

The index of self-hostable projects — ranked, categorized and served with live updates.
Own your data and stop giving it to big tech.

google logo

gemma-4-26B-A4B-it

google·Instruct·released 11 Mar 2026

Apache 2.025.8BA4BMoE256K ctx ≈18 GB
Your GPU
ollama run gemma4:26b
All 6 runners ↓

Gemma is a family of open models built by Google DeepMind. Gemma 4 models are multimodal, handling text and image input (with audio supported on E2B, E4B, and 12B) and generating text output. This release includes open-weights models in both pre-trained and instruction-tuned variants.

How much VRAM does gemma-4-26B-A4B-it need?

At Q4_K_M gemma-4-26B-A4B-it needs about 18 GB with an 8K context (18 GB at Q4, 53 GB at full precision). Pick your GPU to see what fits and how fast it runs.

--vram
any
--gpus
--quant

Needs: ≈18 GB · Q4_K_M · 8K context — Pick a GPU to check the fit.

Change quant, context or KV cache to compare
Advanced: context 8K · KV cache 16-bit
Context
KV cache

Smallest GPU for each quant and context (KV cache 16-bit)

Quantisation8K16K32K64K128K256K
Q3_K_M
Q4_K_M
Q5_K_M
Q6_K
Q8_0
FP16

≤16 GB card24–32 GB48–96 GBmore

QuantisationQualityWeightsKV cacheTotal
Q2_KHeavy lossSmallest, noticeably worse qualityHeavy loss7.9 GB est.1.9 GB≈11 GB
Q3_K_MNoticeable lossSmall, some quality lossNoticeable loss12 GB est.1.9 GB≈15 GB
Q4_K_MrecommendedSmall lossRecommended default for local useSmall loss15 GB est.1.9 GB≈18 GB
Q5_K_MNear-losslessNear-lossless for most tasksNear-lossless17 GB est.1.9 GB≈20 GB
Q6_KNear-losslessPractically losslessNear-lossless20 GB est.1.9 GB≈23 GB
Q8_0LosslessLossless in practice, twice the size of Q4Lossless25 GB file1.9 GB≈29 GB
FP16 / BF16LosslessFull precision, as releasedLossless47 GB file1.9 GB≈52 GB

Estimates for 8K tokens of context: weights + KV cache + runtime overhead. All experts stay in memory, but only the active ones are read per token. Speed assumes memory-bound decoding at your GPU’s bandwidth. Pick a GPU above and the fit column checks every row against it. Weights marked “file” use the real GGUF download size; “est.” rows are computed from the parameter count.

How to pick a quant

Take the biggest quant that fits with room left for context. Q4_K_M is the usual sweet spot. Below Q4 quality drops fast, above Q6 you mostly pay memory for nothing. Big models (70B+, large MoE) tolerate low quants better than small ones.

  • LosslessF16, Q8_0
  • Near-losslessQ6_K, Q5_K_M
  • Small lossQ5_K_S, Q4_K_M, IQ4_NL, IQ4_XS, Q4_K_S
  • Noticeable lossQ4_0, Q3_K_L, Q3_K_M, IQ3_M
  • Heavy lossIQ3_XXS, Q3_K_S, IQ2_M, Q2_K

Typical quality loss, not measured per model. Tiers come from KL divergence against FP16 on Llama 3 8B: llama.cpp perplexity README: LLaMA 3 8B scoreboard (KLD and ΔPPL vs FP16, Wikitext-2); llama.cpp quantize tool: ΔPPL per quant type @ Llama-3-8B

How much context fits?

Model quant
KV cache

At 8K context, Q4_K_M needs ≈18 GB(15 GB weights + 1.9 GB KV cache + 1.2 GB overhead). Pick a GPU above to see how much context fits.

0204060801K2K4K8K16K32K64K128K256KGB

Weights + overheadKV cache

Can the Tesla P40 run gemma-4-26B-A4B-it? →

Speed vs quality per quant

Smaller quants run faster and fit in less memory, but lose quality.

Q2_KQ3_K_MQ4_K_MQ5_K_MQ6_KQ8_0F16

Fits in VRAMTight fitCPU offload✕ won't runQuality is typical KL divergence vs FP16, not measured for this model.

Real-world speed of gemma-4-26B-A4B-it

Tokens per second measured by people running it at home. A GPU shows a median once 3 reports agree on the same setup. Reports far from our estimate wait for a review. Measure yours with llama-bench →

No medians yet. Be one of the first 3 to report a setup.

All reports

Run gemma-4-26B-A4B-it locally

--quantQ4_K_Mdefault; pick your GPU to tailor itunsloth/gemma-4-26B-A4B-it-GGUF

From the Ollama librarydocs
ollama run gemma4:26b

The library build picks its own default quantisation.

UD-Q4_K_M straight from Hugging Facedocs
ollama run hf.co/unsloth/gemma-4-26B-A4B-it-GGUF:UD-Q4_K_M

Install Ollama

OpenAI-compatible server with a web UIdocs
llama-server -hf unsloth/gemma-4-26B-A4B-it-GGUF:UD-Q4_K_M

Downloads the GGUF on first run. Installed through llama.app? The same flags work with `llama serve`.

Install llama.cpp

Download with the lms CLIdocs
lms get unsloth/gemma-4-26B-A4B-it-GGUF@UD-Q4_K_M

Then load it from the LM Studio app or with `lms load`.

Install LM Studio

Serve an OpenAI-compatible APIdocs
vllm serve google/gemma-4-26B-A4B-it
Smaller FP8 builddocs
vllm serve RedHatAI/gemma-4-26B-A4B-it-FP8-dynamic
Docker Compose (NVIDIA)docs
yaml
services:
  vllm:
    image: vllm/vllm-openai:latest
    command: google/gemma-4-26B-A4B-it
    ports:
      - "8000:8000"
    ipc: host
    volumes:
      - ~/.cache/huggingface:/root/.cache/huggingface
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

API on port 8000.

Install vLLM

Serve an OpenAI-compatible APIdocs
sglang serve google/gemma-4-26B-A4B-it --host 0.0.0.0 --port 30000
Docker Compose (NVIDIA)docs
yaml
services:
  sglang:
    image: lmsysorg/sglang:latest
    command: sglang serve google/gemma-4-26B-A4B-it --host 0.0.0.0 --port 30000
    ports:
      - "30000:30000"
    ipc: host
    volumes:
      - ~/.cache/huggingface:/root/.cache/huggingface
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

API on port 30000.

Install SGLang

Ask about an imagedocs
python
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="google/gemma-4-26B-A4B-it", dtype="auto", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg"},
            {"type": "text", "text": "Describe this image."},
        ],
    }
]
out = pipe(text=messages, max_new_tokens=256)
print(out[0]["generated_text"][-1]["content"])

Install Transformers

Benchmarks

BenchmarkScoreRankSourceDate
MMLU-Pro82.6%#31 / 409 self-reportedApr 2026
GPQA Diamond82.3%#49 / 104 self-reportedApr 2026
Humanity's Last Exam17.2%#54 / 63 self-reportedApr 2026
MMMU-Pro73.8%#15 / 25 self-reportedMay 2026
ParseBench58.5%#15 / 38 independentApr 2026

Self-reported scores come from the model card and are not independently verified. Rank is among all models with a score on that benchmark.

Quantised versions and downloads

RepositoryFormatQuantSizeDownloads
nvidia/Gemma-4-26B-A4B-NVFP4SafetensorsNVFP4—1.6M
RedHatAI/gemma-4-26B-A4B-it-NVFP4SafetensorsNVFP4—467.7K
RedHatAI/gemma-4-26B-A4B-it-FP8-dynamicSafetensorsFP8—1.4M
unsloth/gemma-4-26B-A4B-it-GGUFGGUFUD-IQ2_XXS9.9 GB1.8M
unsloth/gemma-4-26B-A4B-it-GGUFGGUFUD-IQ2_M10 GB1.8M
unsloth/gemma-4-26B-A4B-it-GGUFGGUFUD-Q2_K_XL11 GB1.8M
unsloth/gemma-4-26B-A4B-it-GGUFGGUFUD-IQ3_S11 GB1.8M
unsloth/gemma-4-26B-A4B-it-GGUFGGUFUD-IQ3_XXS11 GB1.8M
unsloth/gemma-4-26B-A4B-it-GGUFGGUFUD-Q3_K_M13 GB1.8M
unsloth/gemma-4-26B-A4B-it-GGUFGGUFUD-Q3_K_XL13 GB1.8M

About gemma-4-26B-A4B-it

Gemma is a family of open models built by Google DeepMind. Gemma 4 models are multimodal, handling text and image input (with audio supported on E2B, E4B, and 12B) and generating text output. This release includes open-weights models in both pre-trained and instruction-tuned variants. Gemma 4 features a context window of up to 256K tokens and maintains multilingual support in over 140 languages. Featuring both Dense and Mixture-of-Experts (MoE) architectures, Gemma 4 is well-suited for tasks like text generation, coding, and reasoning. The models are available in five distinct sizes: E2B, E4B, 12B, 26B A4B, and 31B. Their diverse sizes make them deployable in environments ranging from high-end phones to laptops and servers, democratizing access to state-of-the-art AI. Gemma 4 introduces key capability and architectural advancements: • Reasoning – All models in the family are designed as highly capable reasoners, with configurable thinking modes. • Extended Multimodalities – Processes Text, Image with variable aspect ratio and resolution support (all models), Video, and Audio (featured natively on the E2B, E4B, and 12B models). • Diverse & Efficient Architectures – Offers Dense and Mixture-of-Experts (MoE) variants of different sizes for scalable deployment. • Optimized for On-Device – Smaller models are specifically designed for efficient local execution on laptops and mobile devices.

Full model card on Hugging Face →

Other Gemma 4 models

Whole family →
google logo5.1BCommercial OK≈3.8 GB↓ 118K
google logo5.1BCommercial OK≈3.8 GB↓ 3.2M
google logo8BCommercial OK≈5.9 GB↓ 474.5K
google logo8BCommercial OK≈5.9 GB↓ 4.2M
google logo12BCommercial OK≈11 GB↓ 52.8K
google logo12BCommercial OK≈11 GB↓ 1.8M
google logo26.5BA4BCommercial OK≈18 GB↓ 72.3K
google logo31.3BCommercial OK≈27 GB↓ 7.5M
google logo32.7BCommercial OK≈27 GB↓ 61K

Alternatives to gemma-4-26B-A4B-it

All alternatives →
Qwen logo

Qwen3.8-27B

Qwen·Vision

≈19 GB

Following the widespread community adoption of the Qwen3.5 and Qwen3.6 series, we are pleased to introduce Qwen3.8, the most capable generation in the Qwen open-model family to date.

27.8B256K ctx Apache 2.0
7.8MAug 2026ollama
Qwen logo

Qwen3.6-27B

Qwen·Vision

≈19 GB

Following the February release of the Qwen3.5 series, we're pleased to share the first open-weight variant of Qwen3.6. Built on direct feedback from the community, Qwen3.6 prioritizes stability and real-world utility, offering developers a more intuitive, responsive, and genuinely productive coding…

27.8B256K ctx Apache 2.0
2.3MApr 2026ollama
google logo≈18 GB

DiffusionGemma is a generative model built by Google DeepMind. Based on the 26B A4B Mixture-of-Experts (MoE) Gemma 4 architecture, DiffusionGemma generates tokens using discrete diffusion. This open-weights model is multimodal, handling text, image, and video inputs to generate text output.

25.8BA4B256K ctx Apache 2.0
628KJun 2026gguf
Qwen logo

Qwen3.5-27B

Qwen·Vision

≈19 GB

Over recent months, we have intensified our focus on developing foundation models that deliver exceptional utility and performance.

27.8B256K ctx Apache 2.0
1.7MFeb 2026ollama
Qwen logo

Qwen3.6-35B-A3B

Qwen·Vision

≈22 GB

Following the February release of the Qwen3.5 series, we're pleased to share the first open-weight variant of Qwen3.6. Built on direct feedback from the community, Qwen3.6 prioritizes stability and real-world utility, offering developers a more intuitive, responsive, and genuinely productive coding…

36BA3B256K ctx Apache 2.0
4.4MApr 2026ollama
openbmb logo

MiniCPM-V-4.7-35B-A3B

openbmb·Vision

≈22 GB

GitHub · On-device results · Evaluation · Examples · Usage · Model weights

35.2BA3B256K ctx Apache 2.0
585.8KOct 2026gguf

gemma-4-26B-A4B-it FAQ

How much VRAM does gemma-4-26B-A4B-it need?
About 18 GB at Q4_K_M with an 8K context. Q8_0 needs about 29 GB and full precision about 53 GB. Longer contexts need more memory for the KV cache.
Can gemma-4-26B-A4B-it run on the Tesla P40?
Yes. It fits at Q5_K_M (about 20 GB), at an estimated 67 tokens/s. It fits comfortably on the Radeon RX 7900 XTX, Instinct MI50 32GB, RTX 5090.
Can I use gemma-4-26B-A4B-it commercially?
Yes. It is released under Apache 2.0. Permissive: use, modify and sell freely; keep the notice and licence text.
What is the context length of gemma-4-26B-A4B-it?
262,144 tokens (256K). The full window needs far more memory than an 8K context.
How do I run gemma-4-26B-A4B-it with Ollama or llama.cpp?
With Ollama: ollama run gemma4:26b. With llama.cpp: llama-server -hf unsloth/gemma-4-26B-A4B-it-GGUF:UD-Q4_K_M, which starts an OpenAI-compatible server.

Data from Hugging Face, the model card and public leaderboards. Memory and speed figures are estimates; real numbers depend on the runtime, context length and drivers. Always check the licence yourself before commercial use.