本機 Gemma 4 e2b vs 雲端 Gemma 4 31B：100 個任務跑完，誰真的比較快、比較準？Local Gemma 4 e2b vs Cloud Gemma 4 31B: After 100 Tasks, Which Is Actually Faster and More Accurate?

I'd been curious about local vs cloud inference for a while — not the spec-sheet differences, but what actually happens when you throw 100 real tasks at each: time, accuracy, the real bill. This run pits three models against each other: local gemma4:e2b, cloud's older-generation gemma-3n-e2b-it, and cloud's newer gemma-4-31b-it (the latter two via NVIDIA NIM's free tier) — 100 real tasks, measured.

Why these three models?

The two-model pairing first:

1. A cleaner same-family comparison: local gemma4:e2b (5.1B) and cloud google/gemma-4-31b-it (NVIDIA NIM free tier) are the same Gemma 4 family — only the execution environment changes (local vs cloud), isolating "where it runs" instead of confounding the comparison with a completely different architecture.
2. Reflects a realistic scenario: cloud providers tend to default to bigger models than a laptop can run. 31B vs 5.1B isn't an artificially unfair pairing — it's what most people actually run into: what fits on your machine and what's available in the cloud are simply different sizes.

The third model came later: there's no NIM equivalent of the local gemma4:e2b. NIM's catalog has a similarly-named gemma-3n-e2b-it, but that's a completely different generation (Gemma 3n, not Gemma 4). After confirming that, it got pulled in as a third route anyway — not to force a comparison that doesn't exist, but to add a "generation" dimension: an older, smaller cloud model sitting alongside local and the newer, bigger cloud model. That lets "is it slow because it's cloud" and "is it slow because it's big and new" get separated instead of tangled together.

How do local and cloud actually run under the hood?

Before the numbers, it's worth understanding what each side is actually doing — that's what explains the speed gap later, not magic.

Local gemma4:e2b (Ollama): Ollama runs on llama.cpp under the hood, and its model file format is GGUF — training-time weights (e.g. 32-bit float) get compressed into lower-bit integer representations, which shrinks file size, speeds up loading, and cuts inference memory, at the cost of some precision. Ollama's default quantization level is Q4_K_M (4-bit). gemma4:e2b is also a "thinking model": before writing its answer, it reasons in a separate internal channel (thinking), which draws from the same token budget as the final answer (response / content) but is accounted separately — this is exactly the root cause of the pitfall below.

Cloud gemma-4-31b-it (NVIDIA NIM): NIM isn't a model — it's NVIDIA's containerized inference microservice, packaging GPU-optimized engines (TensorRT-LLM, vLLM, Triton Inference Server — with KV-cache, paged attention, batching, FP8, etc. already tuned) behind an OpenAI-compatible REST API. Every request you send runs on NVIDIA's GPU cluster through that optimized engine; the free tier means you're sharing a queue with other users, with no dedicated capacity guarantee — a reasonable explanation for the tail latency measured here.

This explains the pitfall we hit: gemma4:e2b's thinking channel defaults to on. If num_predict (the local-side output cap) is set too low, the reasoning process consumes the entire budget and the final response field comes back empty — even though eval_count shows tokens were generated. Getting a real answer requires explicitly setting think: false in the request.

How were the 100 tasks designed?

Three categories, modeled on real usage:

Three runners:

• Local: gemma4:e2b, serial (one at a time)
• Cloud (older generation): google/gemma-3n-e2b-it (NVIDIA NIM free tier), concurrency 4
• Cloud (newer generation): google/gemma-4-31b-it (NVIDIA NIM free tier), concurrency 4

One real pitfall worth recording: gemma4:e2b is a thinking model. The first run didn't set think: false, and all 40 summaries came back as empty strings — the num_predict budget got consumed by the internal reasoning trace before any answer was written. Adding think: false and rerunning got 100/100 real content.

Who's actually faster?

First surprise: local beat the newer cloud model, even with cloud running 4-way concurrency. The intuitive assumption is that local takes hours and an API takes minutes — the measured reality is the opposite, because this comparison pits a queued free tier serving a 31B model against a purpose-optimized 5.1B local model.

Second, bigger surprise: the fastest cloud model isn't the new one — it's the old one. gemma-3n-e2b-it (older generation, small size) finished the batch in just 47.3 seconds — faster than local, with completely consistent latency and zero tail. That means "cloud is slow" was never actually true; what actually slows things down is "big model + free-tier queueing." Control for model size, and cloud can beat local outright.

Cloud 31B's tail latency is a real reliability concern: 10% of tasks took over 10 seconds, with the slowest at 47 seconds. Local and the older-generation cloud model both had zero tail latency. If your use case cares about p99 latency (real-time interaction, say), that number matters more than the average.

How does the bill work out?

• Local electricity: total inference time was only 102.3 seconds. At a documented estimated draw of 20W (not a live measurement — this machine has no sudo access to run powermetrics, disclosed here as an estimate), the cost comes to about NT$0.0026, rounding to zero.
• Cloud cost (both routes): US$0.00. This run used NVIDIA NIM's free tier (ADR-016's scoped exception) for both gemma-3n-e2b-it and gemma-4-31b-it, so the actual bill is zero either way. This does not represent real paid-tier pricing — NVIDIA's official rate for either model wasn't verified in this run, and no number is guessed here.

For 100 tasks, the money gap is NT$0 on both sides. I'd guessed the gap would land around "a lunch box's worth" — the measured reality doesn't even reach that; both sides are free. At this scale, "which is cheaper" was never a meaningful question to begin with.

Classification accuracy: newer and bigger isn't automatically better

Summary and translation are open-ended generation — all three sides read as coherent and faithful to the source on inspection, with differences in style rather than correctness, so they don't suit a hard score. Here are the local vs. newer-cloud examples so you can judge for yourself:

Summary example (S001)

• Local: "Supplier's chip lead time slipped two weeks, need to warn the client early; payment terms moved to net-45, finance needs to rerun cash flow; next check-in set for Thursday."
• Cloud 31B: "Q3 chip delivery delayed two weeks, client needs notice; payment terms changed to net-45 requiring a cash flow re-forecast; next meeting set for Thursday."

Translation example (T001)

• Local: "在本地運行大型語言模型是以犧牲便利性來換取控制權。你放棄了雲端 API 近乎即時的擴展能力，但你換來的是對你資料的完全所有權，以及可預測、重複性的成本，而不是隨著使用量而不斷增加的帳單。"
• Cloud 31B: "在本地端執行大型語言模型是以便利性換取控制權。雖然你放棄了雲端 API 近乎即時的擴展能力，但你獲得了對數據的完全所有權，且成本固定可預測，而非隨著使用量增加而成長的帳單。"

Classification is different — all 40 items have known ground truth, so a real accuracy number is possible:

The same reversal as speed: the most accurate model isn't the newest or biggest — it's the older-generation gemma-3n-e2b-it. Local's common failure mode was mislabeling "personal" as "work," plus inventing a fifth category ("social") outside the four given options; gemma-3n-e2b-it only missed 3, the same kind of "personal" misjudgment but far less often. "Newer and bigger means more accurate" doesn't hold on this classification task — the local small model is fine for open-ended generation but clearly weaker on tasks needing precise judgment within strict option boundaries; and picking a cloud model isn't "pick the newer one," it's "test both and look at the actual numbers."

What does the wider community find on local vs cloud?

Sources:

• Local LLMs vs Cloud APIs: 2026 Total Cost of Ownership Analysis — SitePoint
• Local LLM vs Cloud LLM — The Latency Truth Nobody Talks About — Medium
• Self-Hosting LLMs vs Cloud APIs: Cost, Speed, Privacy 2026 — Effloow
• Thinking mode produces empty responses — GitHub Issue #16583, ollama/ollama

This isn't the only experiment on local vs cloud economics. Two threads of external evidence are worth cross-checking against:

On our own thinking-model pitfall — someone else already filed it. GitHub issue #16583 on ollama/ollama documents the exact same bug, on the exact same model family: Gemma 4 (E4B and 12B variants) with thinking enabled, num_predict consumed entirely by the reasoning trace, empty content field. The reporter's fix matches ours — think: false — and with it disabled, they measured ~0.5s latency on an RTX 3090 with 6/6 accuracy on their task. This isn't a one-off fluke on this machine; it's a known characteristic of Gemma 4's thinking mode.

On general local-vs-cloud economics, our result is the exception, not the rule — and that's worth being honest about. 2026 community benchmarks generally find: time-to-first-token is typically 50–200ms locally vs 200–800ms for cloud APIs (favoring local, consistent with our result); but on total cost of ownership, a solo developer's local hardware investment typically breaks even against API spend around 7 months in, and medium-volume teams around 18–24 months — meaning at low, occasional usage, cloud APIs usually win on total cost, because you never pay for idle hardware. Our 100-task run is too small a sample to hit that break-even math; it only shows the per-task marginal cost, where local's marginal cost is close to zero. On quality, the general finding is that the best local open models still lag frontier cloud models by roughly 10–20% on complex reasoning — in the same direction and rough magnitude as our own classification gap (72.5% vs 87.5%, a 15-point spread).

The takeaway: our speed result is real but shouldn't be over-generalized — it's specific to this pairing (a small, well-optimized local model vs a large model on a queued free tier). The quality gap and the thinking-model pitfall, on the other hand, match the wider pattern other people have already documented.

Mechanism analysis: does the theory hold up?

Sources:

• Gemma 3n model overview — Google AI for Developers
• Understanding Gemma 3n: How MatFormer Gives You Many Models in One — Hugging Face
• Inverse Scaling: When Bigger Isn't Better (arXiv 2306.09479)

Three models, three different profiles — not random noise. Each has an architectural or training explanation, though the explanatory strength isn't the same across all three findings, so this breaks it down honestly.

Speed gap: strongest explanation, matches architecture almost exactly. gemma-3n-e2b-it is fast not because it's "old" — it's because it uses a MatFormer (Matryoshka Transformer) nested design. E2B isn't an independently trained small model; it's a nested sub-network jointly trained alongside the larger E4B, activating only the inner parameters at inference, paired with Per-Layer Embedding (PLE) that keeps most embedding parameters out of model memory and loads them on demand. Google's own documentation states it plainly: E2B is labeled 5B parameters, but the effective parameter load at runtime is only about 1.9 billion. That means every request costs inherently less compute on NVIDIA's GPUs than a conventional dense model of the same nominal label — faster to serve, less queueing pressure — which lines up exactly with why it was the fastest of the three with zero tail latency. gemma-4-31b-it, by contrast, is a conventional dense architecture where every token across all 31B parameters gets computed, so free-tier queueing naturally drags longer. On speed, theory and measurement line up cleanly — no surprise here.

Accuracy gap: theory-backed, but not as simple as "old beats new." It's easy to misread this as "older models are more accurate" — the more precise framing is "format compliance doesn't simply track parameter count." Research has found that adherence to strict output formats (like "answer with exactly one of these four words") can vary by 40-60 percentage points between models, and that gap is driven mainly by whether a model was specifically fine-tuned for format constraints, not by model size. Separately, the Inverse Scaling Prize research documents a counterintuitive but verified phenomenon: as scale increases, models sometimes lean more heavily on memorized internal priors, overriding precise compliance with narrow instructions (like "pick exactly one of these four categories") — one of four identified causes. gemma-3n-e2b-it's MatFormer joint-training may let its small inference path inherit quality learned during co-training with the larger model, which could explain why it did unexpectedly well on this narrow, forced-choice task — but the same research stresses that most inverse-scaling cases eventually reverse with further scale (a U-shaped curve): scale isn't useless, its relationship with task type just isn't monotonic.

Local's lower accuracy: quantization isn't the main cause, model size is. gemma4:e2b runs Q4_K_M 4-bit quantization. General industry measurements of this quantization level typically show only 1-3% accuracy loss on benchmarks like MMLU — far smaller than the 15-20 point gap measured here. Honestly: quantization alone doesn't explain this gap. The more defensible explanation is the combination of the local model's raw 5.1B size and the lack of specialized tuning for this exact forced-choice format — not quantization taking the blame on its own.

A limitation worth admitting honestly: the classification set is only 40 items, and gemma-3n-e2b-it beat gemma-4-31b-it by just 2 questions (37 vs 35). The direction is theory-consistent, but the sample size isn't large enough to call this a stable statistical law — it's better treated as a signal worth watching, not an established rule.

When should you pick local, and when should you pick cloud?

Verdict

Waveform: INTERFERENCE

This isn't "local wins" or "cloud wins" — it isn't even something the local-vs-cloud axis alone can decide. Two variables came apart in this run: location (local/cloud) and generation/size (older-small/newer-big). On location, local beat the newer cloud model. On generation, cloud's older small model beat local outright, on both speed and accuracy. The second variable is what actually decided the outcome, not the first.

What changed for me after these 100 tasks: daily note-class tasks (summaries, translation — most of my volume) stay local, for free, fast-enough, data-never-leaves-the-machine reasons. Tasks needing strict classification or decisions move to the cloud, but which cloud model gets picked now means testing more than one rather than defaulting to "newer and bigger." The free tier's tail latency is a reminder: if you're putting a cloud API in a latency-sensitive path, check the paid tier's actual SLA first — these numbers don't transfer directly.

See gemma4 on Ollama Library →

NVIDIA NIM free tier →