Wiki

Google AI Edge Gallery

Google AI Edge Gallery

Google’s open-source, on-device showcase app for LLMs and multimodal models (google-ai-edge/gallery, Kotlin/Android + iOS) and the runtime stack under it — LiteRT-LM, LiteRT, and the now-deprecated MediaPipe GenAI API. This page is the code-level deep-dive: repository layering, bundle-format internals, the actual shipped feature/model set, and sourced failure modes. LLM inference on Android keeps the cross-runtime Android survey (LiteRT-LM as one row among Qualcomm Genie/GenieX, ONNX Runtime GenAI, Gemini Nano) and the ready-made-apps comparison; this page is what backs that row up.

Architecture: four repositories, one dependency chain

flowchart TB
    subgraph Gallery["google-ai-edge/gallery (Kotlin/Android)"]
        UI["UI screens"]
        DL["Model lifecycle<br/>ModelManagerViewModel, DownloadWorker"]
        SESSION["Session construction<br/>LlmChatModelHelper"]
    end
    subgraph LiteRTLM["google-ai-edge/LiteRT-LM"]
        LOADER["Bundle loader"]
        FACTORY["Backend factory"]
    end
    subgraph LiteRT["google-ai-edge/LiteRT"]
        CMAPI["Compiled Model API"]
        DISPATCH["Vendor dlopen dispatch"]
    end
    VENDOR["Vendor .so<br/>libLiteRtDispatch_Qualcomm.so, etc."]
    MEDIAPIPE["google-ai-edge/mediapipe<br/>tasks/genai — @Deprecated, deps = []"]

    UI --> DL --> SESSION
    SESSION -- "litertlm-android:0.11.0" --> LOADER
    LOADER --> FACTORY --> CMAPI --> DISPATCH --> VENDOR
    SESSION -.->|"zero dependency"| MEDIAPIPE
LayerOwnerWhere
UI, screens, chat stateGalleryui/**
Model lifecycle (allowlist fetch, download, resume, HF auth)Galleryworker/DownloadWorker.kt, ui/modelmanager/ModelManagerViewModel.kt
Bundle parsing + session constructionLiteRT-LM (litertlm-android AAR)runtime/util/litert_lm_loader.h, runtime/core/engine_advanced_impl.cc
Backend dispatch (CPU/GPU/NPU)LiteRT-LM → LiteRT Compiled Model APIruntime/executor/llm_litert_compiled_model_executor_factory.cc
Vendor NPU silicon bindingLiteRT, at runtime via dlopenlitert/runtime/dispatch/litert_dispatch.cc
Legacy, unused by GalleryMediaPipe GenAI (deprecated)mediapipe/tasks/java/.../genai/llminference/

Gallery depends only on LiteRT-LM (litertlm-android:0.11.0) — there is zero MediaPipe dependency in the app. A repo-wide search for "tasks-genai" / "com.google.mediapipe" returns nothing. A second, independent execution path exists for Android’s system AICore service (runtime/aicore/AICoreModelHelper.kt, on ML Kit’s genai-prompt) — see Generative UI on Android for that surface in depth; it is not part of the LiteRT-LM path this page covers.

The dependency is gone, but the naming and product copy have not caught up. A helper still called cleanUpMediapipeTaskErrorMessage (common/Utils.kt) runs on the error path of both model helpers, and allowlist entries still describe models as “ready for deployment on Android using the MediaPipe LLM Inference API” — user-facing text pointing at the deprecated runtime. Read “zero MediaPipe dependency” as a statement about the build graph, not about the absence of the string in the tree.

MediaPipe’s GenAI LlmInference API is deprecated at the source level, not just in docs, and Gallery never calls it. Every public class in the Java package carries @Deprecated with “Migrate to LiteRT LM instead.” A dated commit, 83fdad9b (2026-01-30), titled “Remove the CPU-only MediaPipe LLM Inference Engine \ Please use LiteRT LM instead,” deleted the entire mediapipe/tasks/cc/genai/inference/ tree, the iOS genai subtree, and the Python .task bundler — about 100 files. A later commit (80f4a5bb, 2026-02-25) restored LlmInference.java as a pointer only, already carrying @Deprecated on arrival; its BUILD target wires in deps = [], no native engine. Google’s docs corroborate without dating the cutoff: “The MediaPipe LLM Inference API … is now in maintenance-only mode. New features and optimizations will be focused on LiteRT-LM.” This settles, with a commit date, what LLM inference on Android already states in passing — LiteRT-LM as “successor to the now-maintenance-only MediaPipe LLM Inference API.”

One build-consistency risk worth flagging: LiteRT-LM’s Bazel WORKSPACE and its CMake build pin two different LiteRT commits (0a033900... vs. fb16353a..., the latter dated “Updated on 2026-03-24”) — do not assume “LiteRT-LM tracks LiteRT main.”

Bundle formats: .task vs. .litertlm

.task is the older, MediaPipe-era format: a plain ZIP (detected by "PK" header bytes), a filename→byte-range map, no metadata proto, no external-weights section, no embedding section. There is no producer for it left in either repo — the bundler script was deleted in the same 83fdad9b commit above and now 404s. It remains readable only for backward compatibility with pre-2026 artifacts already in the wild.

.litertlm is a FlatBuffer-defined, sectioned container (magic bytes "LITERTLM", versioned in code as 1.6.0), structurally unrelated to a zip. Its root table holds system_metadata and a list of SectionObjects typed by AnySectionDataType: GenericBinaryData, TFLiteModel, SP_Tokenizer, LlmMetadataProto, HF_Tokenizer_Zlib, TFLiteWeights, EmbeddingMetadataProto, among others. In practice one bundle packages the LLM config (a protobuf), a tokenizer (SentencePiece binary or zlib-compressed HF tokenizer JSON), one or more TFLite graphs (prefill/decode/embedder/vision/audio adapters), optionally external weights, and optionally embedding metadata. Produced by a real CLI (litert-lm-builder) and consumed by LitertLmLoader, which mmaps sections on demand.

.litertlm is the only format that can carry vision/audio adapters, LoRA, external weights, or an embedding section — the repository’s own test fixtures make this concrete: exactly two .task fixtures exist and both are plain-text-only, while every fixture touching vision, audio, LoRA, or multi-prefill is .litertlm-only. No document states outright why one supersedes the other; this is reconstructed from history and fixtures, not from a changelog line.

Features shipped

The canonical feature list is object BuiltInTaskId:

Feature (UI label)Runtime mechanism
AI ChatLiteRT-LM Engine/Conversation, text-only
Ask Imagesame Engine, supportImage = true; vision pinned to GPU by the allowlist
Audio Scribesame Engine, supportAudio = true; audio backend hard-pinned to CPU (“must be CPU for Gemma 3n”)
Prompt Labsame Engine, single-turn, no conversation history
Agent Skillsreal tool/function calling; a ToolProvider list flows into ConversationConfig(tools = ...). Four tool backends live in ToolDispatcher.kt, of which the Model Context Protocol (io.modelcontextprotocol:kotlin-sdk, RunMcpTool.kt) is one — not a requirement of the mechanism as a whole
Mobile Actions, Tiny Gardenthe same tool-calling path (not necessarily the MCP backend specifically), running a FunctionGemma-270M fine-tune each

“Function calling” in this stack is named Agent Skills, and it is genuine structured tool-calling — but correction: MCP is one tool backend among several, not a requirement of Agent Skills as a whole. LiteRT-LM itself ships a plain reflection-based Kotlin tool API with no MCP dependency at all — @Tool/@ToolParam-annotated methods on a ToolSet interface, wrapped via tool() into a ToolProvider (kotlin/java/com/google/ai/edge/litertlm/Tool.kt, Config.kt). Gallery’s own ToolDefinition.kt (interface ToolDefinition : ToolSet) uses this native path for most tools; MCP shows up only inside one specific tool implementation (RunMcpTool.kt) that exposes a single generic runMcpTool(toolName, input) function and internally dispatches to an MCP server. MCP is Gallery’s own layered choice for one tool, not something the underlying library requires. See on-device-agent-tool-loops for the fuller tool-calling-API-surface picture, including that MediaPipe’s older LlmInference API never had any tool-calling surface at all, at any point in its source history — a full grep of its Java tree for tool/function returns only an unrelated java.util.function.Function lambda usage.

There is no RAG feature anywhere in this codebase — no embedding model, no vector store, no retrieval step; a full-tree search for rag, embed, vector, gecko, objectbox, sqlite returns zero matches. This is worth stating precisely because the format isn’t the limiting factor: .litertlm already reserves an EmbeddingMetadataProto section and LitertLmLoader already exposes GetEmbeddingMetadata() — the runtime supports embedding models, the app simply ships no feature that uses one.

Thinking mode is real and traceable: a distinct THOUGHT_CHANNEL in the streaming callback, gated per-model by the allowlist’s capabilities array (e.g. Gemma-4-E2B lists ["llm_thinking", "speculative_decoding"]).

The app self-reports benchmark stats through a dedicated harness, not a marketing claim. BenchmarkViewModel.runBenchmark() calls LiteRT-LM’s own exported benchmark() function directly and serializes the result into an LlmBenchmarkStats protobuf with fields prefillSpeed/decodeSpeed/ timeToFirstToken — the same metric names as Google’s published docs table below, so the in-app number measures the identical quantities on the user’s own device.

Models: the live allowlist

The app fetches a version-pinned allowlist at runtime (model_allowlists/${VERSION_NAME}.json), not the stale repo-root model_allowlist.json. Nine entries, all litert-community/ or google/ Hugging Face repos:

ModelHF sourceQuantization (filename only)SizeContext
Gemma-4-E2B-itlitert-community/gemma-4-E2B-it-litert-lmnot stated2.59 GB32,000
Gemma-4-E4B-itlitert-community/gemma-4-E4B-it-litert-lmnot stated3.66 GB32,000
Gemma-3n-E2B-itgoogle/gemma-3n-E2B-it-litert-lmint43.66 GB4,096
Gemma-3n-E4B-itgoogle/gemma-3n-E4B-it-litert-lmint44.92 GB4,096
Gemma3-1B-ITlitert-community/Gemma3-1B-ITint4584 MB1,024
Qwen2.5-1.5B-Instructlitert-community/Qwen2.5-1.5B-Instructq81.60 GB4,096
DeepSeek-R1-Distill-Qwen-1.5Blitert-community/DeepSeek-R1-Distill-Qwen-1.5Bq81.83 GB4,096
TinyGarden-270Mlitert-community/functiongemma-270m-ft-tiny-gardenq8289 MB1,024
MobileActions-270Mlitert-community/functiongemma-270m-ft-mobile-actionsq8289 MB1,024

Per-channel vs. per-tensor quantization granularity is absent from every entry across all 13 historical allowlist versions checked — only inferable from the filename suffix (int4, q8), never a structured field.

Hammer is not a Gallery model. It surfaces only as a conversion target in Google’s separate PyTorch-to-LiteRT tooling repo (litert-torch); the upstream family is third-party (MadeAgents/Hammer2.1). Google’s toolchain can convert it — Gallery does not ship it. Gemma 3n comes from Google’s own google/ HF org; everything else, including Gemma 4, comes through litert-community/ instead.

Performance: the full picture, not one number

LLM inference on Android cites 52 tok/s (Gemma 3n/4 E2B, GPU, Galaxy S26 Ultra) as the headline LiteRT-LM figure. That number is real but is one row of an eleven-row table, sourced from developers.google.com/edge/litert-lm/overview (v0.14.0, model Gemma-4-E2B, 2.58 GB):

PlatformBackendPrefill (tok/s)Decode (tok/s)TTFT (s)
Android (Galaxy S26 Ultra)CPU557471.8
Android (Galaxy S26 Ultra)GPU3,808520.3
iOS (iPhone 17 Pro)CPU532251.9
iOS (iPhone 17 Pro)GPU2,878560.3
Linux (Arm + RTX 4090)CPU260354.0
Linux (Arm + RTX 4090)GPU11,2341430.1
macOS (MacBook Pro M4 Max)CPU901421.1
macOS (MacBook Pro M4 Max)GPU7,8351600.1
Windows (Intel Lunar Lake)CPU435302.4
Windows (Intel Lunar Lake)GPU3,751480.3
IoT (Raspberry Pi 5, 16 GB)CPU13387.8

Independently corroborated, not just self-reported. A journalist benchmark (Beebom) ran the exact target stack — Gemma 4 E2B through this app on LiteRT-LM — across real current-generation devices and measured a Galaxy S26 Ultra GPU at TTFT 0.13 s, decode 48.55 tok/s, closely matching the 52 tok/s figure above from an independent methodology and device sample. The same test found an iPhone Air (A19 Pro) GPU at 51.28 tok/s, a Vivo X300 Pro (Dimensity 9500) GPU at only 16.45 tok/s, and a Pixel 10 Pro Fold (Tensor G5) CPU at 10.42 tok/s with a 1.65 s TTFT — a reminder the 52 tok/s figure is Snapdragon-flagship-specific, not universal even among current phones.

GPU decode ranges 8–160 tok/s across this table, not a single figure: a MacBook Pro M4 Max hits 160 tok/s (3× the Galaxy S26 Ultra number), an RTX 4090 hits 143, and a Raspberry Pi 5 floors at 8 tok/s CPU with a 7.8 s TTFT. Every row is CPU or GPU — there is no NPU row for Gemma 4 in Google’s own published table. One Gemma-4 NPU figure does exist off that table, on the litert-community/gemma-4-E2B-it-litert-lm model card’s IoT section: Qualcomm Dragonwing IQ8 (IQ-8275), NPU, prefill 3,747 / decode 31.7 / TTFT 0.3 s, from a separate NPU-converted 2,967 MB artifact at 4,096 context. Dragonwing IQ8 is embedded silicon, so the accurate statement is no Gemma-4 NPU number for a phone, not none at all. Older Gemma 3n NPU numbers also exist on Hugging Face model cards (Vivo X300 Pro NPU: prefill 1,671 tok/s / decode 28.4 tok/s).

The model card is in general a superset of the docs table — it adds Jetson Orin Nano and WebGPU rows, plus a speculative-decoding table where the same S26 Ultra reaches 91.7 tok/s decode against a 51.5 baseline. Google’s own numbers therefore refute reading 52 tok/s as that phone’s ceiling.

One more distinction worth keeping separate: the LiteRT-LM blog post’s “up to 76 tokens/sec on a MacBook Pro” is a WebGPU number, not the same measurement as the 160 tok/s native-Metal figure in the table above — do not collapse the two into one “Mac GPU decode” figure.

This elaborates, rather than contradicts, On-device neural accelerators (NPU / ANE / Hexagon)’s general argument that peak-TOPS/single-number framing is unreliable: here the absent NPU row is not a missing benchmark run but a structural gap (see below).

Where it breaks

RAM gating: soft, dismissable

The README states only an OS floor (“Android 12 and up, and iOS 17 and up”) — no RAM number. The real requirement lives per-model in the allowlist’s minDeviceMemoryInGb (6–12 GB). Enforcement is a dismissable warning with a “Proceed anyway” button, never a hard block. Tracker issue #423 (Android 8.1, 3 GB RAM) asked maintainers to document a minimum — unanswered as of this research.

NPU support: structurally limited, not just a rollout lag

The limit is a runtime one, not compile-time. LITERT_DISABLE_NPU is selected only for @platforms//os:ios (LiteRT-LM/runtime/executor/BUILD), so on Android the NPU executor ships by default. What actually gates it is dlopen of a per-vendor dispatch .so and resolution of the symbol "LiteRtDispatchGetApi"; when no such library is present the loader logs “No dispatch library found” and fails hard. A pure stub returning “unsupported” on every call does exist in the tree (litert_dispatch_dummy.cc), but nothing depends on it — a code search finds it referenced only by its own BUILD file — so it is not the fallback path. This is the concrete, vendor-fragmented failure mode that On-device neural accelerators (NPU / ANE / Hexagon)’s operator-coverage argument predicts in the abstract — a runtime having “an NPU path” and a given device’s NPU actually running are separate claims, and the gap is per-vendor, per-chip, and even per-driver-version:

  • MediaTek: not supported at all, and the app does not fail gracefully — Gallery #920 (Dimensity 8300 Ultra) crashes on NPU switch; maintainer confirms only Qualcomm Snapdragon SoCs are supported.
  • Qualcomm (the best-supported vendor) still fails on specific chip/driver combinations: LiteRT-LM #2226, a QNN system-library version mismatch on a Galaxy S25 Ultra despite using the documented QAIRT version; LiteRT-LM #1979, CPU/GPU/NPU all register and the vendor .so loads, yet inference still fails inside Qualcomm’s own QNN initialization.
  • Intel: VPU generations are not interchangeable — LiteRT-LM #2451, a .litertlm built for one VPU generation fails with “expects VPUX37XX, but model reports VPUX40XX” on a newer chip.
  • Samsung Exynos: fails at engine creation for every model tried — Gallery #882/#548, a raw INTERNAL error inside the compiled-model executor. A structurally similar error also appears on MediaTek (#557) — not an Exynos-exclusive error class.
  • Google’s own first-party Tensor silicon fails too, for a hardware reason: Gallery #1008, a base Pixel 10 (12 GB, Tensor G5) running the TPU-labeled Gemma-4-E2B hits the Tachyon TPU driver refusing to dma-buf-map the ~1.9 GB weight section a second time (once for prefill, once for decode) — errno=No space left on device. Works fine on the 16 GB Pixel 10 Pro, same SoC. The proximate cause traces to Gallery’s own code: ModelAllowlist.kt forcibly removes GPU as an accelerator option on Pixel 10 devices specifically, with no fallback, so the TPU path is the only one offered and it is the one that fails.
  • The per-SoC routing mechanism exists in schema but is unpopulated: Gallery #730/#888, a Snapdragon-8-Gen-5 device shows no NPU model to download at all; maintainer confirms socToModelFiles routing “is not fully supported in our external build process.”

Google’s own April 2026 NPU blog post presents Tensor, MediaTek, and Qualcomm as supported, flagging only Tensor’s ML SDK as experimental — the issue tracker shows a materially rougher picture: specific MediaTek chips outright unsupported, specific Qualcomm chips version-mismatching, and Intel VPU generations incompatible with each other.

Licensing: a dynamic, probe-first gate

The app does not hardcode which models need authentication — it probes: an unauthenticated download is attempted first, and only on failure does it check for a stored Hugging Face OAuth token (AppAuth flow against huggingface.co/oauth/...) or launch the auth UI. A separate, narrower gate additionally requires acknowledging Gemma’s terms of use, keyed by an explicit name set that excludes the Gemma 4 models — either a release- process drift or a deliberate distinction in the Gemma-4 repos’ actual HF gating status; the source alone doesn’t settle which. If a token is present but the server still 403s (terms not yet accepted on Hugging Face itself), the app opens a Custom Tab to the agreement and retries. Building from source requires registering one’s own HF OAuth app first — the public repo’s client credentials are literal placeholders.

Mid-range and older hardware: varied failures, not uniformly “low RAM”

  • Crash-on-launch on Huawei’s EMUI, independent of any model download (#599) — unresolved as of this research.
  • An OS-version regression, not raw RAM, killing a specific model: Pixel 6a (6 GB) on Android 17 Beta 4, Gemma-4-E4B crashes consistently having worked on Beta 3 (#701). The reporter attributes this to Android 17’s new per-app MemoryLimiter, but offers a competing hypothesis (Safer Dynamic Code Loading) in the same report and no maintainer has confirmed either — treat the cause as open, and the regression itself as the sourced fact.
  • A high-RAM device still failing, for a feature-specific reason: a 12 GB A19 Pro iPhone runs Gemma-4-E4B fine normally but crashes specifically with Thinking Mode enabled (#703) — a counterexample against treating every crash as a pure capacity problem.
  • Unbounded KV-cache growth in long chats (#856): crashes outright on GPU/NPU past a hidden token ceiling; loops instead on CPU. Root cause identified: LiteRT-LM’s raw KV-cache primitives (SaveCheckpoint, RewindToCheckpoint, ClearKVCache, DeleteTokensFromKvCache) exist with no coordinating eviction policy (LiteRT-LM #1878, open) — nothing in the runtime trims old turns as context grows, so any app built on it (Gallery included) must implement its own context-window eviction. See on-device-agent-tool-loops for the fuller KV-cache-reuse survey this traces from.
  • No graceful degradation on CPUs lacking modern SIMD: raw SIGILL on devices without ARM NEON/SVE, or an unexplained “OpenCL library was not found” (#543).

See also

  • LLM inference on Android — the cross-runtime Android survey this page’s LiteRT-LM row backs up in full detail; also carries the ready-made-apps comparison table Gallery appears in.
  • On-device neural accelerators (NPU / ANE / Hexagon) — the general operator-coverage/ vendor-fragmentation argument this page’s per-vendor NPU failures concretely instantiate, from a different (Gallery/LiteRT-LM) angle than Qualcomm Genie.
  • On-device ML runtimes (Core ML vs LiteRT) — the general Core ML vs. LiteRT + vendor- delegate layer LiteRT-LM’s Compiled Model API and dlopen dispatch sit on top of.
  • Generative UI on Android — the app’s second, independent execution path (Android AICore / ML Kit GenAI), not part of the LiteRT-LM stack above.
  • on-device-agent-tool-loops — reuses this page’s Gemma-4-E2B benchmark table for prefill/decode cost arithmetic, and carries the fuller KV-cache- reuse survey (llama.cpp, LiteRT-LM, MLC LLM, ExecuTorch) behind the #856/#1878 root-cause explanation above.

Sources

Trending Tags

personal