artificial intelligence
8 articles tagged artificial intelligence.
-
Transformer Key-Value Cache During Generation
A transformer KV cache keeps earlier attention keys and values so each generated token avoids rebuilding the entire prefix.
-
The AI Agent Tool-Calling Loop
How a model requests tools, how the host validates and runs them, and how results become context for the next model turn.
-
Speculative Decoding with Draft and Target Models
Speculative decoding uses a fast model to propose tokens and a slower model to verify several of them in parallel, reducing generation latency.
-
Transformer KV Cache During Text Generation
How transformers store attention keys and values to avoid recomputing prior tokens, and why longer contexts consume more inference memory.
-
Mixture-of-Experts Routing and Load Balancing
How MoE routers send tokens to a few expert networks while capacity limits and balancing losses prevent overloaded experts.
-
The Tool-Calling Loop in AI Agents
How an AI model requests a tool, how the runtime validates and runs it, and how the result becomes the next model input.
-
Speculative Decoding with Draft and Target Models
How a small model proposes token sequences for a large model to verify, reducing generation latency while preserving the large model’s distribution.
-
LLM Key-Value Caches During Autoregressive Generation
How cached attention keys and values eliminate repeated prefix work during generation, and why the speedup consumes substantial memory.