Learning about AutoAgents by benchmarking Claude Code vs. OpenCode
tl;dr ¶
- I’m extending AutoAgents so that it supports Z.ai, a frontier AI company
- Z.ai’s GLM-5 model might or might not compete with Claude Opus 4.5 at a very competitive price.
- I’m testing both the OpenCode CLI vs. Claude Code CLI, and Z.ai’s GLM-5 against
- Claude is considerably faster, but neglects testing. When I ask it about testing, it suggests how to run a test, and when running that command and the test runner executes 0 tests, it says “Right, the test doesn’t exist yet. Let me add it.” – but everything works fine.
- OpenStack writes tests, but not ones that actually contact the API. When I ask it to do that, it does, and the test fails without fixing it, and when I copy-paste the error code from running it, it gets fixed.
- Compare their commits here and here.
Alternative to OpenClaw in Rust ¶
Lately I’ve been playing around with OpenClaw. I really like the whole idea, and I don’t like the codebase. It uses Node.js and that whole ecosystem just smells like security incidents. I did some research into who is making AI frameworks in Rust, and I found two interesting projects:
- AutoAgents: a modern multi‑agent framework in Rust
for building intelligent, autonomous agents powered by Large Language Models (LLMs/SLMs) and
Ractor (inspired from Erlang’s
gen_server). I’ve been using actors with tokio extensively before, and I love Erlang. With AutoAgents’ 390 GitHub stars and Ractor’s 2.000+ stars, they’re popular but small.- Pros:
- Rig: a modular, scalable library for building LLM-powered applications. With 6.100+ stars on GitHub, this is the more mature choice. While more popular and mature, there’s no ReAct loop built into Rig, and I don’t really need its RAG support at this point.
Neither of these are actually OpenClaw alternatives. OpenClaw is an application, and both of these are libraries/frameworks for making AI agents.
There were runner-ups like swarms-rs, Anda, rs-graphs-llm, rigs, agentai; I’ll provide a shallow comparison of these,
A related article to the benchmarking part is builder.io’s OpenCode vs. Claude Code. Their takeaway: Claude Code is built for speed. OpenCode is built for thoroughness.
Before showing the benchmark prompt and evaluating the results, here’s a quick appendix:
- AutoAgents overview
- Rig overview
- The plan produced by Claude Code
- The plan produced by OpenCode + GLM-5
The benchmark prompts ¶
This directory contains a Rust project called AutoAgents. It currently lacks z.ai API integration. There’s a documentation website at https://docs.z.ai/api-reference/introduction and I’m using the GLM Coding Plan, which the documentation tells me needs to use the https://api.z.ai/api/coding/paas/v4 endpoint instead of the general endpoint. Analyse how AutoAgents deals with similar providers and help me extend AutoAgents to be able to connect with z.ai.
The follow-up prompt:
Assuming I have a Z.ai token, how do I run a simple test that this integration works? Since this is based on the DeepSeek API integration, please add a test for Z.ai in an equivalent way as how DeepSeek is tested.
Evaluating ¶
The Claude version is slightly more thorough (more string aliases, models test update,
debug-friendly smoke test), while the OpenCode version has a cleaner builder integration (passes
through more builder fields like base_url and top_p). But functionally they do the same thing:
wire up the same API endpoint, same default model (glm-5), same base URL, same feature capabilities.
I’m not confident to replace Claude Code with this stack yet: If it ain’t broken, don’t fix it!
But as a cost-effective alternative that appears on par, OpenStack + Z.ai seems very promising!
AutoAgents overview ¶
Architecture ¶
- Executor Layer: ReAct and basic executors for agentic loops; streaming response support
- Memory Layer: Configurable sliding-window memory with pluggable backends; shared memory between agents
- Agent Definitions: Both
DirectAgent(single-threaded) andActorAgent(distributed, using Ractor) - Tool System: Derive macros for type-safe tool definition; WASM sandbox runtime for sandboxed tool execution
- LLM Providers: Unified interface supporting OpenAI, Anthropic, local inference (Mistral-rs, LlamaCpp)
- Multi-Agent Orchestration: Typed pub/sub communication; environment management for agent-to-agent messaging
Context Management ¶
- Sliding-window memory maintains conversation history up to configurable token limits
- Memory hooks allow agents to reflect on their own execution and update internal state
- Extensible memory backends enable plugging in vector stores, databases, or external retrieval systems
- Shared memory blocks across multiple agents in same environment
Distributed Capabilities ¶
- ActorAgent variant runs on Ractor, enabling distributed agent instances across nodes
- Built-in telemetry with OpenTelemetry integration for observing agent behavior at scale
- Handles agent lifecycle, spawning, and supervision automatically
Strengths ¶
- Comprehensive tool ecosystem (toolkit crate includes pre-built integrations)
- Type-safe tool derivation macros reduce boilerplate
- Memory system is central, not bolted-on
- WASM tool sandbox prevents malicious/buggy tool code from crashing agent
- OpenTelemetry built-in; production observability out-of-the-box
- Active community (Liquidos AI team + growing contributors)
- Supports edge, cloud, and browser (WASM) targets
Weaknesses ¶
- Smaller ecosystem compared to Python LangChain/LangGraph
- Documentation is good but examples are still growing
- Ractor integration adds complexity for simple single-agent use cases
Best For ¶
- Production systems with 10+ persistent agents
- Complex tool calling workflows
- When sandboxing and memory safety matter
Rig overview ¶
Architecture ¶
- Unified Provider Interface: Abstract trait-based design; swap providers (OpenAI, Anthropic, Cohere, local) without code changes
- Agent Builder: Fluent API for configuring agents with preamble (system prompt), context documents, tools, and model parameters
- Chat/Completion/Prompt Traits: Agents implement multiple trait interfaces; same agent can be used for chat, completion, or prompt modes
- RAG Integration: First-class support for vector stores (MongoDB, in-memory, Qdrant via companion crates); dynamic context retrieval at prompt-time
- Tool System: Define tools as structs implementing the
Tooltrait; automatic JSON schema generation for structured tool calling - Streaming: Built-in streaming response support for real-time agent output
Context Management ¶
- Agent builder accepts static context documents; they are always included in the prompt
- Dynamic context retrieval via RAG: query-time embedding search pulls relevant documents into context
- Temperature, max_tokens, and other model parameters can be set per-agent or overridden per-request
- Supports custom system prompts (preambles) that define agent behavior and constraints
Distributed/Perpetual Capabilities ¶
- Rig itself is single-process, but designed to be embedded in async Rust services (Tokio-based)
- Use Rig agents within larger orchestration frameworks (actor systems, job queues) for distributed execution
- Telemetry hooks for tracing agent execution; works with OpenTelemetry collectors via Langfuse integration
Strengths ¶
- Excellent abstraction over multiple LLM providers; minimal provider lock-in
- RAG system is well-designed and flexible
- Tool system is clean and type-safe
- Active ecosystem: companies like Cairnify, Ryzome, deepwiki-rs are built on Rig
- Good documentation with tutorials (arXiv agent, Discord bot examples)
- Streaming support enables real-time agent responses
- Logging/tracing integration through Langfuse
Weaknesses ¶
- No built-in distributed actor support; must compose with other frameworks
- Memory management is delegated to external RAG system; no native “persistent memory” concept
- Less opinionated about agent loop patterns; ReAct/agentic patterns require custom implementation
- Tool calling loop is implicit; harder to inspect/debug agent decision-making
Best For ¶
- Applications requiring multi-provider flexibility
- RAG-heavy agents
- Integrating agents into web services
Claude’s Plan: Add z.ai LLM Provider to AutoAgents ¶
Context ¶
AutoAgents currently supports 10 LLM providers but lacks z.ai integration. The user wants to use the GLM Coding Plan which uses a dedicated endpoint (https://api.z.ai/api/coding/paas/v4/). The z.ai API is OpenAI-compatible (same request/response format for chat completions), so we can follow the DeepSeek pattern — a thin wrapper around OpenAICompatibleProvider<T>.
Key z.ai API Details ¶
- Auth:
Authorization: Bearer <API_KEY>(same as OpenAI) - Chat endpoint:
POST {base_url}/chat/completions - GLM Coding Plan base URL:
https://api.z.ai/api/coding/paas/v4/ - General API base URL:
https://api.z.ai/api/paas/v4/ - Default model:
glm-5(flagship, 200K context, 131K max output) - SSE streaming: OpenAI-compatible (
data: [DONE]terminator) - Tool calling: Supported, but
tool_choiceonly accepts"auto"(no"required"/"none") - Structured output:
response_formatsupports{"type": "json_object"}but not full JSON schema mode
Changes ¶
1. Create crates/autoagents-llm/src/backends/zai.rs
¶
Follow the DeepSeek pattern (deepseek.rs):
- Define
ZAIConfigimplementingOpenAIProviderConfig:PROVIDER_NAME:"ZAI"DEFAULT_BASE_URL:"https://api.z.ai/api/coding/paas/v4/"(GLM Coding Plan as default per user’s subscription)DEFAULT_MODEL:"glm-5"SUPPORTS_STRUCTURED_OUTPUT:false(no JSON schema mode, onlyjson_object)SUPPORTS_PARALLEL_TOOL_CALLS:falseSUPPORTS_STREAM_OPTIONS:trueSUPPORTS_REASONING_EFFORT:false
- Define
ZAIstruct wrappingOpenAICompatibleProvider<ZAIConfig> - Implement
ChatProviderby delegating to the inner provider (all 5 methods) - Stub
CompletionProviderandEmbeddingProvider(return errors) - Default
ModelsProvider - Implement
LLMBuilder<ZAI>::build()passing all builder fields through - Unit tests matching DeepSeek’s test coverage
2. Register in crates/autoagents-llm/src/backends/mod.rs
¶
Add:
#[cfg(feature = "zai")]
pub mod zai;
3. Add feature flag in crates/autoagents-llm/Cargo.toml
¶
- Add
zai = []feature - Add
"zai"to thefullfeature list
4. Add ZAI variant to LLMBackend in crates/autoagents-llm/src/builder.rs
¶
- Add
ZAIvariant to the enum - Add
"zai" | "z.ai" | "zhipu" => Ok(LLMBackend::ZAI)toFromStr
Files to Modify ¶
| File | Change |
|---|---|
crates/autoagents-llm/src/backends/zai.rs | New file — z.ai provider implementation |
crates/autoagents-llm/src/backends/mod.rs | Add feature-gated zai module |
crates/autoagents-llm/Cargo.toml | Add zai feature flag |
crates/autoagents-llm/src/builder.rs | Add ZAI variant to LLMBackend enum + FromStr |
Verification ¶
cargo build --all-features— confirms compilationcargo test -p autoagents-llm --all-features— runs all tests including new z.ai unit testscargo clippy --all-features --all-targets -- -D warnings— lint checkcargo fmt --check— formatting check
OpenCode + Z.ai GLM-5’s Plan ¶
Analysis Summary ¶
z.ai API Characteristics ¶
- OpenAI-compatible: Uses the same request/response format as OpenAI
- Endpoints:
- General: https://api.z.ai/api/paas/v4
- Coding (GLM Coding Plan): https://api.z.ai/api/coding/paas/v4
- Chat endpoint: /chat/completions
- Auth: Bearer token via Authorization: Bearer <API_KEY>
- Default model: glm-5
- Supported features: streaming, function calling/tools, thinking mode, structured output (json_object)
AutoAgents Backend Pattern ¶
The simplest approach is to follow the DeepSeek pattern, which:
- Defines a config struct implementing OpenAIProviderConfig
- Wraps OpenAICompatibleProvider
- Delegates all methods to the inner provider
- Implements LLMBuilder for the builder pattern
Integration Plan ¶
- Create new backend module: crates/autoagents-llm/src/backends/zai.rs
// Key implementation points:
struct ZaiConfig;
impl OpenAIProviderConfig for ZaiConfig {
const PROVIDER_NAME: &'static str = "Z.AI";
const DEFAULT_BASE_URL: &'static str = "https://api.z.ai/api/paas/v4/";
const DEFAULT_MODEL: &'static str = "glm-5";
const SUPPORTS_STREAM_OPTIONS: bool = true;
// Note: Structured output uses json_object, not json_schema
}
pub struct Zai {
provider: OpenAICompatibleProvider<ZaiConfig>,
}
- Add feature flag in Cargo.toml
zai = []
# Add to full feature list
- Register backend in backends/mod.rs
#[cfg(feature = "zai")]
pub mod zai;
- Add to LLMBackend enum in builder.rs
pub enum LLMBackend {
// ... existing variants
Zai,
}
- Support both endpoint variants The user should be able to configure:
- Default (general endpoint)
- Coding endpoint via base_url(“https://api.z.ai/api/coding/paas/v4/")
Files to Create/Modify ¶
| File | Action |
|---|---|
| crates/autoagents-llm/src/backends/zai.rs | Create - New backend module |
| crates/autoagents-llm/src/backends/mod.rs | Modify - Add #[cfg(feature = “zai”)] pub mod zai; |
| crates/autoagents-llm/Cargo.toml | Modify - Add zai = [] feature |
| crates/autoagents-llm/src/builder.rs | Modify - Add Zai to LLMBackend enum and FromStr impl |
Do you want me to proceed with this plan, or would you like any modifications? Specifically:
- Should the default endpoint be the general one (/api/paas/v4) or the coding one (/api/coding/paas/v4)?
- Should we add explicit support for the “thinking” feature (chain-of-thought), or defer that for a future enhancement?