SPAgent is a foundation agent for the physical & spatial world
A general-purpose foundation agent framework for multimodal tool calling, connecting vision-language models with an open-ended ecosystem of expert tools.
A general-purpose foundation agent framework for multimodal tool calling, connecting vision-language models with an open-ended ecosystem of expert tools.
Mix and match any combination of expert tools. Add or remove tools at runtime with a single function call.
Integrate any tool into the ecosystem. Define a schema, plug it in, and the agent will use it automatically.
Purpose-built prompts for 3D spatial understanding. Grounded perception in complex physical environments.
GRPO training with multi-turn tool calling and an offline Pi3X cache, powered by ms-swift.
AgentMemory preserves text, images, tool calls, and results across multi-turn conversations.
Run expert tools concurrently with GPT, Qwen, or local OpenAI-compatible model backends.
26 registered tool integrations spanning perception, grounding, 3D understanding, document intelligence, and generation.
High-accuracy monocular depth estimation for dense depth maps from a single image.
Promptable and universal semantic, instance, and panoptic image segmentation.
Open-vocabulary object detection driven by natural-language prompts and referring expressions.
Fast local object detection plus lightweight CPU frontal-face detection.
Open-vocabulary detection, segmentation, annotation, and visualization.
Dense optical flow estimation between image pairs with raw motion fields.
OCR and structured recognition for text, tables, charts, formulas, and seals.
Lightweight visual question answering, captioning, and point grounding.
Multimodal reasoning with QA, captioning, and annotated point outputs.
Referring and reasoning-based object detection from image-text prompts.
3D point cloud reconstruction from single or multiple images, with Pi3X adding smoother metric-scale outputs.
Feed-forward multi-view 3D reconstruction with camera pose, depth, and geometry prediction in one pass.
Universal metric 3D reconstruction for dense point clouds, depth, poses, and multi-view geometry.
Object orientation, symmetry, and two-image relative rotation estimation.
Promptable 3D object detection from a single RGB image using text, boxes, or points.
Fast text-to-image generation through a local SGLang server.
Text-to-video and image-to-video generation across Google, OpenAI, and DashScope APIs.
Local first-frame video generation from a reference image and motion prompt.
from spagent import SPAgent from spagent.models import GPTModel from spagent.tools import DepthEstimationTool, SegmentationTool # Create model and tools model = GPTModel(model_name="gpt-4o-mini") tools = [ DepthEstimationTool(use_mock=True), SegmentationTool(use_mock=True) ] # Mock mode needs no expert servers agent = SPAgent(model=model, tools=tools, max_workers=4) result = agent.solve_problem( "image.jpg", "Analyze depth relationships and main objects" ) print(result['answer'])
Use step() with AgentMemory for stateful multi-turn sessions, or connect real expert servers with use_mock=False.
Agent loop, parallel execution, workflow routing, AgentMemory, and data collection.
Standardized ToolResult envelopes, typed payloads, rendering, and contribution CI.
26 catalog integrations using local, client/server, and cloud API deployments.
GPT, Qwen, and local OpenAI-compatible multimodal models.
Multi-turn tool-calling rewards with offline Pi3X cache and ms-swift.
Unified quick evaluation across VLMEvalKit and spatial reasoning benchmarks.
Open-source and ready to use. Deploy expert tools, connect your model, and reason about the physical world.
