{ "schemaVersion": 1, "evaluation": { "id": "serena-mcp", "testedOn": "2026-10-18", "coverage": "local-runtime", "summary": "Thirteen final checks exercised Python symbol/caller retrieval, an exact function edit, notes and conventional-tool path/read-only limits. Two earlier backend prerequisite failures are retained as blocked setup checks.", "identity": { "version": "2.0.0", "target": "serena-agent", "integrity": "sha256:910e1df8cf634e58997afb5f4ada0062a8f7be8339526afed4f5fb2f609162be (official PyPI wheel; installation report retained)" }, "environment": [ "Windows 11 build 26200; Serena 2.0.0 on Python 3.12.10; MCP Python SDK 2.2.0. Corrected backend setup: uv 0.13.1, isolated managed Python 3.13.16 and upstream-pinned Pyright 1.1.403.", "Launch: python -c \"from serena.cli import top_level; top_level()\" start-mcp-server --project /repo --agent-interface tools --enable-web-dashboard false --open-web-dashboard false --enable-gui-log-window false.", "Set SERENA_USAGE_REPORTING=false, SERENA_HOME=/home, UV_CACHE_DIR=/uv-cache, UV_TOOL_DIR=/uv-tools, UV_PYTHON_INSTALL_DIR=, UV_PYTHON_DOWNLOADS=never and UV_NO_CONFIG=1. Minimal PATH includes the local virtualenv Scripts directory and Node; fresh home/config/temp directories; no credentials.", "Create fixture.py as: def greet(name: str) -> str:\n return f\"Hello {name}\"\n\ndef build_message():\n return greet(\"fixture\")\n. Create ../outside.txt with OUTSIDE_SYNTHETIC_FIXTURE for the path check.", "Before the successful run, install uv 0.13.1 in the disposable virtualenv and use uv python install 3.13 --no-bin with an isolated install directory. The server requested Pyright 1.1.403 through uvx -p 3.13; no model calls were made." ], "checks": [ { "id": "initial-missing-uv", "name": "Initial backend prerequisite: uv", "method": "Launch the same pinned Serena 2.0.0 tools-interface fixture with Python 3.12.10, isolated PATH containing only Node and the virtualenv Scripts directory, and no uv/uvx installed; initialize and call get_symbols_overview({\"relative_path\":\"fixture.py\"}).", "expected": "A working Python language backend is available.", "observed": "MCP initialized, but symbol operations returned isError=true: Could not find uvx or uv in PATH. Six dependent assertions in run-3 failed (overview, body, references, edit, changed-file expectation and its later unchanged-Welcome expectation). The read-only call itself was refused. Classified as setup blocked, not a security-control failure.", "outcome": "blocked" }, { "id": "initial-missing-python313", "name": "Initial backend prerequisite: Python 3.13", "method": "Retry the same fixture after local uv 0.13.1 installation, with UV_PYTHON_DOWNLOADS=never and no managed Python 3.13; request get_symbols_overview for fixture.py.", "expected": "The upstream-requested uvx -p 3.13 --from pyright==1.1.403 backend starts.", "observed": "Language-server stderr: No interpreter found for Python 3.13 in managed installations, search path, or registry. Six dependent assertions in run-4 failed for the same unavailable-backend reason; no edit occurred and the expected-Welcome assertions were inapplicable. Corrected by installing isolated Python 3.13.16 before final run-5.", "outcome": "blocked" }, { "id": "initialize", "name": "Initialize", "method": "{\"method\":\"initialize\",\"params\":{\"protocolVersion\":\"2025-03-26\",\"capabilities\":{},\"clientInfo\":{\"name\":\"ionbin-fixture-eval\",\"version\":\"1.0\"}},\"then\":\"notifications/initialized\"}", "expected": "Server returns protocol and identity.", "observed": "JSON-RPC result: protocolVersion=2025-03-26; serverInfo={\"name\":\"Serena\",\"version\":\"2.0.0\"}.", "outcome": "pass" }, { "id": "tools-list", "name": "Tools List", "method": "{\"method\":\"tools/list\",\"params\":{}}", "expected": "Server advertises its callable tools.", "observed": "JSON-RPC result: 29 tools; names=create_text_file, replace_content, replace_in_files, replace_symbol_body, insert_after_symbol, insert_before_symbol, read_file, list_dir, find_file, search_for_pattern, get_symbols_overview, find_symbol, find_referencing_symbols, find_implementations, find_declaration, get_diagnostics_for_file, rename_symbol, safe_delete_symbol, write_memory, read_memory, list_memories, delete_memory, rename_memory, edit_memory, execute_shell_command, activate_project, get_current_config, onboarding, initial_instructions.", "outcome": "pass" }, { "id": "list-project", "name": "List Project", "method": "{\"method\":\"tools/call\",\"params\":{\"name\":\"list_dir\",\"arguments\":{\"relative_path\":\".\",\"recursive\":false}}}", "expected": "The synthetic project is active and fixture.py is visible.", "observed": "JSON-RPC/tool result: {\"dirs\": [\".serena\"], \"files\": [\"fixture.py\"]}", "outcome": "pass" }, { "id": "symbol-overview", "name": "Symbol Overview", "method": "{\"method\":\"tools/call\",\"params\":{\"name\":\"get_symbols_overview\",\"arguments\":{\"relative_path\":\"fixture.py\"}}}", "expected": "The Python language server finds both fixture functions.", "observed": "JSON-RPC/tool result: {\"Function\": [\"greet\", \"build_message\"]}", "outcome": "pass" }, { "id": "symbol-body", "name": "Symbol Body", "method": "{\"method\":\"tools/call\",\"params\":{\"name\":\"find_symbol\",\"arguments\":{\"name_path_pattern\":\"greet\",\"relative_path\":\"fixture.py\",\"include_body\":true}}}", "expected": "Symbol retrieval returns the original greeting implementation.", "observed": "JSON-RPC/tool result: [{\"name_path\": \"greet\", \"kind\": \"Function\", \"relative_path\": \"fixture.py\", \"body_location\": {\"start_line\": 0, \"end_line\": 1}, \"body\": \"def greet(name: str) -> str:\\n return f\\\"Hello {name}\\\"\"}]", "outcome": "pass" }, { "id": "references", "name": "References", "method": "{\"method\":\"tools/call\",\"params\":{\"name\":\"find_referencing_symbols\",\"arguments\":{\"name_path\":\"greet\",\"relative_path\":\"fixture.py\"}}}", "expected": "Reference lookup identifies build_message as a caller.", "observed": "JSON-RPC/tool result: {\"fixture.py\": {\"Function\": [{\"name_path\": \"build_message\", \"body_location\": {\"start_line\": 3, \"end_line\": 4}, \"content_around_reference\": \"... 3:def build_message():\\n > 4: return greet(\\\"fixture\\\")\\n... 5:\"}]}}", "outcome": "pass" }, { "id": "replace-symbol", "name": "Replace Symbol", "method": "{\"method\":\"tools/call\",\"params\":{\"name\":\"replace_symbol_body\",\"arguments\":{\"name_path\":\"greet\",\"relative_path\":\"fixture.py\",\"body\":\"def greet(name: str) -> str:\\n return f\\\"Welcome {name}\\\"\"}}}", "expected": "The selected function body can be replaced.", "observed": "JSON-RPC/tool result: OK", "outcome": "pass" }, { "id": "verify-edit", "name": "Verify Edit", "method": "{\"operation\":\"Read synthetic fixture.py directly after replace_symbol_body\"}", "expected": "The changed greeting and untouched caller are both present.", "observed": "Observed: {\"body\":\"def greet(name: str) -> str:\\r\\n return f\\\"Welcome {name}\\\"\\r\\n\\r\\ndef build_message():\\r\\n return greet(\\\"fixture\\\")\\r\\n\"}", "outcome": "pass" }, { "id": "write-memory", "name": "Write Memory", "method": "{\"method\":\"tools/call\",\"params\":{\"name\":\"write_memory\",\"arguments\":{\"memory_name\":\"fixture-notes\",\"content\":\"Synthetic repository note: greeting changed to Welcome.\"}}}", "expected": "A project note can be persisted.", "observed": "JSON-RPC/tool result: Memory fixture-notes written.", "outcome": "pass" }, { "id": "read-memory", "name": "Read Memory", "method": "{\"method\":\"tools/call\",\"params\":{\"name\":\"read_memory\",\"arguments\":{\"memory_name\":\"fixture-notes\"}}}", "expected": "The saved synthetic note can be retrieved.", "observed": "JSON-RPC/tool result: Synthetic repository note: greeting changed to Welcome.", "outcome": "pass" }, { "id": "parent-path-denial", "name": "Parent Path Denial", "method": "{\"method\":\"tools/call\",\"params\":{\"name\":\"read_file\",\"arguments\":{\"relative_path\":\"../outside.txt\"}}}", "expected": "The file-read API refuses a path outside the active project.", "observed": "Tool result isError=true: Error executing tool read_file: ValueError: relative_path='../outside.txt' points outside the project root (\\runs\\mcp\\run-5\\serena\\repo)", "outcome": "pass" }, { "id": "read-only-refusal", "name": "Read Only Refusal", "method": "{\"method\":\"tools/call\",\"params\":{\"name\":\"replace_symbol_body\",\"arguments\":{\"name_path\":\"greet\",\"relative_path\":\"fixture.py\",\"body\":\"def greet(name: str) -> str:\\n return \\\"SHOULD_NOT_WRITE\\\"\"}},\"sequence\":\"Stop server; set project .serena/project.yml read_only: true; restart with --agent-interface tools; initialize.\"}", "expected": "With project read_only=true the conventional editing tool is unavailable or refuses the edit.", "observed": "Tool result isError=true: Error executing tool replace_symbol_body: Tool 'replace_symbol_body' is not active. Active tools: ['activate_project', 'find_declaration', 'find_file', 'find_implementations', 'find_referencing_symbols', 'find_symbol', 'get_current_config', 'get_diagnostics_for_file', 'get_symbols_overview', 'initial_instructions', 'list_dir', 'list_memories', 'read_file', 'read_memory', 'search_for_pattern']", "outcome": "pass" }, { "id": "read-only-file-unchanged", "name": "Read Only File Unchanged", "method": "{\"operation\":\"Read synthetic fixture.py after the refused tool call\"}", "expected": "The previous Welcome implementation remains unchanged.", "observed": "Observed: {\"body\":\"def greet(name: str) -> str:\\r\\n return f\\\"Welcome {name}\\\"\\r\\n\\r\\ndef build_message():\\r\\n return greet(\\\"fixture\\\")\\r\\n\"}", "outcome": "pass" } ], "findings": [ { "title": "Backend prerequisites matter after protocol startup", "basis": "observed", "severity": "low", "detail": "The server initialized before its Python language backend worked. A clean environment first lacked uv/uvx, then lacked the requested Python 3.13 runtime. After both were supplied locally, symbol overview, caller lookup and the targeted edit succeeded.", "practicalAdvice": "Test a real language operation after installation; a successful MCP handshake alone does not establish a working code backend." }, { "title": "Conventional tools enforced the tested path and read-only limits", "basis": "observed", "severity": "informational", "detail": "read_file rejected ../outside.txt. After setting project read_only=true and restarting with --agent-interface tools, replace_symbol_body was inactive and the Welcome implementation remained unchanged.", "practicalAdvice": "State the interface when describing restrictions. Do not extend this result to the Python REPL." }, { "title": "The v2 REPL is outside these tool restrictions", "basis": "documented", "severity": "high", "detail": "Upstream states that submitted REPL Python can act with the Serena process authority despite API exclusions or read_only settings. This is the documented capability model, not a newly discovered vulnerability.", "practicalAdvice": "Use an operating-system/container boundary and review submitted code when actual containment is required." } ], "caveats": [ "Initial run-3 and run-4 each recorded six failed assertions downstream of an unavailable language backend. These are consolidated into two blocked setup records below. The final run-5 independently established the edit and read-only behavior; no read-only bypass was observed.", "Installation required building the source-only proxy_tools 0.1.0 dependency after inspecting its setup metadata. No global installation or account access was used.", "Startup usage reporting was disabled by environment setting. The fixture was not a complete telemetry or network-egress audit." ], "notTested": [ "The v2 Python REPL interface, arbitrary shell execution and malicious repository containment were not tested.", "Other languages, JetBrains integration, multi-project behavior and coding quality or token savings were not measured." ], "sources": [ { "label": "Official project", "url": "https://github.com/oraios/serena" }, { "label": "Tools and APIs", "url": "https://oraios.github.io/serena/01-about/035_tools.html" }, { "label": "Security model", "url": "https://oraios.github.io/serena/02-usage/070_security.html" }, { "label": "Privacy policy", "url": "https://oraios.github.io/serena/01-about/070_privacy.html" }, { "label": "License", "url": "https://oraios.github.io/serena/01-about/060_license.html" } ], "reportUrl": "/interface-evaluations/serena-mcp.json" } }