diff options
Diffstat (limited to 'skills/tool-bench-opt.json')
| -rw-r--r-- | skills/tool-bench-opt.json | 9 |
1 files changed, 9 insertions, 0 deletions
diff --git a/skills/tool-bench-opt.json b/skills/tool-bench-opt.json new file mode 100644 index 0000000..3f15452 --- /dev/null +++ b/skills/tool-bench-opt.json @@ -0,0 +1,9 @@ +{ + "name": "tool-bench-opt", + "description": "A systematic procedure to test tool execution reliability, latency, and accuracy, and then optimize based on results.", + "tags": [], + "category": "general", + "created": "2026-06-22T18:45:40.356Z", + "updated": "2026-06-22T18:45:40.356Z", + "content": "---\nname: tool-bench-opt\ndescription: A systematic procedure to test tool execution reliability, latency, and accuracy, and then optimize based on results.\ntags: tool-calling, optimization, benchmark, quality-assurance\n---\n\n# Tool Call Benchmark & Optimization\n\n## 🎯 Objective\nEnsure the AI-to-Terminal bridge is stable and the model is selecting the most efficient tools with minimal hallucinations.\n\n## 🛠️ Execution Steps\n\n### Phase 1: The Stress Test (Verification)\nRun a sequence of diverse tool calls to check for failures in different categories:\n1. **Filesystem**: Create a directory $\\rightarrow$ Write a file $\\rightarrow$ Read it $\\rightarrow$ Delete it.\n2. **System**: Call `get_status` and `run_command` with a simple `ls` or `pwd`.\n3. **Web**: Perform a `web_search` and a `web_fetch` on a reliable URL.\n4. **Profile**: Use `user_set` and `user_get` to verify persistence.\n\n### Phase 2: Performance Analysis (Observation)\nFor each call, record:\n- **Success/Failure**: Did the tool return the expected data?\n- **Latency**: Did the response feel immediate or delayed?\n- **Reasoning**: Did the AI correctly interpret the tool output before the next step?\n- **Hallucinations**: Did the AI \"imagine\" a file existed before reading it?\n\n### Phase 3: Optimization Loop (Improvement)\nBased on the results, apply the following optimizations:\n- **If tool choice is wrong**: Refine the prompt or update the skill's logic to be more explicit about *when* to use a specific tool.\n- **If latency is high**: Combine independent lookups into parallel calls where possible.\n- **If data is truncated**: Use specific flags or split requests (e.g., reading a large file in chunks).\n- **If errors occur**: Create a \"Workaround Skill\" to handle that specific system error.\n\n## 📈 Success Metrics\n- 100% Tool Success Rate.\n- Zero \"Imaginary\" file references.\n- Correct sequence of dependent tool calls (Output A $\\rightarrow$ Input B).\n" +}
\ No newline at end of file |