Unlocking the Agent Toolkit: A Guide to AgentSHAP

By Miriam Horovicz // Dec 2025

AgentSHAP research paper explanation

In the world of AI, we’ve moved past simple chatbots. We are now in the era of Agents - LLMs equipped with "tools" like web browsers, Python interpreters, and SQL databases. They don't just talk; they act.

However, as these agents become more complex, we face a new problem: The Attribution Gap. If your agent gives a wrong answer, was it because the "Search" tool returned bad data, or because the agent ignored the "Calculator"?

At Fiverr Labs, we’ve been obsessed with this "black box" problem. We believe that for AI to be truly useful in professional environments, it must be accountable. This mission led us to develop AgentSHAP, a breakthrough research project that has been accepted to the AAAI 2026 LM Reasoning Bridge in Singapore.

The Concept: Who Gets the Credit?

Imagine a team project where three specialists: Search, Calculator, and Database are working together. If the project succeeds, who was the MVP? You can’t just look at who spoke the most; you have to see who actually added value to the final result.

AgentSHAP solves this using Shapley Values, a Nobel-prize-winning concept from game theory. It treats each tool like a player in a game and calculates their "fair share" of the final response's quality.

How it Works: The A-to-Z of Tool Attribution

  1. The Baseline: We first let the agent answer using all its tools. This is our "perfect" reference point.
  2. The "What-If" Scenarios: AgentSHAP runs the agent multiple times, but each time it "hides" certain tools (e.g., "Answer this math problem, but you only have Search, no Calculator").
  3. Measuring the Semantic Drop: We compare these limited answers to the original. If removing the "Search" tool makes the answer 90% worse, that tool gets a high importance score.
  4. The Fair Split: Because tools often work together (synergy), the Shapley math ensures that importance is distributed fairly, even if tools overlap in their capabilities.

Why This Matters for the Future of XAI

In simple terms, XAI is the technology that turns an AI "Black Box" into a "Glass Box," allowing us to see the reasoning behind every decision.

  • Trust Calibration: You can verify if the agent actually relied on your internal "Legal Database" or just guessed based on its training data.
  • Cost Efficiency: If a "Premium Research API" always gets a low importance score, you can remove it to save on API costs without hurting performance.

Seeing it in Action: Code Example

AgentSHAP is model-agnostic. Whether you use GPT-4o, Claude, or a local Llama model, it treats the agent as a "black box." Here is how you can implement it today:

from tokenshap import AgentSHAP

# 1. Define your tools (executors)
weather_tool = create_function_tool(name="get_weather", executor=lambda args: "72°F")
stock_tool = create_function_tool(name="get_stock", executor=lambda args: "$150.25")

# 2. Initialize AgentSHAP with your model and toolset
agent_shap = AgentSHAP(model="llama3", tools=[weather_tool, stock_tool])

# 3. Analyze a specific prompt
# "What's the weather in NYC and how is AAPL doing?"
results_df, shapley_values = agent_shap.analyze(
    prompt="Check NYC weather and AAPL price", 
    sampling_ratio=0.5
)

# 4. Visualize the "Fair Credit" for each tool
agent_shap.plot_tool_importance()

Efficiency Through Sampling

Calculating every tool combination is mathematically expensive ($2^n$). To solve this, our team implemented Monte Carlo Sampling. This allows AgentSHAP to provide a statistically accurate importance score in seconds, making it viable for real-world production monitoring.

By bringing this level of clarity to the agentic layer, we can finally move from "hoping the AI is right" to "knowing why the AI is right."

Paper and code can be found here https://www.arxiv.org/abs/2512.12597 https://github.com/GenAISHAP/TokenSHAP