{
  "id": "https://agentreport.io/how-we-test/",
  "slug": "how-we-test",
  "title": "An agent should leave you with proof",
  "description": "Our testing method: a real task, explicit permissions, a result you can inspect, and a record of the work still left to you.",
  "url": "https://agentreport.io/how-we-test/",
  "markdownUrl": "https://agentreport.io/articles/how-we-test.md",
  "jsonUrl": "https://agentreport.io/articles/how-we-test.json",
  "format": "methodology",
  "type": "Methodology",
  "published": "2026-10-07",
  "modified": "2026-10-07",
  "author": {
    "name": "Agent Report",
    "kind": "house"
  },
  "status": "Editorial standard",
  "disclosure": "This page describes how we intend to test. It is not a claim that we have run a benchmark or tested every product we cover. Hands-on reports will cite it.",
  "language": "en",
  "topics": [
    "evaluation",
    "permissions-and-safety"
  ],
  "entities": [],
  "sections": [
    {
      "title": "Give the task a finish line",
      "url": "https://agentreport.io/how-we-test/#give-the-task-a-finish-line"
    },
    {
      "title": "Record the environment",
      "url": "https://agentreport.io/how-we-test/#record-the-environment"
    },
    {
      "title": "Separate setup from execution",
      "url": "https://agentreport.io/how-we-test/#separate-setup-from-execution"
    },
    {
      "title": "Count the help",
      "url": "https://agentreport.io/how-we-test/#count-the-help"
    },
    {
      "title": "Inspect the result where it lives",
      "url": "https://agentreport.io/how-we-test/#inspect-the-result-where-it-lives"
    },
    {
      "title": "Repeat before generalizing",
      "url": "https://agentreport.io/how-we-test/#repeat-before-generalizing"
    },
    {
      "title": "Make the evidence easy to read",
      "url": "https://agentreport.io/how-we-test/#make-the-evidence-easy-to-read"
    }
  ],
  "sources": [],
  "minutes": 3,
  "markdown": "# An agent should leave you with proof\n\nOur testing method: a real task, explicit permissions, a result you can inspect, and a record of the work still left to you.\n\nCanonical: https://agentreport.io/how-we-test/\nAuthor: Agent Report\nPublished: 2026-10-07\nModified: 2026-10-07\nEvidence: Editorial standard\nDisclosure: This page describes how we intend to test. It is not a claim that we have run a benchmark or tested every product we cover. Hands-on reports will cite it.\n\nAn agent saying “done” is the beginning of verification. For a document task, we need the document. For a code change, we need the change and a relevant check. For a booking, we need the confirmation from the service that accepted it.\n\nAgent Report will evaluate agents through tasks with inspectable outcomes. This is our proposed method. It is not a claim that we have already run a benchmark or tested every product we cover.\n\n## Give the task a finish line\n\nBefore the run, write down what success would look like. “Research a trip” leaves too much room for interpretation. “Find three options for these dates, check the total prices and cancellation terms, and link the booking pages” has a result a reader can inspect.\n\nThe task should represent something a person actually needs. We will publish its constraints with the result so readers can judge whether the trial resembles their own work.\n\n## Record the environment\n\nEach hands-on report should name the product, trial date, plan, relevant version, device and connected services. We will state when a feature is unavailable to us. If a model or setting can affect the result, that belongs in the report too.\n\nAvailability changes. A successful run in one account is evidence about that run, not a promise that every reader has the same access.\n\n## Separate setup from execution\n\nConnecting accounts, configuring permissions and learning the interface all take time. We will record those steps separately from the agent’s running time, and distinguish active supervision from waiting.\n\nA ten-minute task that needs twenty minutes of preparation can still be useful repeatedly. Readers need both numbers to decide.\n\n## Count the help\n\nWe will record corrections, repeated instructions, manual recovery and steps completed outside the product. If the tester rescues a run, the rescue stays in the story.\n\nWe will also identify the point at which the user must make a decision: approving a recipient, a purchase, a file change or an external action. Approval should describe what will happen and keep the scope understandable.\n\n## Inspect the result where it lives\n\nA summary of an action is not its receipt. We will verify the actual file, application, destination or service when the task permits it. If we cannot inspect the outcome, we will mark it unverified.\n\nA failed step is useful evidence. We want to know whether the agent detects the failure, explains the remaining work and leaves the user able to recover.\n\n## Repeat before generalizing\n\nA single trial can establish that something happened once. It cannot establish a reliability rate. We will disclose trial counts and avoid turning one successful demonstration into a numerical score for the whole product.\n\nWhen a comparison is practical, we will keep task conditions aligned and explain differences in access, setup or capabilities. We will publish observations before awarding rankings.\n\n## Make the evidence easy to read\n\nOur articles will distinguish four kinds of information:\n\n- **Vendor claim:** a capability described by the company.\n- **Source review:** our reading of linked documentation or announcements.\n- **Hands-on observation:** an outcome we directly inspected during a disclosed trial.\n- **Analysis:** our interpretation, including uncertainty and questions for further testing.\n\nEvery article should say which kind of work it contains. Material corrections should carry a dated note. Relationships to products we cover should be disclosed where they affect the article.\n\nOur job is to help you decide what is worth trying, what deserves access, and what still needs your attention.\n",
  "text": "An agent should leave you with proof Our testing method: a real task, explicit permissions, a result you can inspect, and a record of the work still left to you. Canonical: https://agentreport.io/how-we-test/ Author: Agent Report Published: 2026-10-07 Modified: 2026-10-07 Evidence: Editorial standard Disclosure: This page describes how we intend to test. It is not a claim that we have run a benchmark or tested every product we cover. Hands-on reports will cite it. An agent saying “done” is the beginning of verification. For a document task, we need the document. For a code change, we need the change and a relevant check. For a booking, we need the confirmation from the service that accepted it. Agent Report will evaluate agents through tasks with inspectable outcomes. This is our proposed method. It is not a claim that we have already run a benchmark or tested every product we cover. Give the task a finish line Before the run, write down what success would look like. “Research a trip” leaves too much room for interpretation. “Find three options for these dates, check the total prices and cancellation terms, and link the booking pages” has a result a reader can inspect. The task should represent something a person actually needs. We will publish its constraints with the result so readers can judge whether the trial resembles their own work. Record the environment Each hands-on report should name the product, trial date, plan, relevant version, device and connected services. We will state when a feature is unavailable to us. If a model or setting can affect the result, that belongs in the report too. Availability changes. A successful run in one account is evidence about that run, not a promise that every reader has the same access. Separate setup from execution Connecting accounts, configuring permissions and learning the interface all take time. We will record those steps separately from the agent’s running time, and distinguish active supervision from waiting. A ten-minute task that needs twenty minutes of preparation can still be useful repeatedly. Readers need both numbers to decide. Count the help We will record corrections, repeated instructions, manual recovery and steps completed outside the product. If the tester rescues a run, the rescue stays in the story. We will also identify the point at which the user must make a decision: approving a recipient, a purchase, a file change or an external action. Approval should describe what will happen and keep the scope understandable. Inspect the result where it lives A summary of an action is not its receipt. We will verify the actual file, application, destination or service when the task permits it. If we cannot inspect the outcome, we will mark it unverified. A failed step is useful evidence. We want to know whether the agent detects the failure, explains the remaining work and leaves the user able to recover. Repeat before generalizing A single trial can establish that something happened once. It cannot establish a reliability rate. We will disclose trial counts and avoid turning one successful demonstration into a numerical score for the whole product. When a comparison is practical, we will keep task conditions aligned and explain differences in access, setup or capabilities. We will publish observations before awarding rankings. Make the evidence easy to read Our articles will distinguish four kinds of information: - Vendor claim: a capability described by the company. - Source review: our reading of linked documentation or announcements. - Hands-on observation: an outcome we directly inspected during a disclosed trial. - Analysis: our interpretation, including uncertainty and questions for further testing. Every article should say which kind of work it contains. Material corrections should carry a dated note. Relationships to products we cover should be disclosed where they affect the article. Our job is to help you decide what is worth trying, what deserves access, and what still needs your attention."
}
