{
  "id": "https://agentreport.io/digitalocean-signals-agent-regression-tests/",
  "slug": "digitalocean-signals-agent-regression-tests",
  "title": "DigitalOcean turns agent trouble into a test case",
  "description": "Signals and Simulations enter public preview, connecting production traces to candidate tests. Supported harnesses and mocked actions define the limits.",
  "url": "https://agentreport.io/digitalocean-signals-agent-regression-tests/",
  "markdownUrl": "https://agentreport.io/articles/digitalocean-signals-agent-regression-tests.md",
  "jsonUrl": "https://agentreport.io/articles/digitalocean-signals-agent-regression-tests.json",
  "format": "news",
  "type": "News reporting",
  "published": "2026-10-13",
  "modified": "2026-10-13",
  "eventDate": "2026-10-12",
  "author": {
    "name": "Agent Report Editorial",
    "kind": "house"
  },
  "status": "Source-based reporting",
  "disclosure": "Written and source-checked with Codex on October 13, 2026; copy reviewed with Grok. Product behavior is attributed to DigitalOcean's announcement and documentation. No live agent, privacy or simulation tests were performed.",
  "language": "en",
  "topics": [
    "evaluation",
    "permissions-and-safety"
  ],
  "entities": [
    {
      "name": "DigitalOcean",
      "type": "Organization",
      "url": "https://www.digitalocean.com/"
    }
  ],
  "sections": [
    {
      "title": "The useful unit is the failed session",
      "url": "https://agentreport.io/digitalocean-signals-agent-regression-tests/#the-useful-unit-is-the-failed-session"
    },
    {
      "title": "A simulation still has boundaries",
      "url": "https://agentreport.io/digitalocean-signals-agent-regression-tests/#a-simulation-still-has-boundaries"
    },
    {
      "title": "The data service has a different access status",
      "url": "https://agentreport.io/digitalocean-signals-agent-regression-tests/#the-data-service-has-a-different-access-status"
    }
  ],
  "sources": [
    {
      "title": "launch post",
      "url": "https://www.digitalocean.com/blog/introducing-signals-and-simulations",
      "publisher": "digitalocean.com"
    },
    {
      "title": "Signals instructions",
      "url": "https://docs.digitalocean.com/products/managed-agents/agent-harness-runtime/how-to/use-signals/",
      "publisher": "docs.digitalocean.com"
    },
    {
      "title": "simulation documentation",
      "url": "https://docs.digitalocean.com/products/evaluations/how-to/create-simulation/",
      "publisher": "docs.digitalocean.com"
    },
    {
      "title": "Agent Data Services",
      "url": "https://www.digitalocean.com/blog/introducing-agent-data-services",
      "publisher": "digitalocean.com"
    }
  ],
  "minutes": 3,
  "markdown": "# DigitalOcean turns agent trouble into a test case\n\nSignals and Simulations enter public preview, connecting production traces to candidate tests. Supported harnesses and mocked actions define the limits.\n\nCanonical: https://agentreport.io/digitalocean-signals-agent-regression-tests/\nAuthor: Agent Report Editorial\nPublished: 2026-10-13\nModified: 2026-10-13\nEvent date: 2026-10-12\nEvidence: Source-based reporting\nDisclosure: Written and source-checked with Codex on October 13, 2026; copy reviewed with Grok. Product behavior is attributed to DigitalOcean's announcement and documentation. No live agent, privacy or simulation tests were performed.\n\n## At a glance\n\n- DigitalOcean announced public previews of Signals and Simulations on October 12.\n- Signals identifies sessions worth investigating; Simulations evaluates candidate changes across conversations.\n- The separate Agent Data Services product is in private preview.\n\nDigitalOcean has [put Signals and Simulations into public preview](https://www.digitalocean.com/blog/introducing-signals-and-simulations), offering a route from a problematic agent session to a test of the proposed fix. The October 12 announcement targets teams operating agents on its platform.\n\nSignals sorts potential trouble into user interaction, tool execution and infrastructure problems. A flagged session can become an evaluation scenario. Simulations then runs a candidate through a multi-turn conversation and grades the resulting sequence of actions. These are vendor-described capabilities, not results from an Agent Report trial.\n\n## The useful unit is the failed session\n\nOur assessment: preserving the original failure alongside a candidate result could make agent maintenance more disciplined. A hypothetical support agent might select the right customer record but mishandle a correction two messages later. A test of the first answer alone would miss the problem. The complete exchange provides a more useful regression case.\n\nDigitalOcean's [Signals instructions](https://docs.digitalocean.com/products/managed-agents/agent-harness-runtime/how-to/use-signals/) set a practical limit: collection starts only with conversations begun after an environment is enabled. It cannot retrospectively analyze the old failure that prompted a team to sign up. Disabling collection also leaves previously collected data in place; deletion is a separate request.\n\nThe launch announcement says the opt-in service is designed to redact personal information and keep traces within the customer's Team boundary. We have not inspected those controls or their outputs.\n\n## A simulation still has boundaries\n\nThe launch post says Simulations, in this preview, works with supported Harness Runtime environments, including Claude Code, Codex and OpenCode. It explicitly excludes third-party agents such as LangChain, Hermes and CrewAI from this preview. A team should establish compatibility before treating the announcement as a replacement for its existing evaluation setup.\n\nDigitalOcean's [launch post](https://www.digitalocean.com/blog/introducing-signals-and-simulations) describes optional mocks for tools that write data or incur costs. Its [simulation documentation](https://docs.digitalocean.com/products/evaluations/how-to/create-simulation/) explains scenario creation and judge scoring, but does not document how to configure those mocks. Our reading: a successful mocked action demonstrates behavior inside that simulation. It does not prove that the real integration will succeed. Retain a separate integration check for the operation whose effect matters.\n\n## The data service has a different access status\n\nThe separately announced [Agent Data Services](https://www.digitalocean.com/blog/introducing-agent-data-services) brings records, vectors and agent session data into a managed catalog with SQL and API access. DigitalOcean labels it private preview. Combining the launches into one claim of general availability would erase a meaningful buying constraint.\n\nFor a first evaluation, choose an already understood failure and an unchanged baseline. Compare both with the proposed fix, then inspect the tool trace as well as the judge's score. We have not run that evaluation. The practical question is whether the product helps a team preserve a failure it can reliably catch next time.\n",
  "text": "DigitalOcean turns agent trouble into a test case Signals and Simulations enter public preview, connecting production traces to candidate tests. Supported harnesses and mocked actions define the limits. Canonical: https://agentreport.io/digitalocean-signals-agent-regression-tests/ Author: Agent Report Editorial Published: 2026-10-13 Modified: 2026-10-13 Event date: 2026-10-12 Evidence: Source-based reporting Disclosure: Written and source-checked with Codex on October 13, 2026; copy reviewed with Grok. Product behavior is attributed to DigitalOcean's announcement and documentation. No live agent, privacy or simulation tests were performed. At a glance - DigitalOcean announced public previews of Signals and Simulations on October 12. - Signals identifies sessions worth investigating; Simulations evaluates candidate changes across conversations. - The separate Agent Data Services product is in private preview. DigitalOcean has put Signals and Simulations into public preview, offering a route from a problematic agent session to a test of the proposed fix. The October 12 announcement targets teams operating agents on its platform. Signals sorts potential trouble into user interaction, tool execution and infrastructure problems. A flagged session can become an evaluation scenario. Simulations then runs a candidate through a multi-turn conversation and grades the resulting sequence of actions. These are vendor-described capabilities, not results from an Agent Report trial. The useful unit is the failed session Our assessment: preserving the original failure alongside a candidate result could make agent maintenance more disciplined. A hypothetical support agent might select the right customer record but mishandle a correction two messages later. A test of the first answer alone would miss the problem. The complete exchange provides a more useful regression case. DigitalOcean's Signals instructions set a practical limit: collection starts only with conversations begun after an environment is enabled. It cannot retrospectively analyze the old failure that prompted a team to sign up. Disabling collection also leaves previously collected data in place; deletion is a separate request. The launch announcement says the opt-in service is designed to redact personal information and keep traces within the customer's Team boundary. We have not inspected those controls or their outputs. A simulation still has boundaries The launch post says Simulations, in this preview, works with supported Harness Runtime environments, including Claude Code, Codex and OpenCode. It explicitly excludes third-party agents such as LangChain, Hermes and CrewAI from this preview. A team should establish compatibility before treating the announcement as a replacement for its existing evaluation setup. DigitalOcean's launch post describes optional mocks for tools that write data or incur costs. Its simulation documentation explains scenario creation and judge scoring, but does not document how to configure those mocks. Our reading: a successful mocked action demonstrates behavior inside that simulation. It does not prove that the real integration will succeed. Retain a separate integration check for the operation whose effect matters. The data service has a different access status The separately announced Agent Data Services brings records, vectors and agent session data into a managed catalog with SQL and API access. DigitalOcean labels it private preview. Combining the launches into one claim of general availability would erase a meaningful buying constraint. For a first evaluation, choose an already understood failure and an unchanged baseline. Compare both with the proposed fix, then inspect the tool trace as well as the judge's score. We have not run that evaluation. The practical question is whether the product helps a team preserve a failure it can reliably catch next time."
}