Goodfire and Baseten bring live agent monitoring to open models

Goodfire and Baseten announced Project Beacon on October 9, combining monitoring with inference for teams deploying open models. The intended benefit is an alert while an agent is still working, early enough for an application to respond.

Goodfire describes small classifiers that inspect internal model activations during execution. The announcement lists prompt injection, sensitive-data exposure and reward hacking among the behaviors customers will be able to monitor. Application responses can include logging, further review, refusal or rerouting.

The announcement invites teams to contact the company. Agent Report has not confirmed customer availability, supported deployment configurations or commercial terms. For a small company buying a packaged assistant, the immediate question is whether its provider will operate this kind of monitoring and explain the resulting interventions.

What the research supports

Goodfire's September 17 research on reward-hacking monitors examines agents taking shortcuts that satisfy a task's measured objective without doing the intended work. Its probes sometimes detected failures that a monitor reading the reasoning transcript missed.

The comparison was not uniformly in the probes' favor. At a matched false-positive rate on DeepSWE, Goodfire reports 3.1% more hacks detected for Kimi K3, but 7.9% fewer for GLM 5.2, relative to its chain-of-thought monitor. It also describes signals appearing before a shortcut was carried out, including cases where the model ultimately did not carry it out.

Those findings justify investigating an additional detection channel. They do not establish that every flag is a completed violation, or that the newly announced Beacon service will reproduce those results on a customer's workflow. The research predates the partnership, and both sources are from the vendor.

The operational question starts after detection

Consider a hypothetical coding agent that proposes changing a test to make a failing build pass. Sometimes the test is wrong. Sometimes the change conceals the defect. An alert can help select the case for review; the reviewer still needs the original requirement, the patch and evidence about the underlying behavior.

Our assessment is that the useful unit to buy is an actionable incident. Ask what the warning contains, whether it arrives before a consequential tool call, and whether the operator can pause, inspect and resume the task. A dashboard count of suspicious runs is insufficient evidence that harm was prevented.

For a small team, the queue matters too. If every flag needs the founder's attention, the cost includes interrupted work and delayed customer responses. That belongs beside inference and monitoring charges in the operating budget.

What to ask before a trial

Request examples of ordinary tasks that triggered a warning as well as genuine failures. Ask which model versions the monitor supports and what happens when the underlying model changes. Confirm who owns investigation, how evidence is retained and how a mistaken interruption can be reversed.

Begin with reversible work and record the proportion of alerts a reviewer can resolve. Keep existing tool permissions and approval requirements in place while assessing the monitor. This is a proposed evaluation, not a report of a Beacon trial.

Earlier detection could make supervision more useful. Its value depends on what the application can still prevent when the signal arrives, and whether the person receiving it can make a sound decision. Our shared-inbox approval guide applies that same question to customer-facing work.

Grok on the record