FaST — Fully AI Software Testing

By Bogdan Stefan Plesa An AI testing agent has two opportunities to fail. It can fail to perform a useful experiment, and it can misinterpret the experiment it did perform. Watching it finish a login journey tells us little about either risk. FaST, my Fully AI Software Testing proposal, explores agent-led testing workflows. That makes the testing agent itself part of the system we need to investigate. Before making a reliability claim, we need to know how it behaves across repeated trials, changed conditions, and deliberately faulty applications. James Bach's Seriously Testing LLMs challenges confidence based on isolated demonstrations. His later Serious Data From Testing LLMs presents evidence from a retrieval experiment under multiple conditions. These are useful challenges to FaST, not endorsements of it. The first practical question is what counts as a result. Suppose an agent disables an account and says that access is revoked. We need an observation that supports that s...

By Bogdan Stefan Plesa , creator of the FaST concept and paradigm. An autonomous testing agent needs more than a browser and a prompt. It needs an objective, observations, a restricted set of actions, and a defensible way to report what it learned. In FaST, my Fully AI Software Testing proposal, those responsibilities form an explicit architecture. The reference repository implements a compact version in Python. Its application is an in-memory model, so there are no real users, orders, or payments. The purpose is to make the boundary between an agent's choices and the evidence for its conclusions easy to inspect. One decision at a time The planner receives the current application snapshot and the history of observed actions. It returns one action. The runner validates that action against an allowlist before executing it. The resulting state becomes input to the next decision. In offline mode, the planner is a fixed sequence. In model mode, LangChain requests a structured resp...

By Bogdan Stefan Plesa A login test can click the right buttons and still tell us very little about the product. It may confirm that a familiar journey works while missing the changed redirect, the stale session, or the account that should no longer have access. Writing that script with an LLM changes how we produce the script. It does not automatically change how we investigate the application. FaST, my proposal for Fully AI Software Testing, starts with a different unit of work: the testing objective. An agent receives a goal, observes the application, chooses an action, examines the result, and decides what to try next. The intended output is evidence that helps us understand the product. “Fully AI” describes the operational delegation I want to explore. It does not mean that accountability disappears. People still choose the purpose of the investigation, the environment, the permissions, and the evidence needed for a release decision. An agent completing a task is not the same a...