Who’s checking all that AI work?

The easiest number to inflate in an AI business is the amount of stuff it produces. More drafts. More tasks. More updates about the tasks. I would rather know what got finished, what needed fixing, and whether the customer came back.

Anthropic’s October 16 study of its own workplace is useful because it describes both rising output and the attention required to manage it. It reports less emphasis on junior technical hiring and a growing need for judgment around agent work. Those are company observations, not proof that another business should copy its staffing choices. Anthropic is studying a workplace deeply organized around the product it sells.

A different setting offers a useful comparison. Generative AI at Work, an NBER working paper issued in 2023, examined a conversational assistant used by customer-support workers. Its abstract reports productivity gains, with larger benefits for less experienced workers. The point here is not to turn an older study into a prediction about today’s systems. It is that assistance and replacement are different operating choices, and results depend on the job being measured.

For an entrepreneur, that is permission to think more specifically. A studio chasing qualified bookings has a different finish line from a service desk resolving a customer’s problem. The right question is what has to be true before a task is considered complete. If the answer is simply that the software produced a response, the business has defined success too early.

Imagine a small creative agency using an assistant to prepare project briefs. This is a hypothetical workflow, not a reported case. One brief can be generated quickly and still omit the deadline, misunderstand the budget, or assign the wrong deliverables. A finished brief has the necessary information, has been checked by someone responsible, and can guide the next person’s work. The extra text is not the product. A usable handoff is.

I would test one routine process before making a sweeping staffing decision. Define the outcome, record the time spent checking and correcting, and compare the finished result with the previous way of working. Include the awkward cases rather than selecting only the clean demonstrations. If the tool creates a new queue for the owner to review every evening, that time belongs in the calculation.

There is also a constructive alternative to treating beginners as a cost to erase. Ask whether the system helps a newer worker learn the standards behind a good result. Can that person explain why a response was accepted, catch an error, and handle an exception? Those are better questions than whether someone can make the largest volume of material in an hour.

I am bullish on tools that let a small team do more valuable work. I am less impressed by an operation that becomes busier describing its productivity. Give the system a defined job, a responsible reviewer, and a finish line that means something outside the software. Then count the work that crosses it.