KhaqanWhat is human review queue (human in the loop)?
A human review queue is a holding area where an AI system's outputs wait for a person to approve, correct or reject them before they take effect, keeping the system's speed while catching its mistakes before they reach customers or the accounts.
Human in the loop is the general principle; the review queue is its concrete form. An agent or pipeline produces a proposed action (a draft bill, a reply to a customer, a lead with a suggested owner, a stock adjustment) together with its evidence and a confidence score. The proposal lands in a list. A person opens it, sees the source document beside the extracted result, and approves, edits or rejects it with one action. Only then does the ERP change or the message send.
It matters because language models are fluent when they are wrong. A pipeline that is right most of the time and confidently wrong the rest will, without a queue, post wrong numbers into the accounts at machine speed, and the errors are found weeks later by someone who has no idea where they came from. The queue converts that into a few minutes of checking per day, and the corrections it collects are the best data for improving the pipeline.
I keep every AI system I run behind a queue of this kind. The design details matter: show the evidence next to the proposal so the check is fast, sort by risk (value, novelty of the supplier, low confidence) rather than by arrival, record who approved what, and measure the override rate per category so you know what to fix.
The common mistake is a queue that is too easy to approve. If the reviewer can clear a hundred items with a hundred clicks and no evidence on screen, the queue is theatre; people rubber-stamp under time pressure. The second mistake is never graduating anything: a category with a near-zero override rate over a long period can be auto-approved with sampling, so the reviewer's attention stays on the cases that need it.
Related terms
- AI agent in business operations: An AI agent in business operations is software that uses a large language model to complete a defined task, such as answering a sales enquiry or processing a bill, by reading inputs, choosing actions and calling business systems such as an ERP through tools.
- document OCR into an ERP: Document OCR into an ERP is the pipeline that turns scanned or emailed documents, most often supplier bills, into structured ERP records: extract the text with optical character recognition, interpret the fields, match them to existing data and queue the result for human approval.