Skip to main content

Review Queue

The review queue surfaces topics that require human attention -- typically because the classification engine encountered a validation error, the LLM returned an ambiguous segment type, or a reclassification job flagged the topic for manual inspection. Super admins review these topics, provide corrections, and the system learns from each correction to improve future classifications.

This guide covers the review status workflow, how topics enter the queue, how to process reviews, and how the learning system benefits from corrections.

Review Status Workflow​

Each topic in the topics table has two review-related columns (added by migration 0029_topic_review_status.sql):

ColumnTypeDescription
review_statusTEXTCurrent review state. NULL means no review needed
review_reasonTEXTHuman-readable explanation of why review is needed

The lifecycle of a reviewed topic:

NULL (normal) → 'needs_review' (flagged) → 'reviewed' (corrected by admin)

An index on review_status WHERE review_status IS NOT NULL ensures efficient querying of topics in the review queue without impacting queries for normal topics.

How Topics Enter the Queue​

Topics are flagged for review (review_status = 'needs_review') in these situations:

  1. Invalid segment_type from LLM: During reclassification (both full and column-level), if the LLM returns an invalid segment_type value or the post-processing raises a validation error containing "Invalid segment_type", the reclassify handler sets review_status = 'needs_review' and review_reason to the error message.

  2. Human review signal: If the LLM error message contains "needs human review", the topic is flagged similarly.

  3. Manual flagging: Super admins can manually flag topics for review through the admin UI.

Design Decision: No Automatic Fallback

The system deliberately does not auto-fix segment_type errors with a local fallback. As stated in the architecture: "There is no segment type fallback until we reach 99% accuracy. It is supposed to error and provide the super admin the opportunity to train the system." This ensures every ambiguous classification gets human review.

Browsing the Review Queue​

Navigate to Admin > Topics > Review tab.

Review Queue

Listing Topics for Review​

API: GET /api/admin/topics/review

Query parameters:

ParameterDescriptionDefault
statusFilter by review statusneeds_review
pagePage number1
pageSizeItems per page (1--100)25

The response includes:

{
  "topics": [
    {
      "id": "topic_abc123",
      "topicName": "Salesforce CRM",
      "parentCategory": "Business Technology",
      "taxonomyType": "Technology & Telecom",
      "segmentType": "",
      "reviewStatus": "needs_review",
      "reviewReason": "Invalid segment_type: null returned by LLM",
      "createdAt": "2026-02-20T10:30:00.000Z"
    }
  ],
  "total": 42,
  "reviewCount": 42
}

The reviewCount field always reflects the total number of topics with needs_review status, regardless of the current filter. This is used for badge counts in the admin UI.

Outstanding Issues Endpoint​

For a quick overview of all issues requiring attention, use:

API: GET /api/admin/topics/outstanding

This returns:

  • Up to 20 topics needing review with full details
  • Total count of topics needing review
  • Recent reclassify jobs with classification errors

This endpoint powers the proactive issue surfacing when a super admin opens the classify chatbot.

Reviewing a Topic​

Step-by-Step Review Process​

  1. Open the review queue (Admin > Topics > Review)
  2. Click on a topic to view its details
  3. Read the reviewReason to understand why the topic was flagged
  4. Examine the topic's current classification data:
    • Topic name
    • Parent category (41-type)
    • Taxonomy type (13-group)
    • Current segment type (may be empty or invalid)
  5. Select the correct segment type from the dropdown:
    • B2B -- Business-to-Business
    • B2C -- Business-to-Consumer
    • B2B2C -- Business-to-Business-to-Consumer
    • B2E -- Business-to-Education
    • B2G -- Business-to-Government
  6. Provide a reason for the correction (this trains the learning system)
  7. Submit the review

Submitting a Correction​

API: POST /api/admin/topics/review

{
  "topicId": "topic_abc123",
  "segmentType": "B2B",
  "reason": "Salesforce CRM is enterprise software sold to businesses"
}

All three fields are required. The segmentType must be one of the five valid values.

What Happens on Submission​

When a correction is submitted, the system performs several cascading operations:

  1. Fetch current topic: Reads the full topic record from the database
  2. Regenerate classification layers: Runs runFullClassification() with the corrected segment type to update all 7 classification layers
  3. Regenerate platform outputs: Uses the template engine with all active platform configs to regenerate Trade Desk, LiveRamp, Internal, and custom platform names and descriptions
  4. Update the topic: Writes the corrected segment type, regenerated platform outputs, and sets review_status = 'reviewed' and clears review_reason
  5. Train the learning system: Calls learnFromHumanCorrection() to record the correction for future classification improvement
tip

The reason you provide when submitting a correction is valuable training data. Write clear, specific reasons that explain why the topic belongs to a particular segment type. For example: "University textbook publisher -- sells to educational institutions" is more useful than "it's B2E".

Classification Errors Dashboard​

In addition to the review queue, the admin panel provides a classification errors view that shows reclassify jobs with errors:

API: GET /api/admin/topics/classification-errors

Returns up to 20 recent reclassify jobs that have error_count > 0, with their full error arrays. Each error includes:

FieldDescription
idTopic ID (if available)
nameTopic name (if available)
messageError message describing the failure

This view helps identify patterns in classification failures, such as:

  • Systematic LLM misclassifications
  • Validation errors from malformed LLM responses
  • Topics in categories that consistently fail

Retrying Failed Topics​

From the errors dashboard, you can retry failed topics:

API: POST /api/admin/jobs/[jobId]/retry-errors

This creates a new reclassify job targeting only the topics that errored in the original job. The new job:

  • Inherits the mode (local/LLM) from the source job
  • Uses force: true to ensure all errored topics are reprocessed
  • Stores a retryOf reference to the original job for traceability

Learning From Corrections​

The review process feeds into AudienceGPT's learning system. Each human correction is stored and used to:

  1. Improve local classification: The learning store records the correct segment type for topic patterns, which future local classifications can reference
  2. Track accuracy: Corrections create accuracy comparison records that measure how far off the original classification was
  3. Build training data: Accumulated corrections form a dataset that can be used to fine-tune classification prompts

The learning is best-effort -- if the learning store fails, the correction still saves successfully. The try/catch around learning calls ensures review submissions never fail due to learning system issues.

Filtering Reviewed Topics​

To view topics that have already been reviewed:

API: GET /api/admin/topics/review?status=reviewed

This shows all topics where an admin has submitted a correction. You can use this to:

  • Audit past review decisions
  • Check for patterns in corrections
  • Verify that corrected topics have proper platform outputs

Best Practices​

  1. Process the review queue regularly -- topics in the queue represent classification gaps that affect data quality across all organizations
  2. Write descriptive correction reasons -- the learning system uses these to improve; vague reasons provide less value
  3. Check related topics after a correction -- if "Salesforce CRM" was miscategorized, other CRM topics might have similar issues. Use the search to find and review them proactively
  4. Run a dedup sweep after bulk corrections -- correcting segment types changes embeddings, which may create or resolve near-duplicates
  5. Monitor the errors dashboard after reclassify jobs -- catch systematic issues early before they affect too many topics

Next Steps​