Skip to content
TruthVouch Docs

TruthOps

TruthOps provides operational monitoring and governance for autonomous AI agents in your organization. Register agents, set autonomy levels, configure policies, and monitor health in real-time.

What Is TruthOps?

TruthOps is the operations layer for managing AI agents at scale. As your organization deploys more autonomous AI systems, TruthOps helps you:

  • Inventory: Register all agents with metadata (owner, risk level, type)
  • Control: Set autonomy levels (how much can agents decide independently?)
  • Configure: Define policies (escalation rules, approval gates, data retention)
  • Monitor: Track agent health (accuracy, latency, uptime, cost)
  • Override: Manually correct agent decisions when needed
  • Audit: Complete audit trail for compliance

Key Concepts

Agents

An agent is an autonomous AI system that makes decisions or takes actions with limited human oversight.

Examples: Customer service chatbot, hiring screener, content moderation, loan approval, data classifier

Not agents: ChatGPT (user controls), analytics tool (provides info only), report generator (no decisions)

Autonomy Levels

Every agent operates at a specific autonomy level (1-5):

  • Level 5: Fully autonomous (no human involvement)
  • Level 4: Autonomous + monitoring (humans monitor, not approve)
  • Level 3: Escalation-based (routine decisions automated; complex escalated)
  • Level 2: Human-decides (AI assists; humans approve)
  • Level 1: Informational only (AI provides info; humans decide everything)

Your customer service agent might be Level 3 (handle simple questions, escalate complex ones). Your hiring screener is Level 2 (AI scores; human hires).

Health Metrics

Track agent performance against targets:

  • Accuracy: % of decisions meeting quality standard
  • Latency: Response time
  • Availability: Uptime %
  • Cost: Monthly spending
  • Throughput: Decisions per minute

Getting Started

Step 1: Audit Your Agents

What autonomous AI systems do you have?

  • Customer service chatbot?
  • Hiring/recruiting tools?
  • Content moderation?
  • Financial (loan approval, fraud detection)?
  • Data analytics or insights agents?
  • Email filtering/triage?
  • Code generation or dev tools?

List all agents your organization uses.

Step 2: Register in TruthOps

For each agent, provide:

  • Name: Descriptive identifier
  • Type: Category (Customer Service, Hiring, Finance, etc.)
  • Owner: Responsible person/team
  • Risk level: Critical/High/Medium/Low
  • LLM provider: Which LLM powers it?
  • Users affected: How many people use this?

Go to TruthOps → Agents → Register New Agent

Step 3: Set Autonomy Level

For each agent, choose autonomy level (1-5) based on:

  • Risk (impact if wrong)
  • Reversibility (can errors be fixed easily?)
  • Regulatory requirement (is human approval required?)

Example:

  • Customer service (deflect 40% of tickets) → Level 3 (escalate complex)
  • Resume screener (AI scores; human hires) → Level 2 (human decides)
  • Email auto-reply (low-risk summary) → Level 5 (fully autonomous)

Go to TruthOps → Autonomy to set levels and controls

Step 4: Configure Policies

Define rules for how agents should behave:

  • Approval gates (when escalate to human?)
  • Confidence thresholds (when uncertain, what to do?)
  • Rate limits (prevent API overages)
  • Data handling (what data to retain/delete?)
  • Fallback (what if agent fails?)

Go to TruthOps → Configuration to set policies

Step 5: Monitor Health

Watch agent performance in real-time:

  • Accuracy metrics
  • Response latency
  • Uptime/SLA compliance
  • Cost tracking
  • Error rates

Go to TruthOps → Monitoring to see dashboard

Key Features

Agent Inventory

Central registry of all agents:

  • Metadata (type, owner, risk level, LLM provider)
  • Performance targets (accuracy, latency, uptime, cost)
  • Health status (green/yellow/red)
  • Quick-access to configuration and monitoring

Autonomy Controls

Define what each agent can decide independently:

  • Confidence thresholds (escalate if uncertain)
  • Approval gates (humans sign off on critical decisions)
  • Escalation rules (who to escalate to, when)
  • Override capability (users can override agent decisions)

Real-Time Monitoring

Dashboard shows agent health:

  • Performance trends
  • Alert status and history
  • Cost tracking
  • Incident management

Manual Overrides

Correct agent decisions when needed:

  • Review decision logs
  • Override incorrect decisions
  • Provide feedback (helps improve agent)
  • Bulk corrections (fix multiple errors at once)

Audit Trail

Complete logging for compliance:

  • All decisions logged
  • Overrides tracked (who, when, why)
  • Incident history
  • Compliance reporting

Core Modules

Agents

Register and classify all autonomous AI agents. View inventory, ownership, risk levels, and basic metadata.

Learn more about Agent Inventory →

Autonomy

Define autonomy levels (1-5) for each agent. Set controls, approval gates, and escalation policies.

Learn more about Autonomy Levels →

Configuration

Configure monitoring settings, alert thresholds, policies, cost limits, and integrations.

Learn more about Configuration →

Monitoring

Real-time dashboard for agent health. Track performance metrics, respond to alerts, manually override decisions.

Learn more about Monitoring →

Use Cases

Use Case 1: Risk Management

You’ve deployed 5 AI agents across the organization. You need to ensure each one operates within risk tolerance:

Solution:

  1. Register agents in TruthOps
  2. Classify by risk (Critical/High/Medium/Low)
  3. Set autonomy level appropriate for risk
  4. Configure approval gates and escalation
  5. Monitor health and respond to incidents

Outcome: Confident that agents are operating safely; audit-ready compliance.

Use Case 2: Cost Control

AI spending is growing; you need visibility into costs per agent:

Solution:

  1. Track LLM API costs per agent in TruthOps
  2. Set monthly budgets and alerts (80%, 100%)
  3. Monitor cost trends over time
  4. Implement cost controls (rate limiting, cheaper models, batch processing)

Outcome: Keep AI spending within budget; identify optimization opportunities.

Use Case 3: Performance Management

Agent accuracy drifting; you need to detect and fix issues quickly:

Solution:

  1. Set accuracy target (e.g., 85%)
  2. Configure alert when accuracy drops below target
  3. Receive alert within 1 hour
  4. Investigate root cause (data drift? new input type?)
  5. Update agent or knowledge base to fix
  6. Monitor for recovery

Outcome: Catch accuracy issues quickly; minimize customer impact.

Best Practices

  1. Start with inventory — Know what agents you have before you can govern them
  2. Right autonomy level — Match autonomy to risk; start conservative, increase gradually
  3. Clear ownership — Assign one person responsible for each agent
  4. Monitoring baseline — Monitor for 2-4 weeks before setting alerts (establish normal range)
  5. Alert fatigue — Be selective about alerts; tune thresholds to signal real problems
  6. Regular review — Quarterly check: are autonomy levels still appropriate? Are metrics targets still reasonable?

Next Steps

  1. Take inventory — List all autonomous AI agents in your org
  2. Register in TruthOps — Add each agent with metadata
  3. Classify by risk — Critical/High/Medium/Low
  4. Set autonomy levels — How much can each decide independently?
  5. Configure policies — Approval gates, escalation, data handling
  6. Monitor & iterate — Track health; adjust as needed