Remediation Execution Design

Last updated: 2026-07-09

Problem

Nirofy already ingests signals, uses AI to summarize incident context, and proposes remediation plans. The missing product layer is a trusted way to apply those plans.

The key design constraint is that Nirofy should not position the AI model as an always-on autonomous production operator. The safer and more valuable product is a remediation control plane:

AI diagnoses and proposes candidate actions.
Policy decides whether an action is blocked, approval-gated, or auto-eligible.
A runner executes a known action type with scoped credentials.
Nirofy verifies the outcome and records the evidence.
The incident, runbook, and policy systems learn from the result.

This solves the SRE and DevOps pain point without asking customers to trust unbounded AI mutation rights in production.

Research Signals

Mature remediation systems converge on the same shape:

Event-driven automation platforms separate triggers, rules, actions, workflows, packs, and audit history. StackStorm describes automated remediation as workflow execution that can freeze and page a human if something goes wrong. Source: https://docs.stackstorm.com/overview.html
PagerDuty Automation Actions gives responders push-button diagnostic and remediation jobs from incident context, including web, mobile, Slack, workflow, and orchestration entry points. It also captures action output back into the incident record. Source: https://support.pagerduty.com/main/docs/automation-actions
Kubernetes supports dry-run mutation requests, which evaluate validation and admission without persistence or side effects. This is useful for preflight checks before live changes. Source: https://kubernetes.io/docs/reference/using-api/api-concepts/#dry-run
Kubernetes RBAC separates namespaced Role permissions from cluster-wide ClusterRole permissions, which supports scoped remediation runners per environment, namespace, or service boundary. Source: https://kubernetes.io/docs/reference/access-authn-authz/rbac/
GitOps systems such as Argo CD support ordered sync phases and hooks, which are useful when remediation should create or approve a declarative change instead of mutating the cluster directly. Source: https://argo-cd.readthedocs.io/en/stable/user-guide/sync-waves/

The product lesson: remediation is not just "execute command." It is action selection, blast-radius control, operator approval, execution, verification, rollback planning, and auditability.

Current Nirofy State

Existing pieces:

AI diagnosis returns suggestedFixes with remediationActionType.
Remediation policy gates actions by service, environment, severity, action type, and rate limit.
Approval, rejection, queueing, execution status, and audit history already exist.
Dashboard surfaces incident-level remediation and a remediation control center.
The executor currently maps action types directly to Kubernetes client operations.
Runbook schema and docs exist, but there is no core-platform runbooks module yet.

Gap:

There is no generic remediation runner contract.
There is no preflight plan object with blast radius, permissions, estimated impact, rollback path, and verification probes.
There is no connector model for non-Kubernetes remediation paths such as Argo CD, GitHub PRs, Terraform, cloud APIs, PagerDuty/Rundeck jobs, or customer-hosted runners.
Execution result is mostly status text, not structured evidence that can improve future recommendations.

Product Principle

Nirofy should sell "trusted remediation operations," not "AI with production access."

The AI should recommend and explain. The control plane should decide, execute, verify, and audit.

Target User Experience

Incident Detail

The remediation section should show one primary recommended action at a time:

Problem summary
Recommended action
Why this action
Confidence
Evidence used
Blast radius
Policy decision
Preflight result
Rollback plan
Expected verification signal

Primary controls:

Run Preflight
Approve
Execute
Reject
Create PR
Open Runbook

For production incidents, the most important UX is not automation density. It is making the next action obvious and defensible.

Remediation Control Center

The control center should become the cross-incident operations queue:

Pending approvals
Failed executions
Executing actions
Auto-approved actions waiting on runner capacity
Policy-blocked suggestions
High-confidence repeat fixes eligible for auto-approval

Each action should answer:

What will change?
Where will it change?
Who or what requested it?
Which policy allowed or blocked it?
What credentials or runner will be used?
How will Nirofy know it worked?
What is the rollback path?

Runbooks

Runbooks should be the bridge between AI advice and executable remediation.

Recommended model:

AI suggests a runbook and action parameters.
Published runbooks declare executable steps, required inputs, preflight probes, verification probes, and rollback steps.
Operators can execute a runbook step-by-step or approve automated execution.
Successful remediations can propose runbook updates after the incident.

Architecture

flowchart LR
  Ingest["Ingestion signals"] --> Incident["Incident correlation"]
  Incident --> Diagnosis["AI diagnosis"]
  Diagnosis --> Plan["Remediation plan"]
  Plan --> Policy["Policy + guardrails"]
  Policy -->|blocked| Audit["Audit + operator explanation"]
  Policy -->|approval required| Approval["Human approval"]
  Policy -->|auto eligible| Queue["Execution queue"]
  Approval --> Queue
  Queue --> Runner["Remediation runner"]
  Runner --> Preflight["Preflight"]
  Preflight --> Execute["Execute action"]
  Execute --> Verify["Verify health"]
  Verify --> Result["Structured result"]
  Result --> Audit
  Result --> Incident
  Result --> Learning["Suggestion ranking + runbook improvements"]

Core Domain Objects

RemediationPlan

Created by AI, runbook matching, or an operator.

Fields:

id
incidentId
organizationId
source: ai | runbook | operator | policy
summary
rootCauseHypothesis
actions[]
evidence[]
confidenceScore
createdAt

RemediationAction

One executable candidate action.

Fields:

id
planId
incidentId
actionType
target
parameters
riskLevel
blastRadius
policyDecision
preflightStatus
executionStatus
verificationStatus
rollbackPlan
runnerRef

Runner

Executes a constrained action type.

Fields:

id
kind: kubernetes | gitops | github | cloud | script | webhook | external-runbook
scope: environment, cluster, namespace, service, repo, account
capabilities[]
credentialRef
status
lastHeartbeatAt

ExecutionResult

Structured output from preflight, execution, and verification.

Fields:

actionId
runnerId
phase: preflight | execute | verify | rollback
status
startedAt
completedAt
logsRef
metrics
changedResources[]
error
operatorReadableSummary

Runner Model

Start with three runner types.

1. Built-In Kubernetes Runner

Use for the current MVP actions:

restart-service
scale-pods
rollback-deployment
adjust-resources
clear-cache
run-script

Required upgrades:

Add preflight with dry-run or read-only checks where possible.
Validate target deployment exists and is owned by the incident service.
Capture before/after deployment state.
Store changed Kubernetes resources as structured result data.
Run health verification probes after mutation.
Keep run-script heavily restricted or move it behind customer-hosted runners.

2. GitOps Runner

Use when customers prefer declarative production changes.

Actions:

create GitHub PR with manifest change
annotate Argo CD application
request sync
monitor app health

This is safer for production because the source of truth remains Git. It should be the default path for resource changes that are not emergency break-glass actions.

3. External Runbook Runner

Use for customer-owned automation systems.

Actions:

call PagerDuty/Rundeck job
call StackStorm webhook
call internal platform API
receive callback with execution result

This lets Nirofy become the intelligence and control layer without replacing existing automation investments.

Safety Model

Safety checks should be layered:

Static action allowlist by organization and environment.
Service-level policy override.
Rate limit by action type, service, and environment.
Target ownership check against service catalog.
Runner capability check.
Preflight validation.
Human approval when risk exceeds policy.
Post-execution verification.
Automatic rollback suggestion or rollback execution when explicitly approved.

Never let AI-generated free-form shell run directly in production. If run-script exists, it should point to a pre-registered script or job with typed parameters, not arbitrary model output.

Execution Lifecycle

Recommended status model:

PROPOSED
PREFLIGHT_QUEUED
PREFLIGHT_RUNNING
PREFLIGHT_PASSED
PREFLIGHT_FAILED
PENDING_APPROVAL
APPROVED
EXECUTION_QUEUED
EXECUTING
VERIFYING
COMPLETED
FAILED
ROLLBACK_RECOMMENDED
ROLLBACK_QUEUED
ROLLING_BACK
ROLLED_BACK
REJECTED
BLOCKED

The existing statuses can remain for compatibility, but the next execution design should separate preflight, approval, execution, verification, and rollback. Today those are partly collapsed into details text.

Implementation Phases

Phase 1: Make Existing Execution Trustworthy

Goal: turn current Kubernetes execution into a safer, inspectable workflow.

Backend:

Add a remediation_action_executions table for attempts and phase results.
Add preflight and verification result shapes.
Extend ExecuteRemediationUseCase into phase-based execution.
Store changed resources and verification evidence.
Add runner capability abstraction while keeping the built-in Kubernetes runner as the only implementation.

Dashboard:

Add preflight status and verification status to action detail.
Show "what will change" before approval.
Show before/after health evidence after execution.

Phase 2: Add Runbook-Backed Actions

Goal: make remediation repeatable and operator-owned.

Backend:

Implement the missing core-platform runbooks module for the existing schema.
Let runbook versions define inputs, action tags, preflight steps, execution steps, verification steps, and rollback steps.
Link AI suggestions to published runbooks.

Dashboard:

Show recommended runbooks beside suggested fixes.
Let an operator execute a runbook step or approve the whole action.
Capture whether the runbook resolved the incident.

Phase 3: Add External Runners

Goal: integrate with how SRE teams already operate.

Runners:

GitOps/GitHub runner
PagerDuty/Rundeck runner
Webhook runner
Customer-hosted script runner

Backend:

Add runner registration, heartbeat, scopes, and capabilities.
Add signed job leases so external runners can poll safely.
Add callback API for phase results.

Dashboard:

Show runner identity, scope, heartbeat, and last result.
Warn when no runner can execute the selected action.

Phase 4: Learn From Outcomes

Goal: compound operational knowledge.

Learning loop:

Rank future suggestions by prior success rate for the same service, incident class, and environment.
Recommend policy changes after repeated safe successes.
Recommend runbook updates after manual fixes.
Detect remediations that repeatedly fail or make incidents worse.

MVP Recommendation

The next build should not start with full autonomy. It should start with:

RemediationPlan and structured action/result data.
Built-in Kubernetes runner abstraction.
Preflight plus verification phases.
"What will change" and "how we know it worked" UI.
Runbook matching as the bridge from AI suggestion to executable action.

This gives SREs and DevOps engineers a practical product: fewer tabs, faster decisions, safer execution, and better post-incident evidence.

Open Questions

Should production default to GitOps execution where possible, with direct Kubernetes mutation reserved for emergency actions?
Which customer-owned runner should be first: PagerDuty/Rundeck, generic webhook, or GitHub/Argo CD?
Should run-script remain a first-class action type, or should it become an external runner capability only?
What minimum verification evidence is required before an action can be marked COMPLETED?
Should auto-approval require multiple successful historical executions for that service and action type?