Skip to main content

Design: Apprise Notification Abstraction

Overview​

Claude Ops integrates the Apprise CLI as its universal notification layer, enabling operators to receive health check summaries, remediation reports, and human-attention alerts through any combination of 80+ notification services. The design prioritizes zero-code configuration (a single environment variable), graceful degradation (silent no-op when unconfigured), and alignment with Claude Ops' shell-command-in-markdown execution model.

Architecture​

Component Interaction​

+-----------------------------------------------------------+
|                    Docker Container                        |
|                                                           |
|   entrypoint.sh                                           |
|     |                                                     |
|     +-- Reads CLAUDEOPS_APPRISE_URLS from environment     |
|     |   (if set, includes in agent system prompt)         |
|     |                                                     |
|     +-- claude --model haiku --prompt-file tier1-observe  |
|           |                                               |
|           +-- Agent reads prompt instructions             |
|           |   (notification steps reference Apprise)      |
|           |                                               |
|           +-- Agent executes: apprise -t "..." -b "..."   |
|               "$CLAUDEOPS_APPRISE_URLS"                   |
|               |                                           |
|               +-- Apprise CLI parses URL schemes          |
|               +-- Dispatches to service-specific plugins  |
|               +-- ntfy, Slack, email, Discord, etc.       |
+-----------------------------------------------------------+
        |                    |                    |
        v                    v                    v
   +---------+         +---------+         +---------+
   | ntfy    |         | Slack   |         | Email   |
   | server  |         | webhook |         | SMTP    |
   +---------+         +---------+         +---------+

Apprise runs entirely within the Claude Ops container as a CLI subprocess. There are no sidecars, message queues, or relay services. The agent invokes apprise via Bash the same way it invokes curl or docker -- as a shell command from within its prompt-driven execution context.

Installation in Docker Image​

Apprise is installed in the Docker image as a Python package:

# System dependencies include python3 and pip3
RUN apt-get update && apt-get install -y --no-install-recommends \
    python3 python3-pip python3-venv ...

# Apprise for notifications
RUN pip3 install --break-system-packages apprise

The --break-system-packages flag is required because the base image (node:22-slim) uses a Debian-based distribution with PEP 668 externally-managed environment markers. Apprise is installed system-wide rather than in a virtual environment to ensure it is available on the default PATH for the agent's Bash commands.

This introduces a Python runtime dependency into a primarily Node.js-based container. The trade-off is accepted because:

  • Python 3 and pip are lightweight additions to the Debian base
  • Apprise's dependency tree is self-contained
  • No alternative provides comparable breadth of notification service support without additional infrastructure

Environment Variable Flow​

.env file or docker-compose.yaml
  |
  +-- CLAUDEOPS_APPRISE_URLS=ntfy://ntfy.sh/ops,slack://...
        |
        +-- docker-compose.yaml passes to container environment
              |
              +-- entrypoint.sh reads from environment
                    |
                    +-- Conditional inclusion in agent system prompt:
                    |   if [ -n "${CLAUDEOPS_APPRISE_URLS:-}" ]; then
                    |       ENV_CONTEXT+=" CLAUDEOPS_APPRISE_URLS=${CLAUDEOPS_APPRISE_URLS}"
                    |   fi
                    |
                    +-- Agent references $CLAUDEOPS_APPRISE_URLS in Bash commands

The conditional inclusion in entrypoint.sh (lines 79-81) ensures that:

  • When set, the variable is available to the agent for constructing Apprise commands
  • When unset, the variable is absent from the agent's context, and prompt instructions tell the agent to silently skip notifications

Notification Event Flow​

Health Check Cycle
  |
  +-- All healthy?
  |     |
  |     +-- YES: Daily digest due?
  |     |         |
  |     |         +-- YES: Send daily digest via Apprise
  |     |         +-- NO: Exit quietly
  |     |
  |     +-- NO: Escalate to Tier 2
  |             |
  |             +-- Tier 2 fixes issue?
  |             |     |
  |             |     +-- YES: Send auto-remediation report via Apprise
  |             |     +-- NO (cooldown exceeded): Send human attention alert via Apprise
  |             |     +-- NO (needs more): Escalate to Tier 3
  |             |                           |
  |             |                           +-- Fixed: Send Tier 3 remediation report
  |             |                           +-- Not fixed: Send human attention alert
  |             |
  |             +-- Services in cooldown: Send human attention alert via Apprise

Data Flow​

Apprise URL Parsing​

Apprise URLs encode the notification service, credentials, and target in a single string:

scheme://[credentials@]host[:port]/target[?params]

Examples:
  ntfy://ntfy.sh/my-topic
  slack://tokenA/tokenB/tokenC/#channel
  mailto://user:pass@smtp.gmail.com?to=ops@example.com
  discord://webhook_id/webhook_token
  tgram://bot_token/chat_id
  pagerduty://integration_key@routing_key

Multiple URLs are comma-separated in CLAUDEOPS_APPRISE_URLS. Apprise handles the parsing and dispatching internally -- the agent does not need to understand URL schemes. It passes the entire variable value to the apprise CLI.

Message Construction by Tier​

Each tier constructs notification messages differently based on the context available:

Tier 1 (Observe):

# Daily digest
apprise -t "Claude Ops: Daily Health Summary" \
  -b "Services checked: 12
Healthy: 10
Degraded: 1 (service-x: high response time)
Down: 1 (service-y: HTTP 503)
In cooldown: 0" \
  "$CLAUDEOPS_APPRISE_URLS"

Tier 2 (Investigate/Remediate):

# Successful remediation
apprise -t "Claude Ops: Auto-remediated service-y" \
  -b "Issue: Container was in CrashLoopBackOff (5 restarts in 10 min).
Action: docker restart service-y.
Status: Service responding HTTP 200, latency 45ms." \
  "$CLAUDEOPS_APPRISE_URLS"

# Needs human attention
apprise -t "Claude Ops: Needs human attention -- service-y" \
  -b "Issue: Container keeps crashing after restart.
Cooldown limit reached: 2/2 restarts in 4h window.
Attempts: Restarted twice, checked logs for OOM.
Last error: out of memory (container limit 256MB)." \
  "$CLAUDEOPS_APPRISE_URLS"

Tier 3 (Full Remediation):

# Remediated
apprise -t "Claude Ops: Remediated service-y (Tier 3)" \
  -b "Root cause: Database connection pool exhausted due to leaked connections.
Actions taken:
  1. Restarted PostgreSQL
  2. Waited for connection recovery
  3. Restarted service-y
  4. Verified health check passes
Verification: All services healthy, connection count normal.
Recommendations: Review connection pool settings, add connection timeout." \
  "$CLAUDEOPS_APPRISE_URLS"

# Not fixed
apprise -t "Claude Ops: NEEDS HUMAN ATTENTION -- service-y" \
  -b "Issue: Persistent database corruption.
Investigation: Data directory has corrupted WAL files.
Attempted: Controlled restart, checked disk space (OK).
Why it failed: WAL corruption requires manual pg_resetwal.
Recommended next steps: SSH to host, run pg_resetwal, verify data integrity.
Current state: PostgreSQL down, all dependent services degraded." \
  "$CLAUDEOPS_APPRISE_URLS"

Key Decisions​

Why Apprise over direct API integrations​

Direct API integrations (Option 2 in the ADR) would avoid the Python dependency but require:

  • Writing and maintaining a separate curl command for each notification service
  • Managing service-specific credential formats (webhook URLs, SMTP credentials, API tokens)
  • Duplicating notification logic across tier prompts for each supported service
  • Increasing prompt complexity proportional to the number of supported services

Apprise abstracts all of this behind a single CLI invocation and URL-based configuration, keeping the prompts service-agnostic.

Why CLI invocation instead of library usage​

Claude Ops is a runbook system where the agent executes shell commands from markdown instructions. Using Apprise as a Python library would require either:

  • Writing Python scripts that the agent calls (introducing application code into a no-code runbook)
  • Having the agent write inline Python (fragile and error-prone)

CLI invocation aligns with the same pattern used for curl, docker, dig, and all other operations -- the agent constructs a shell command and executes it via Bash.

Why a single environment variable​

Each alternative approach evaluated in the ADR required more complex configuration:

  • Direct API integrations: per-service environment variables
  • AWS SNS: IAM credentials, topic ARNs, subscription management
  • Webhook approach: relay service configuration

A single comma-separated environment variable means operators can configure notifications by adding one line to their .env file. Adding a new notification channel is appending a URL to an existing string.

Why graceful degradation over required configuration​

Notifications are an operational convenience, not a core function. Claude Ops' primary job is to detect and remediate infrastructure issues. Making notifications a hard requirement would mean:

  • New deployments fail until notification is configured
  • Test environments need notification targets
  • Air-gapped environments without outbound notification access cannot run Claude Ops

Silent no-op when unconfigured ensures Claude Ops works immediately out of the box, with notifications as an opt-in enhancement.

Trade-offs​

Gained​

  • Universal coverage: 80+ notification services supported without any per-service code or configuration blocks. Operators choose their preferred service by specifying its URL.
  • Zero-code integration: Notifications are CLI commands in markdown prompts. No application code, no Python scripts, no compiled modules.
  • Configuration simplicity: One environment variable. Add a channel by appending a URL. Remove a channel by deleting a URL. No image rebuilds.
  • Service independence: Apprise runs entirely within the container. No external relay services, message queues, or cloud provider dependencies.
  • Graceful opt-in: Notifications work when configured and silently disappear when not, with no error states or required setup.

Lost​

  • Python dependency: The Docker image carries python3, pip3, and Apprise's Python dependency tree in what is otherwise a Node.js container. This increases image size.
  • No delivery guarantees: If Apprise fails to deliver (network error, misconfigured URL, service outage), there is no retry, no dead letter queue, no delivery confirmation. The notification is lost.
  • URL syntax learning curve: Each notification service has its own Apprise URL scheme with service-specific parameters. Operators must consult Apprise documentation to construct valid URLs.
  • Community dependency: Apprise is community-maintained open-source software without commercial support. A breaking change, abandonment, or security vulnerability would require migration to an alternative.
  • No priority/urgency differentiation: All notifications use the same Apprise invocation. There is no built-in mechanism to send human-attention alerts at a higher priority than daily digests, unless the notification service supports priority via URL parameters.

Future Considerations​

Priority-based routing​

A future enhancement could support routing different notification categories to different targets:

  • CLAUDEOPS_APPRISE_DIGEST_URLS for daily digests (low priority, email)
  • CLAUDEOPS_APPRISE_ALERT_URLS for human attention alerts (high priority, PagerDuty)
  • CLAUDEOPS_APPRISE_URLS as the fallback for all categories

This would allow operators to avoid alert fatigue by routing routine digests to email while sending critical alerts to paging systems.

Notification templating​

The current design has each tier's prompt file define the notification message format inline. A future evolution could introduce notification templates (markdown files in a notifications/ directory) that standardize message format across tiers and make the format independently configurable.

Delivery status tracking​

If notification reliability becomes critical, a future enhancement could:

  • Log each Apprise invocation's exit code and stderr to the results directory
  • Track notification delivery success rates over time
  • Alert on sustained notification delivery failures (meta-notification problem)

Retry with backoff​

For environments where notification delivery is critical, a simple retry mechanism could be added:

  • Retry failed Apprise invocations up to 3 times with exponential backoff
  • This could be implemented as a wrapper script rather than modifying Apprise itself
  • Only worth pursuing if notification loss proves to be a real operational problem