> ## Documentation Index
> Fetch the complete documentation index at: https://docs.plugged.in/llms.txt
> Use this file to discover all available pages before exploring further.

# Observability Overview

> Production-grade observability stack for Plugged.in OAuth 2.1 implementation

# Observability Stack

Plugged.in implements a comprehensive observability stack for OAuth 2.1 operations, enabling real-time monitoring, security event detection, and performance analysis.

## Architecture

The observability stack consists of three pillars:

<CardGroup cols={3}>
  <Card title="Logs (Loki)" icon="file-lines">
    Structured JSON logs for all OAuth operations, security events, and errors
  </Card>

  <Card title="Metrics (Prometheus)" icon="chart-line">
    Real-time counters, histograms, and gauges for performance tracking
  </Card>

  <Card title="Dashboards (Grafana)" icon="chart-mixed">
    Unified visualization combining logs and metrics for insights
  </Card>
</CardGroup>

## System Diagram

```
┌─────────────────────────────────────────────────────────────┐
│                     pluggedin-app                           │
│                                                              │
│  ┌───────────────────┐         ┌──────────────────┐        │
│  │  OAuth Operations │────────▶│ Structured Logs  │        │
│  │  (token refresh,  │         │ (Pino + JSON)    │────┐   │
│  │   PKCE, etc.)     │         └──────────────────┘    │   │
│  └───────────────────┘                                  │   │
│           │                                             │   │
│           ▼                                             │   │
│  ┌───────────────────┐                                 │   │
│  │ Prometheus Metrics│──────────┐                      │   │
│  │ (prom-client)     │          │                      │   │
│  └───────────────────┘          │                      │   │
└──────────────────────────────────┼──────────────────────┼───┘
                                   │                      │
                    ┌──────────────▼──────┐     ┌────────▼────────┐
                    │   Prometheus        │     │   Promtail      │
                    │   (Metrics Store)   │     │  (Log Shipper)  │
                    └──────────┬──────────┘     └────────┬────────┘
                               │                         │
                               │          ┌──────────────▼────────┐
                               │          │     Loki              │
                               │          │   (Log Aggregation)   │
                               │          └──────────┬────────────┘
                               │                     │
                    ┌──────────▼─────────────────────▼────────┐
                    │              Grafana                     │
                    │  (Unified Dashboards & Alerting)         │
                    └──────────────────────────────────────────┘
```

## Key Features

### 📊 Comprehensive Metrics

17 Prometheus metrics track all OAuth operations:

* **OAuth Flows**: Initiation, completion, duration (counters + histograms)
* **Token Operations**: Refresh attempts, success rate, rotation tracking
* **PKCE Security**: State creation, validation, cleanup metrics
* **Security Events**: Code injection attempts, token reuse detection, integrity violations
* **Discovery**: RFC 9728 metadata discovery, success rates
* **Registration**: Dynamic client registration (RFC 7591)

### 📝 Structured Logging

All logs use JSON format for Loki compatibility:

* **OAuth Events**: Flow tracking, state transitions, token operations
* **Security Events**: Suspicious activity, attack detection, compliance violations
* **Performance**: Timing, duration, resource usage
* **Errors**: Detailed error context with stack traces

### 🔍 Automatic Redaction

Sensitive data is automatically redacted from logs:

* Access tokens, refresh tokens
* PKCE code verifiers
* Client secrets
* Authorization codes

## Quick Start

### Prerequisites

```bash theme={null}
# Ensure pluggedin-observability stack is running
cd /path/to/pluggedin-observability
docker-compose up -d
```

This starts:

* Prometheus ([http://localhost:9090](http://localhost:9090))
* Loki ([http://localhost:3100](http://localhost:3100))
* Grafana ([http://localhost:3000](http://localhost:3000))
* Promtail (log shipping)

### Environment Variables

Add to `pluggedin-app/.env`:

```bash theme={null}
# Observability Configuration
SERVICE_NAME=pluggedin-app
APP_VERSION=2.14.0
LOG_LEVEL=info  # trace, debug, info, warn, error

# Optional: Prometheus Push Gateway
PROMETHEUS_PUSH_GATEWAY=http://localhost:9091
```

### Verify Setup

1. **Check Logs**:

```bash theme={null}
# View JSON-formatted OAuth logs
docker logs pluggedin-app | grep oauth | jq .
```

2. **Check Metrics**:

```bash theme={null}
# Prometheus metrics endpoint
curl http://localhost:12005/metrics | grep oauth
```

3. **Query Loki**:

```bash theme={null}
# Query OAuth events from last hour
curl -G -s "http://localhost:3100/loki/api/v1/query_range" \
  --data-urlencode 'query={service_name="pluggedin-app"} |= "oauth"' \
  --data-urlencode "start=$(date -u -d '1 hour ago' +%s)000000000" \
  --data-urlencode "end=$(date -u +%s)000000000" | jq .
```

4. **Access Grafana**:

```
Open http://localhost:3000
Default credentials: admin/admin
```

## What to Monitor

### Critical Metrics

<Card title="Token Reuse Detection" icon="triangle-exclamation" color="#ff0000">
  **Metric**: `oauth_token_refresh_total{status="reuse_detected"}`

  **Alert when**: > 0

  **Action**: Immediate security review - indicates replay attack or race condition
</Card>

<Card title="Code Injection Attempts" icon="shield-exclamation" color="#ff6600">
  **Metric**: `oauth_code_injection_attempts_total`

  **Alert when**: > 0

  **Action**: Review security logs, block attacker IP, audit user accounts
</Card>

<Card title="OAuth Flow Success Rate" icon="chart-line" color="#00aa00">
  **Metric**: `oauth_flows_total{status="success"} / oauth_flows_total`

  **Alert when**: \< 95%

  **Action**: Investigate failures, check auth server connectivity
</Card>

<Card title="Token Refresh Duration" icon="clock" color="#0066ff">
  **Metric**: `oauth_token_refresh_duration_seconds`

  **Alert when**: p99 > 5s

  **Action**: Check network latency to auth servers, database performance
</Card>

### Security Events

Monitor these log events continuously:

* `oauth_refresh_token_reuse_detected` (P0 - Critical)
* `oauth_code_injection_attempt` (P0 - Critical)
* `oauth_integrity_violation` (P1 - High)
* `oauth_ownership_violation` (P1 - High)
* `pkce_replay_detected` (P1 - High)

## Log Levels

Configure based on environment:

```bash theme={null}
# Development
LOG_LEVEL=debug  # Verbose logging for debugging

# Staging
LOG_LEVEL=info   # Standard operational logging

# Production
LOG_LEVEL=warn   # Errors and warnings only (reduces volume)
```

## Performance Impact

The observability stack is designed for minimal overhead:

* **Logging**: \~1-2ms per operation (async I/O)
* **Metrics**: \~0.1ms per increment (in-memory counters)
* **Total**: \< 0.5% CPU overhead in production

## Multi-Instance Considerations

### Redis for Distributed Metrics

<Warning>
  **Production Requirement**: When running multiple application instances, use a **shared metrics backend** to aggregate data across instances.
</Warning>

**Options:**

1. **Prometheus Federation** (Recommended):
   ```yaml theme={null}
   # Each instance exposes /api/metrics
   # Prometheus scrapes all instances
   # Grafana queries aggregated data
   ```

2. **Shared Redis Backend**:
   ```bash theme={null}
   # Configure Redis for rate limiting
   export REDIS_URL="redis://redis-host:6379"

   # Rate limiting data is shared
   # Each instance contributes metrics
   ```

### Performance Optimizations

#### OAuth Config Caching

Each instance maintains an LRU cache (5-minute TTL):

```typescript theme={null}
// Automatic caching reduces database load
// Cache invalidates on config updates
// Max 500 cached configurations per instance
```

**Monitoring:**

* Cache hit rate: `oauth_config_cache_hits / oauth_config_cache_requests`
* Database query reduction: \~80% for frequently used servers

#### Server Ownership Validation

Optimized JOIN query (60-70% faster):

```sql theme={null}
-- Single query replaces 3 sequential queries
SELECT projects.user_id
FROM mcp_servers
INNER JOIN profiles ON mcp_servers.profile_uuid = profiles.uuid
INNER JOIN projects ON profiles.project_uuid = projects.uuid
WHERE mcp_servers.uuid = $1
```

**Monitoring:**

* Query duration: `oauth_ownership_validation_duration_seconds`
* Expected p95: \< 50ms (was 150ms)

#### Request Timeouts

All OAuth API calls timeout after 10 seconds:

```typescript theme={null}
await fetch(endpoint, {
  signal: AbortSignal.timeout(10000)
});
```

**Monitoring:**

* Timeout events: `oauth_request_timeout_total`
* Avg duration: `oauth_request_duration_seconds`

### Multi-Instance Deployment

For complete setup guide:

<Card title="Multi-Instance Deployment" icon="server" href="/deployment/multi-instance">
  Production-ready horizontal scaling with Redis, load balancing, and monitoring
</Card>

## Next Steps

<CardGroup cols={2}>
  <Card title="Log Queries" icon="magnifying-glass" href="/observability/logs">
    Learn LogQL queries for OAuth operations
  </Card>

  <Card title="Metrics & PromQL" icon="chart-line" href="/observability/metrics">
    Explore Prometheus metrics and queries
  </Card>

  <Card title="Grafana Dashboards" icon="chart-mixed" href="/observability/dashboards">
    Build custom dashboards and alerts
  </Card>

  <Card title="OAuth Security" icon="shield-halved" href="/security/oauth-2.1">
    OAuth 2.1 security implementation
  </Card>
</CardGroup>

## Troubleshooting

<AccordionGroup>
  <Accordion title="Metrics not appearing in Prometheus">
    1. Check metrics endpoint: `curl http://localhost:12005/metrics`
    2. Verify Prometheus config targets pluggedin-app
    3. Check Prometheus logs: `docker logs prometheus`
    4. Ensure app is generating OAuth traffic
  </Accordion>

  <Accordion title="Logs not in Loki">
    1. Verify JSON log format: `docker logs pluggedin-app | head -1 | jq .`
    2. Check Promtail config includes app log path
    3. Review Promtail logs: `docker logs promtail`
    4. Test Loki API: `curl http://localhost:3100/ready`
  </Accordion>

  <Accordion title="Grafana can't connect to data sources">
    1. Verify Prometheus URL: `http://prometheus:9090` (Docker network)
    2. Verify Loki URL: `http://loki:3100` (Docker network)
    3. Test connectivity: `docker exec grafana curl http://prometheus:9090/api/v1/status/config`
  </Accordion>

  <Accordion title="High log volume">
    1. Increase `LOG_LEVEL` to `warn` in production
    2. Configure log sampling in Promtail
    3. Set Loki retention policy (default: 30 days)
    4. Archive old logs to S3/GCS
  </Accordion>
</AccordionGroup>