Synchronicity Detection

Synchronicity detection discovers meaningful temporal co-occurrence patterns across anonymized user profiles. Inspired by Carl Jung’s concept of synchronicity — “meaningful coincidences” that are not causally related — this subsystem finds patterns that emerge organically from collective tool usage.

Synchronicity detection uses pure SQL analysis with no LLM calls. It operates entirely on the temporal_events table and stores discovered patterns in gut_patterns with pattern_type='synchronicity'.

How It Works

Temporal Event Collection

Every tool call and observation is recorded as a temporal event. Events are collected in a fire-and-forget manner — failures never block the critical path.

User invokes tool
      |
      v
Record temporal event (async, non-blocking)
      |
      v
temporal_events table:
  - profile_hash (HMAC-SHA256, never raw UUID)
  - tool_name
  - event_type
  - outcome (success/failure/null)
  - context_hash (optional)
  - created_at

Only profile_hash is stored — never the raw profile UUID. This ensures that individual usage patterns cannot be traced back to specific users, even by database administrators.

Three Analysis Types

The synchronicity detector runs three independent analyses in parallel:

Co-occurrence
Failure Correlation
Emergent Workflows

What it finds: Tool pairs that are frequently used together within a short time window.Algorithm:

Identify active tools (those with >= 10 events in the last 30 days)
Use SQL window functions (LEAD) to find sequential tool pairs within the same profile session
Filter to pairs where the gap between uses is less than 5 minutes
Group by tool pair and count distinct profiles
Only surface patterns observed by 3+ unique profiles (k-anonymity)

Example output: “After using pluggedin_memory_search, users frequently use pluggedin_memory_details (47 profiles)”SQL core logic:

WITH sequences AS (
  SELECT
    tool_name,
    LEAD(tool_name) OVER (
      PARTITION BY profile_hash
      ORDER BY created_at
    ) as next_tool,
    LEAD(created_at) OVER (PARTITION BY profile_hash ORDER BY created_at) - created_at as gap
  FROM temporal_events
  WHERE created_at > NOW() - INTERVAL '30 days'
)
SELECT tool_name, next_tool,
       COUNT(DISTINCT profile_hash) as unique_profiles
FROM sequences
WHERE gap < INTERVAL '5 minutes'
GROUP BY tool_name, next_tool
HAVING COUNT(DISTINCT profile_hash) >= 3

What it finds: Tools that have high failure rates at specific times of day or days of the week.Algorithm:

Group events by tool, day-of-week, and hour-of-day
Calculate failure rate per group
Filter to groups where failure rate exceeds 50%
Only surface patterns from 3+ unique profiles

Example output: “github_api_search has 73% failure rate on day 1 (Monday) hour 9 (15 profiles, 42 events)”Use cases:

Detecting API rate-limit windows
Identifying maintenance-related outages
Finding time-of-day reliability patterns

What it finds: Three-tool sequential patterns that recur across multiple profiles.Algorithm:

Use SQL LEAD with offset 1 and 2 to capture three-step sequences
Filter to sequences where total gap is under 15 minutes
Group by tool triple and count distinct profiles
Only surface workflows from 3+ unique profiles

Example output: “Common workflow: pluggedin_search_documents -> pluggedin_get_document -> pluggedin_update_document (12 profiles)”Use cases:

Discovering implicit best-practice workflows
Identifying automation opportunities
Building workflow templates from observed behavior

Pattern Storage

Discovered patterns are stored as gut_patterns entries:

Generate Description

Each pattern is formatted into a human-readable description.

Hash Check

The description is HMAC-SHA256 hashed and checked against existing patterns to avoid duplicates.

Embedding Generation

A vector embedding is generated for the pattern description to enable semantic search.

Store in gut_patterns

New patterns are inserted with pattern_type='synchronicity', initial confidence, and the anonymized profile count.

Reinforce Existing

If the pattern already exists, its occurrence_count is incremented and the profile count is updated.

Concurrency Protection

Synchronicity detection uses PostgreSQL advisory locks to prevent concurrent runs:

// Only one detection can run at a time
const lockResult = await db.execute(
  sql`SELECT pg_try_advisory_lock(738203) as acquired`
);
if (!lockResult.acquired) {
  return { error: 'Synchronicity detection already running' };
}

TABLESAMPLE for Scale

When the temporal events table exceeds 1,000,000 rows, the detector automatically uses TABLESAMPLE BERNOULLI(1%) to sample the data. This keeps analysis time constant regardless of table size while maintaining statistical accuracy.

Privacy Model

Profile Hashing

Raw profile UUIDs are hashed with HMAC-SHA256 before storage. The hash function uses a server-side secret key.

k-Anonymity

Patterns are only surfaced when observed by 3+ unique profile hashes. Individual behaviors cannot be singled out.

Retention Cleanup

Events older than 90 days (configurable) are automatically deleted via the cleanup cron endpoint.

API Reference

Record Temporal Events

POST /api/memory/temporal-events
Authorization: Bearer <api_key>
Content-Type: application/json

{
  "events": [
    {
      "toolName": "pluggedin_search_documents",
      "eventType": "tool_call",
      "outcome": "success"
    },
    {
      "toolName": "pluggedin_get_document",
      "eventType": "tool_call",
      "outcome": "success"
    }
  ]
}

Response:

{
  "success": true,
  "data": { "inserted": 2 }
}

In most cases, temporal events are recorded automatically by the memory system during observations. You only need to call this endpoint directly if building custom integrations.

Cleanup Old Events

POST /api/memory/temporal-events/cleanup
Authorization: Bearer <api_key>

Response:

{
  "success": true,
  "data": { "deleted": 1247 }
}

Trigger Synchronicity Detection

POST /api/memory/sync/detect
Authorization: Bearer <api_key>

Response:

{
  "success": true,
  "data": {
    "coOccurrences": [
      {
        "analysisType": "co_occurrence",
        "toolName": "pluggedin_memory_search",
        "relatedTool": "pluggedin_memory_details",
        "uniqueProfiles": 47
      }
    ],
    "failureCorrelations": [
      {
        "analysisType": "failure_correlation",
        "toolName": "github_api_search",
        "dayOfWeek": 1,
        "hourOfDay": 9,
        "failureRate": 0.73,
        "total": 42,
        "uniqueProfiles": 15
      }
    ],
    "emergentWorkflows": [
      {
        "analysisType": "emergent_workflow",
        "toolName": "search_documents",
        "relatedTool": "get_document",
        "thirdTool": "update_document",
        "uniqueProfiles": 12
      }
    ],
    "patternsCreated": 5
  }
}

Get Detected Patterns

GET /api/memory/sync/patterns
Authorization: Bearer <api_key>

Response:

{
  "success": true,
  "data": [
    {
      "uuid": "abc-123",
      "patternType": "synchronicity",
      "description": "After using pluggedin_memory_search, users frequently use pluggedin_memory_details (47 profiles)",
      "occurrenceCount": 3,
      "uniqueProfileCount": 47,
      "confidence": 0.5,
      "metadata": {
        "source": "synchronicity",
        "analysis_type": "co_occurrence"
      }
    }
  ]
}

SDK Usage

import { PluggedInClient } from 'pluggedinkit-js';

const client = new PluggedInClient({ apiKey: 'your-api-key' });

// Get synchronicity patterns
const patterns = await client.jungian.getSynchronicityPatterns();
for (const p of patterns) {
  console.log(`${p.description} (${p.uniqueProfileCount} profiles)`);
}

from pluggedinkit import PluggedInClient

client = PluggedInClient(api_key="your-api-key")

# Get synchronicity patterns
patterns = client.jungian.get_synchronicity_patterns()
for p in patterns:
    print(f"{p.description} ({p.unique_profile_count} profiles)")

client := pluggedinkit.NewClient("your-api-key")
ctx := context.Background()

patterns, err := client.Jungian.GetSynchronicityPatterns(ctx)
if err != nil {
    log.Fatal(err)
}
for _, p := range patterns {
    fmt.Printf("%s (%d profiles)\n", p.Description, p.UniqueProfileCount)
}

Configuration

Variable	Default	Description
`SYNC_RETENTION_DAYS`	`90`	Temporal event retention period
`SYNC_COOCCURRENCE_WINDOW_DAYS`	`30`	Lookback for co-occurrence
`SYNC_COOCCURRENCE_GAP_MINUTES`	`5`	Max gap between co-occurring tools
`SYNC_FAILURE_WINDOW_DAYS`	`90`	Lookback for failure correlation
`SYNC_WORKFLOW_WINDOW_DAYS`	`30`	Lookback for emergent workflows
`SYNC_WORKFLOW_GAP_MINUTES`	`15`	Max total gap for three-step workflows
`SYNC_MIN_EVENTS_THRESHOLD`	`10`	Minimum events for a tool to be included
`SYNC_ACTIVE_TOOLS_LIMIT`	`200`	Max tools analyzed per run
`SYNC_TABLESAMPLE_PERCENT`	`1`	Sampling percentage for large tables
`SYNC_TABLESAMPLE_TRIGGER_ROWS`	`1000000`	Row count that triggers sampling
`SYNC_CRON_ENABLED`	`true`	Enable/disable the detection cron

Getting Started

Platform Features

PAP Agents

Guides

Tutorials

Security

Deployment

Observability

Testing

Migration Guides

Troubleshooting

Advanced Configuration

Integration Guides

Performance

Synchronicity Detection

Synchronicity Detection

How It Works

Temporal Event Collection

Three Analysis Types

Pattern Storage

Concurrency Protection

TABLESAMPLE for Scale

Privacy Model

Profile Hashing

k-Anonymity

Retention Cleanup

API Reference

Record Temporal Events

Cleanup Old Events

Trigger Synchronicity Detection

Get Detected Patterns

SDK Usage

Configuration

Next Steps

Archetype System

Jungian Intelligence Overview

​Synchronicity Detection

​How It Works

​Temporal Event Collection

​Three Analysis Types

​Pattern Storage

​Concurrency Protection

​TABLESAMPLE for Scale

​Privacy Model

Profile Hashing

k-Anonymity

Retention Cleanup

​API Reference

​Record Temporal Events

​Cleanup Old Events

​Trigger Synchronicity Detection

​Get Detected Patterns

​SDK Usage

​Configuration

​Next Steps

Archetype System

Jungian Intelligence Overview

Synchronicity Detection

How It Works

Temporal Event Collection

Three Analysis Types

Pattern Storage

Concurrency Protection

TABLESAMPLE for Scale

Privacy Model

API Reference

Record Temporal Events

Cleanup Old Events

Trigger Synchronicity Detection

Get Detected Patterns

SDK Usage

Configuration

Next Steps