Advanced Memory Plan

Goal

Turn ai-memory-hub from a searchable conversation and fact store into a richer long-term memory system while preserving the current local-first, deterministic, MCP-native contract.

Advanced memory should remain optional and layered. Raw conversations, normalized facts, generated summaries, vector chunks, and future graph records must keep clear provenance so answers can explain what they are based on.

Scope

[ ] Add graph-style memory for entities and relationships.
[ ] Add relevance decay, importance scoring, pinning, and forgetting workflows.
[ ] Add shared memory spaces and agent-specific filters on top of existing project isolation.
[ ] Add plugin extension points for custom import, enrichment, embedding, and storage behavior.
[ ] Keep advanced memory disabled or conservative until entity, relationship, and retrieval quality can be measured with tests and representative data.

Non-Goals

[ ] Do not replace raw conversation storage with generated memory.
[ ] Do not let graph, decay, or plugin behavior bypass bearer-token ownership and project-membership checks.
[ ] Do not add agent-facing mutation tools for deletion or rewriting; admin mutation workflows remain separate.
[ ] Do not require cloud services for advanced memory features.

Phase 1: Readiness And Data Contracts

[x] Inventory existing memory layers: conversations, chunks, facts, generated summaries, projects, users, and vector metadata.
[x] Define stable IDs and provenance links between derived records and their source conversations, messages, chunks, facts, and summaries.
[x] Add explicit metadata models for derived-memory record type, extractor version, confidence, source scope, and review status.
[x] Define capability flags so providers can report whether they support graph records, decay fields, shared scopes, and plugin-managed metadata.
[x] Add migration and rollback notes for SQLite, Postgres, and MongoDB metadata stores before any schema change lands.

Acceptance criteria:

[x] Every advanced-memory record can point back to source evidence.
[x] Provider support is explicit instead of inferred from provider names.
[x] Existing API, MCP, CLI, and storage behavior remains backward compatible.

Migration notes:

SQLite and Postgres add graph and review tables only in the later storage phase. Rollback can drop those additive tables because raw conversations, chunks, facts, summaries, users, and projects remain the source of truth.
MongoDB remains conservative for graph, decay, and plugin-managed metadata until its document shapes and indexes have contract tests. Rollback is equivalent to leaving advanced-memory collections unread.

Phase 2: Entity Extraction And Memory Graph

[x] Add optional entity extraction behind a feature flag or module boundary.
[x] Start with entity types that already matter to the project: people, projects, tools, providers, organizations, decisions, preferences, and owned items.
[x] Store entities separately from normalized facts while linking entities to facts and source messages.
[x] Add relationship records with predicate, subject, object, confidence, source provenance, and freshness fields.
[x] Add relationship supersession or conflict handling that mirrors fact-layer correction behavior.
[x] Add graph inspection commands for local review before exposing graph data through agent-facing APIs.

Acceptance criteria:

[x] Entity and relationship extraction is deterministic or explicitly marked as generated by a named extractor.
[x] Relationship records never become answer evidence without provenance.
[x] Conflicting relationships are surfaced cautiously instead of silently merged.

Implementation note:

memory.graph_enabled defaults to false. When enabled, deterministic graph extraction stores separate entity and relationship records for local review through aim graph-entities and aim graph-relationships. These records are not exposed through MCP/API answer surfaces in this phase.

Phase 3: Entity And Graph Quality Metrics

Use graph_quality_gate_plan.md as the source of truth for the benchmark, metric definitions, thresholds, and graph-retrieval promotion rule.

[x] Create a representative graph-memory evaluation corpus with source conversations, expected entities, expected relationships, and expected source provenance.
[x] Measure entity extraction precision, recall, and F1 by entity type.
[x] Measure relationship extraction precision, recall, and F1 by relationship predicate.
[x] Measure provenance accuracy: extracted entities and relationships should point to the correct source conversation, message, chunk, or fact.
[x] Track conflict detection quality for contradictory relationships and superseded facts.
[x] Add thresholded evaluation output similar to memory.benchmarks.retrieval_quality, with explicit failure reasons.
[x] Define minimum quality gates before graph records can influence retrieval ranking.

Suggested initial quality gates:

[x] Entity precision should be high enough that graph expansion does not add noisy candidates to normal search.
[x] Relationship precision should be stricter than entity precision because a wrong edge can create a misleading answer path.
[x] Provenance accuracy should be near-perfect for promoted graph records.
[x] Graph-aware retrieval must meet or beat baseline retrieval on relationship questions without regressing ordinary semantic queries.

Acceptance criteria:

[x] A local benchmark can report entity, relationship, provenance, and graph-retrieval metrics without network dependencies.
[x] Quality threshold failures are explicit and actionable.
[x] Graph retrieval remains disabled until the benchmark passes agreed gates.

Phase 4: Graph-Aware Retrieval

[x] Add graph lookup as a retrieval candidate source only after entity, relationship, provenance, and graph-retrieval quality gates pass.
[x] Use graph matches to expand or rerank candidate memories, not to replace vector, keyword, fact, or summary retrieval.
[x] Add result diagnostics showing when graph expansion influenced ranking.
[x] Add graph-aware memory_ask answer bases only if they preserve the current confidence and provenance shape.
[x] Add representative retrieval evaluation cases for graph questions such as "Who works on this project?", "Which tools are tied to this decision?", and "What changed about this preference?"

Acceptance criteria:

[x] Graph retrieval improves targeted relationship questions without reducing precision for ordinary semantic search.
[x] Graph-expanded answers include compact provenance and confidence reasons.
[x] Graph features can be disabled without changing non-graph result shapes.

Implementation note:

Graph retrieval is gated by retrieval.graph_enabled and retrieval.graph_quality_gate_passed. When active, graph relationships add normal conversation candidates and memory_search returns graph diagnostics. memory_ask keeps the current direct_memory answer basis until a separate graph-specific answer contract is warranted.

Phase 5: Relevance Decay, Importance, And Pinning

[x] Define scoring inputs: recency, access frequency, explicit pinning, correction status, confidence, source quality, project activity, and user feedback.
[x] Store decay and importance signals as metadata, not destructive changes to raw memories.
[x] Add pinned memory behavior for facts, conversations, summaries, and graph relationships that should resist decay.
[x] Add stale-memory diagnostics for facts and relationships that are old, low-confidence, or contradicted.
[x] Add tests proving decay changes ranking only within bounded, explainable limits.

Acceptance criteria:

[x] Recent or important records can rank higher without hiding highly relevant older evidence.
[x] Pinned records remain discoverable.
[x] Scoring explanations disclose decay and importance effects.

Implementation note:

retrieval.advanced_scoring_enabled defaults to false. When enabled, the ranker reads bounded signals from metadata.advanced_memory, including pinned, importance, access_count, confidence, updated_at, and contradicted. The scoring helper caps boosts so metadata can nudge ranking without replacing lexical/vector relevance.

Phase 6: Forgetting And Review Workflows

[x] Define "forget" as admin-controlled hiding, archival, or purge semantics with audit records.
[x] Keep default conversation data immutable unless an admin workflow explicitly hides or purges it.
[x] Add review queues for low-confidence facts, conflicting relationships, stale generated summaries, and candidate redundant memories.
[x] Add duplicate and redundant memory consolidation workflows after summary provenance and fact supersession remain stable.
[x] Add export-before-purge guidance for local-first users.

Acceptance criteria:

[x] Users can reduce noisy memory without losing auditability by default.
[x] Forgetting behavior is consistent across metadata and vector providers.
[x] No agent-facing MCP tool can silently delete or rewrite memory.

Implementation note:

Phase 6 adds typed review queue and forget-audit contracts. It does not expose MCP/API mutation tools for deletion or rewriting. Purge audit records carry export_recommended: true so local-first users get explicit export-before-purge guidance in future admin workflows.

Phase 7: Shared Memory And Agent-Specific Filters

[x] Build on existing users, projects, project memberships, and owner/project scoping.
[x] Add explicit shared-memory policies for private, project, team, and imported memory scopes.
[x] Add agent-specific filters for source, project, trusted capture path, allowed derived-memory types, and sensitivity class.
[x] Add tests for cross-user, cross-project, and cross-agent isolation.
[x] Keep default behavior private unless a user or admin explicitly shares a project or memory scope.

Acceptance criteria:

[x] Shared memory works only through explicit membership or policy.
[x] Agents can narrow what memory classes they can read.
[x] Isolation failures are covered by regression tests.

Implementation note:

Phase 7 adds SharedMemoryPolicy and AgentMemoryFilter contracts plus a helper for narrowing derived records by source, project, record type, sensitivity, and trusted capture status. The default visibility remains private.

Phase 8: Plugin Extension Points

[x] Define plugin interfaces only after importer, provider, and enrichment boundaries are stable.
[x] Start with custom capture/import adapters that emit the existing normalized conversation schema.
[x] Add custom ingestion enrichers for entity, relationship, summary, or fact extraction with explicit provenance and versioning.
[x] Add custom embedding providers through existing provider configuration patterns.
[x] Add custom vector or metadata stores only if they can satisfy the provider contract tests.
[x] Document plugin safety expectations for secrets, PII, payload logging, and deterministic test fixtures.

Acceptance criteria:

[x] Plugins cannot bypass validation, auth, ownership, or project boundaries.
[x] Plugin output is testable and auditable.
[x] Core users can ignore plugins entirely.

Implementation note:

memory.plugin_contracts defines protocol-style extension points for capture/import adapters, ingestion enrichers, and embedding-provider plugins. Validators reject owner/project/auth fields, secrets, raw browser artifacts, and malformed enrichment shapes before plugin output can reach core ingestion. No dynamic plugin loading is enabled by default.

Testing

[x] Unit tests for entity and relationship model validation.
[x] Entity and graph quality benchmark tests with threshold failure cases.
[x] Metadata-store contract tests for graph, decay, review, and shared-scope fields.
[x] Retrieval tests comparing baseline search with graph-aware retrieval.
[x] Regression tests for fact and relationship conflicts.
[x] Auth and project-isolation tests for shared memory and agent filters.
[x] CLI/API/MCP shape tests for any advanced-memory public surface.
[x] Representative-data evaluations for graph quality, decay ranking, and consolidation behavior.

Done When

[x] Advanced memory records have clear provenance and confidence.
[x] Graph-aware retrieval measurably helps relationship questions.
[x] Decay and importance improve ranking without hiding relevant older memory.
[x] Shared memory remains explicitly permissioned and private by default.
[x] Plugin extension points are stable, documented, and covered by contract tests.