Design Document: Knowledge Graph Indexer (KGI)

1. Introduction and Goals

This is a system specification and design document that details how the application, “Knowledge Graph Indexer” (KGI), will be built.

1.1 Project Goal

To build a modular, local application that monitors and processes web browsing history and external content (bookmarks, crawls) to construct a semantic knowledge graph of the user’s informational landscape. The goal is to provide actionable insights into research patterns, topical knowledge, and connected ideas over time.

1.2 Scope & Deliverables

The initial scope includes the core indexing engine, the local database setup, and the implementation of basic input adapters (Browser Extension, Local File Loader). Future functionality will be managed via the defined Plugin Architecture.

1.3 Success Criteria

  • High fidelity in capturing metadata and structured content.
  • Ability to define and calculate relationships (Edges) based on multiple criteria (time, topic, tag).
  • The system must maintain an API layer to allow seamless addition of new data sources (Plugins) without modifying core logic.

2. Technical Architecture (High Level)

The KGI will follow a Microservices/Modular Architecture pattern, separating the concerns of Data Capture, Data Processing, and Data Storage.

2.1 System Flow Diagram

[Input Sources] -> [Ingestion Layer] -> [Processing/Semantic Engine] -> [Database (Graph Store)] -> [API/User Interface]

2.2 Core Components

Component Role Description Technologies (Suggested)
Input Adapters Data Collection Captures data from external sources (Browser, Bookmarks, Scrapy). Converts raw source data into a standardized internal format (JSON). Browser API, Python Scripts, REST API Hooks
Message Queue/Bus Decoupling Handles asynchronous data ingestion. Prevents input failure from crashing the processing engine. Redis, RabbitMQ
Processing Engine Logic Core Cleans, enriches, and analyzes the raw data. This is where NLP and graph building rules are applied. Python (SpaCy, NLTK), ML Models
Knowledge Graph Store Persistence Stores all nodes (entities) and edges (relationships). Must support graph traversal queries. Neo4j (Recommended Graph DB)
Plugin Manager Extensibility Manages loading, execution, and lifecycle of external plugins. Python Entry Points, Java Service Loader

3. Data Modeling and Schema

The core of the application is the graph structure. We must define two primary components: Nodes (Entities) and Edges (Relationships).

3.1 Storage Choice

Recommendation: Neo4j (or similar dedicated Graph Database). This is vastly superior to relational databases for complex relationship querying.

3.2 Schema Definition

A. Node (:Website or :Entity) Represents a single piece of visited data (the source).

Property Data Type Description Source
uri String (Unique) The canonical URL of the page. All Inputs
timestamp DateTime Time of capture/visit. Input Adapter
title String Page title (metadata). Input Adapter
abstract Text A summary or parsed description. Processing Engine
raw_metadata JSON Original metadata blob (for debugging/future use). Input Adapter

B. Node (:Concept or :Tag) Represents abstract, reusable ideas.

Property Data Type Description Source
concept_id String (Unique) Internal ID generated by the NLP model. Processing Engine
name String The human-readable concept name (e.g., “Quantum Physics”). Processing Engine
definition Text Description derived from the concept’s usage. Processing Engine

C. Edge (Relationship) Defines how Nodes are connected.

Relationship Type Source Node Target Node Properties Connection Rule
VISITED_BEFORE :Website :Website sequence_order (integer) Defines browsing sequence.
IS_TOPIC :Website :Concept confidence_score (float) Links a page to a semantic concept (via NLP).
IS_RELATED_TO :Concept :Concept strength (float) Links two concepts if they appear frequently together.
HAS_TAG :Website :Tag confidence_score (float) Links a page to a user-defined or extracted tag.

4. Core Feature Implementation Details

4.1 Metadata Retrieval (The Observer)

  • Interface: Browser Extension (e.g., Chrome/Firefox).
  • Functionality: On exit or interval trigger, the extension captures the following data points: URL, Title, Timestamp, and ideally, a small Text Snippet of the current page.
  • Goal: Minimize overhead while ensuring capture fidelity.

4.2 Indexing and Semantic Processing (The Brain)

This module is responsible for transforming raw data into structured graph relationships.

  1. Metadata Parsing:
    • Extract structured data (Schema.org JSON-LD).
    • Use libraries (e.g., BeautifulSoup) to scrape content, prioritizing <article>, <p>, and structured lists.
  2. Semantic Analysis (Topic/Concept):
    • Run the extracted text against NLP Models (e.g., using SpaCy or BERT embeddings).
    • Named Entity Recognition (NER): Identify people, organizations, and places.
    • Topic Modeling: Use LDA or similar models to group related vocabulary into abstract :Concept nodes.
    • Relationship Inference: If Concept A and Concept B appear in the same document, create a potential IS_RELATED_TO edge between them.
  3. Tagging:
    • Auto-Tagging: Generate tags based on the top 5 most frequent NER entities or keywords.
    • Manual Overrides: Provide a UI for the user to manually set tags for better control.

4.3 Modular Design and Plugin System (The Extensibility)

The core system must expose a defined Python/Java API for plugins.

Plugin Architecture Requirements: 1. Interface Definition: Every plugin must adhere to a standard interface (e.g., IInputAdapter). 2. Core Methods: The interface must define at least connect() and process(data_batch). 3. Isolation: Plugins must run in an isolated environment to prevent one plugin failure from crashing the core engine.

Plugin 1: Bookmark Importer

  • Input: Local JSON/HTML file of bookmarked URLs, potentially including hierarchical structure (folder names).
  • Processing: Maps the bookmark hierarchy into a pseudo-sequence (e.g., Bookmarked Folder -> Topic A -> Specific URL). This strengthens the VISITED_BEFORE relationships by inferring intent.

Plugin 2: WebCrawler Interface (Scrapy Integration)

  • Input: A batch of URLs generated by Scrapy (or any other crawler).
  • Process: The plugin must retrieve the full HTML/content of the target URLs, submit them to the core Processing Engine for semantic analysis, and save the resulting Nodes/Edges.
  • Mechanism: The plugin submits a list of URIs to the KGI’s queue, which then triggers the processing engine.

5. Runtime Workflow

This diagram illustrates how data moves from the input sources through the processing engine to the graph database, based on the “Microservices/Modular Architecture” described in Section 2 and 4.

graph TD
    %% STYLES
    classDef input fill:#e1f5fe,stroke:#01579b,stroke-width:2px;
    classDef queue fill:#fff9c4,stroke:#fbc02d,stroke-width:2px;
    classDef process fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px;
    classDef store fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px;
    classDef ui fill:#eceff1,stroke:#455a64,stroke-width:2px;
    classDef plugin fill:#ffffff,stroke:#9e9e9e,stroke-dasharray: 5 5;

    subgraph INPUT_LAYER ["Input Sources (Adapters)"]
        direction TB
        BExt[("Browser Extension<br/>(JS/Metadata)")]:::input
        PlugIn[("Plugin System")]:::plugin
        Bkmrk[("Bookmark Importer")]:::input
        Crawl[("Scrapy Crawler")]:::input
        
        BExt -->|JSON Payload| API
        Bkmrk & Crawl -->|JSON Payload| PlugIn
    end

    subgraph INGESTION_LAYER ["Ingestion Layer"]
        API["REST API Endpoint<br/>(Ingestion Gateway)"]:::input
        MQ[("Message Queue<br/>(Redis)")]:::queue
        
        API -->|Async Publish| MQ
    end

    subgraph PROCESSING_ENGINE ["Processing Engine (The Brain)"]
        Worker["Queue Worker"]:::process
        
        subgraph NLP_PIPELINE ["NLP & Semantic Pipeline"]
            Parser["Content Parser<br/>(BeautifulSoup)"]:::process
            NLP["NLP Engine<br/>(SpaCy/BERT)"]:::process
            Logic["Graph Logic Builder"]:::process
        end
    
        MQ -->|Pull Task| Worker
        Worker --> Parser
        Parser -->|Cleaned Text| NLP
        NLP -->|Entities/Topics| Logic
    end

    subgraph STORAGE_LAYER ["Storage & Graph"]
        GDB[("Neo4j Graph DB")]:::store
        
        Logic -->|Create Nodes/Edges| GDB
        
        subgraph SCHEMA ["Graph Schema"]
            NodeWeb[":Website Node"]
            NodeCon[":Concept Node"]
            EdgeRel["Relationships<br/>(VISITED_BEFORE, IS_TOPIC)"]
        end
    end

    subgraph PRESENTATION_LAYER ["User Interface"]
        UI["Frontend UI<br/>(React/Vue)"]:::ui
        
        GDB -->|Graph Queries| UI
        UI -.->|User Feedback/Tags| API
    end

    %% Connection for Plugin
    PlugIn -->|Standard Interface| API

6. Technical Stack Recommendation

Layer Purpose Recommended Technology Rationale
Backend/Core Logic Processing Engine, API endpoints Python 3.x Excellent ecosystem for NLP (SpaCy, NLTK), and rapid prototyping.
Database Graph Storage Neo4j Dedicated graph database, optimized for traversing relationships (the core function of the application).
Messaging/Queue Asynchronous communication Redis Lightweight, fast, and perfect for handling input queues from multiple sources.
Frontend/UI Visualization, Interaction React / Vue.js Modern framework for building responsive user interfaces.
Browser Extension Input Adapter JavaScript / Manifest V3 Standard technology for browser integration.
Crawling Plugin Input Adapter Scrapy (Python) Industry standard for robust web scraping in Python.

7. Implementation Roadmap (Phases)

Phase Goal Key Deliverables Focus Area
Phase 1: Minimum Viable Product (MVP) Capture and Store basic metadata. Browser Extension (Text Snippet, URL, Time). Basic Node/Edge structure (URL \(\to\) Website). Local API endpoint. Data Input, Basic Storage
Phase 2: Core Intelligence Add semantic depth and structured connections. Semantic Processing Engine (NLP/Topic Modeling). Graph creation logic. Relationship inference (IS_TOPIC). Processing Engine, Graph Modeling
Phase 3: Modularization & Input Expansion Make the system truly pluggable. Implement Plugin Manager API. Bookmark Importer Plugin. Internal WebCrawler (Dummy) endpoint. Architecture, Extensibility
Phase 4: Advanced Plugin Integration Full feature set and robustness. Full Scrapy Plugin implementation. User interface for reviewing/correcting generated tags and concepts. Plugin Depth, UI/UX 

flowchart TD
    %% STYLES
    classDef phase fill:#f9fbe7,stroke:#827717,stroke-width:2px;
    classDef task fill:#ffffff,stroke:#bdbdbd,stroke-width:1px;
    classDef check fill:#e0f2f1,stroke:#00695c,stroke-dasharray: 5 5;

    Start((Start Project)) --> P1

    subgraph P1 ["Phase 1: MVP (Data Input & Basic Storage)"]
        direction TB
        T1_1[/"Task: Setup Python/Neo4j Environment"\]:::task
        T1_2["Task: Build Browser Extension<br/>(Capture URL/Title/Time)"]:::task
        T1_3["Task: Create REST API Endpoint<br/>(Ingest Data)"]:::task
        T1_4["Task: Basic Node Creation<br/>(:Website nodes only)"]:::task
        
        Check1{"Success Criteria:<br/>Data flows from Browser to DB?"}:::check
        
        T1_1 --> T1_2 --> T1_3 --> T1_4 --> Check1
    end

    Check1 -->|Yes| P2

    subgraph P2 ["Phase 2: Core Intelligence (NLP & Graph Logic)"]
        direction TB
        T2_1["Task: Integrate NLP Library<br/>(SpaCy/NLTK)"]:::task
        T2_2["Task: Implement Topic Modeling<br/>(Create :Concept nodes)"]:::task
        T2_3["Task: Build Relationship Logic<br/>(IS_TOPIC, VISITED_BEFORE)"]:::task
        T2_4["Task: Implement Redis Queue<br/>(Async Processing)"]:::task
        
        Check2{"Success Criteria:<br/>Semantic relationships visible?"}:::check
        
        T2_1 --> T2_2 --> T2_3 --> T2_4 --> Check2
    end

    Check2 -->|Yes| P3

    subgraph P3 ["Phase 3: Modularization (Plugin Architecture)"]
        direction TB
        T3_1["Task: Define Plugin Interface<br/>(IInputAdapter)"]:::task
        T3_2["Task: Refactor Core to be Plugin-Agnostic"]:::task
        T3_3["Task: Build Bookmark Importer Plugin"]:::task
        
        Check3{"Success Criteria:<br/>Can add input source w/o<br/>changing core code?"}:::check
        
        T3_1 --> T3_2 --> T3_3 --> Check3
    end

    Check3 -->|Yes| P4

    subgraph P4 ["Phase 4: Advanced Integration & UI"]
        direction TB
        T4_1["Task: Build Scrapy Plugin"]:::task
        T4_2["Task: Develop React/Vue Frontend"]:::task
        T4_3["Task: Add User Feedback Loop<br/>(Manual Tagging/Correction)"]:::task
        
        Check4{"Success Criteria:<br/>Full System Operational?"}:::check
        
        T4_1 --> T4_2 --> T4_3 --> Check4
    end

    Check4 -->|Yes| Finish(((Deploy MVP)))

    %% Apply class styles
    class P1,P2,P3,P4 phase;

This diagram details the step-by-step process to build the application. This serves as the “Build Workflow.”

8. Open Questions / Assumptions

  • Privacy: Since this application handles highly sensitive browsing data, an explicit data anonymization and local-only storage strategy must be enforced.
  • Computational Overhead: Running full NLP models on every visited page may be resource-intensive. The system must implement a caching strategy (e.g., only re-process content if the cached version is older than X days, or if the page has changed significantly).
  • Ambiguity Resolution: When the NLP engine finds multiple possible meanings for a word (Polysemy), the system must use context (the surrounding text and other existing :Concept nodes) to select the most probable meaning.