High-Level Architecture¶

This document describes the system architecture for Pebble DMS.

System Overview¶

Pebble DMS is built as a modular, event-driven architecture designed for:

• Scale: Handle 1TB+ document collections
• Reliability: Zero data loss, async processing
• Extensibility: Easy to add new processors and integrations
• Performance: Fast ingestion and search

Architecture Diagram¶

flowchart TB
    subgraph Clients
        A[Web UI]
        B[REST API Clients]
        C[Batch Upload CLI]
    end

    subgraph API Layer
        D[API Gateway / Load Balancer]
        E[Auth Service]
    end

    subgraph Application Services
        F[Ingestion Service]
        G[Processing Orchestrator]
        H[Classification Service]
        I[Tagging Service]
        J[Search Service]
    end

    subgraph Processing Workers
        K[OCR Worker]
        L[Embedding Worker]
        M[Dedup Worker]
    end

    subgraph Message Queue
        N[Job Queue - Redis/RabbitMQ]
    end

    subgraph Data Stores
        O[(Document Store - S3/MinIO)]
        P[(Metadata DB - PostgreSQL)]
        Q[(Vector Store - Qdrant)]
        R[(Search Index - Meilisearch)]
    end

    A --> D
    B --> D
    C --> D
    D --> E
    E --> F
    E --> J

    F --> N
    N --> K
    N --> L
    N --> M

    K --> G
    L --> G
    M --> G

    G --> H
    G --> I

    H --> P
    I --> P

    F --> O
    G --> P
    L --> Q
    J --> R
    J --> Q

Component Details¶

1. API Gateway¶

Purpose: Single entry point for all client requests.

Endpoints:

POST   /api/v1/documents          # Upload
GET    /api/v1/documents/:id      # Get metadata
GET    /api/v1/documents/:id/file # Download file
DELETE /api/v1/documents/:id      # Delete
POST   /api/v1/search             # Search
GET    /api/v1/tags               # List tags

2. Ingestion Service¶

Purpose: Accept and validate incoming documents.

sequenceDiagram
    participant Client
    participant API
    participant Ingestion
    participant Queue
    participant Storage

    Client->>API: POST /documents (file)
    API->>Ingestion: Validate & store
    Ingestion->>Storage: Save raw file
    Storage-->>Ingestion: File URL
    Ingestion->>Queue: Enqueue processing job
    Ingestion-->>API: Document ID + status
    API-->>Client: 202 Accepted

Responsibilities:

• File format validation
• Size limit enforcement
• Virus scanning (optional)
• Initial metadata extraction
• Queueing for async processing

3. Processing Orchestrator¶

Purpose: Coordinate document processing pipeline.

State Machine:

stateDiagram-v2
    [*] --> Queued
    Queued --> OCR: Start processing
    OCR --> Embedding: Text extracted
    Embedding --> Dedup: Embedding computed
    Dedup --> Classification: Not duplicate
    Dedup --> Duplicate: Is duplicate
    Classification --> Tagging: Classified
    Tagging --> Indexing: Tagged
    Indexing --> Ready: Indexed
    Ready --> [*]

    OCR --> Failed: Error
    Embedding --> Failed: Error
    Failed --> [*]

4. OCR Worker¶

Purpose: Extract text from scanned documents and images.

Configuration:

ocr:
  engine: tesseract
  languages: [eng, hin]
  timeout_seconds: 300
  max_pages: 100
  dpi: 300

5. Embedding Worker¶

Purpose: Generate vector embeddings for semantic operations.

Process:

1. Chunk text into 512-token segments
2. Generate embedding per chunk
3. Mean-pool for document-level embedding
4. Store in vector database

6. Classification Service¶

Purpose: Categorize documents automatically.

flowchart LR
    A[Text] --> B[Tokenize]
    B --> C[Feature Extraction]
    C --> D[Type Classifier]
    C --> E[Category Classifier]
    D --> F[Type + Confidence]
    E --> G[Category + Confidence]

Document Types:

7. Search Service¶

Purpose: Enable document discovery.

Index Schema:

{
  "id": "doc_123",
  "title": "Invoice #456",
  "content": "Full extracted text...",
  "type": "invoice",
  "category": "finance",
  "tags": ["payment", "vendor", "Q1-2024"],
  "created_at": "2024-01-15T10:30:00Z",
  "embedding": [0.123, -0.456, ...]
}

Data Stores¶

Document Store (S3/MinIO)¶

Metadata DB (PostgreSQL)¶

Schema (simplified):

CREATE TABLE documents (
    id UUID PRIMARY KEY,
    filename VARCHAR(255),
    mime_type VARCHAR(100),
    size_bytes BIGINT,
    hash_md5 CHAR(32),
    hash_sha256 CHAR(64),
    status VARCHAR(50),
    type VARCHAR(100),
    category VARCHAR(100),
    confidence DECIMAL(5,2),
    created_at TIMESTAMP,
    processed_at TIMESTAMP
);

CREATE TABLE tags (
    id UUID PRIMARY KEY,
    name VARCHAR(100) UNIQUE,
    created_at TIMESTAMP
);

CREATE TABLE document_tags (
    document_id UUID REFERENCES documents(id),
    tag_id UUID REFERENCES tags(id),
    source VARCHAR(50), -- 'auto' or 'manual'
    PRIMARY KEY (document_id, tag_id)
);

Vector Store (Qdrant)¶

Scalability¶

Horizontal Scaling¶

Performance Targets¶

Security¶

Deployment Options¶

Option 1: Docker Compose (Development)¶

services:
  api:
    image: pebble-dms/api
  worker:
    image: pebble-dms/worker
  postgres:
    image: postgres:15
  qdrant:
    image: qdrant/qdrant
  meilisearch:
    image: getmeili/meilisearch
  minio:
    image: minio/minio
  redis:
    image: redis:7

Option 2: Kubernetes (Production)¶

• Helm chart for deployment
• HPA for worker scaling
• PVC for persistent storage
• Ingress for external access

← Back to Home