AI & ML Overview¶

This document describes the AI and machine learning components used in Pebble DMS for document processing, classification, and tagging.

AI Capabilities Summary¶

1. OCR (Optical Character Recognition)¶

Purpose¶

Extract searchable text from scanned PDFs and images.

Engine Options¶

Architecture¶

flowchart LR
    A[Image/PDF] --> B[Preprocessing]
    B --> C[Deskew]
    C --> D[Denoise]
    D --> E[Binarize]
    E --> F[OCR Engine]
    F --> G[Post-processing]
    G --> H[Text + Confidence]

Preprocessing Pipeline¶

Quality Metrics¶

2. Document Classification¶

Purpose¶

Automatically categorize documents by type and content.

Classification Hierarchy¶

graph TD
    A[Document] --> B{Type}
    B --> C[Invoice]
    B --> D[Contract]
    B --> E[Report]
    B --> F[Letter]
    B --> G[Form]

    C --> H{Category}
    H --> I[Finance]
    H --> J[Procurement]

    D --> K{Category}
    K --> L[Legal]
    K --> M[HR]
    K --> N[Vendor]

Model Options¶

Training Pipeline¶

flowchart LR
    A[Labeled Docs] --> B[Text Extraction]
    B --> C[Preprocessing]
    C --> D[Train/Val Split]
    D --> E[Model Training]
    E --> F[Evaluation]
    F --> G{Accuracy OK?}
    G -->|Yes| H[Deploy]
    G -->|No| I[Tune/More Data]
    I --> E

Confidence Handling¶

3. Embedding Models¶

Purpose¶

Generate vector representations for semantic search and near-duplicate detection.

Model Comparison¶

Embedding Strategy¶

flowchart TB
    A[Document Text] --> B{Length?}
    B -->|<512 tokens| C[Single Embedding]
    B -->|>512 tokens| D[Chunk into 512]
    D --> E[Embed Each Chunk]
    E --> F[Mean Pooling]
    F --> G[Document Embedding]
    C --> G
    G --> H[Store in Qdrant]

Use Cases¶

4. Named Entity Recognition (NER)¶

Purpose¶

Extract structured entities from unstructured text.

Entity Types¶

Model Options¶

Pipeline¶

flowchart LR
    A[Text] --> B[Tokenize]
    B --> C[NER Model]
    C --> D[Entity Spans]
    D --> E[Dedup & Clean]
    E --> F[Entity List]

5. Keyword Extraction¶

Purpose¶

Identify important terms and topics from document content.

Methods¶

KeyBERT Example¶

from keybert import KeyBERT

kw_model = KeyBERT()
keywords = kw_model.extract_keywords(
    doc, 
    keyphrase_ngram_range=(1, 2),
    stop_words='english',
    top_n=10
)
# Returns: [('machine learning', 0.72), ('data processing', 0.68), ...]

6. Topic Modeling (Future)¶

Purpose¶

Discover latent topics across document collection.

Methods¶

Model Training & Evaluation¶

Training Data Requirements¶

Evaluation Metrics¶

Continuous Improvement¶

flowchart LR
    A[Production] --> B[Log Predictions]
    B --> C[Human Review]
    C --> D[Corrections]
    D --> E[Training Data]
    E --> F[Retrain Model]
    F --> G[Evaluate]
    G --> H{Better?}
    H -->|Yes| I[Deploy]
    H -->|No| J[Iterate]
    I --> A

Recommended Stack¶

MVP Configuration¶

ocr:
  engine: tesseract
  version: 5.x
  languages: [eng]

classification:
  model: distilbert-base-uncased
  framework: transformers

embeddings:
  model: all-MiniLM-L6-v2
  framework: sentence-transformers

ner:
  model: en_core_web_lg
  framework: spacy

keywords:
  method: keybert

Hardware Requirements¶

← Back to Home