Processing

Every upload runs through a pipeline

FileNest scans, validates, and enriches every file after upload. Processing is asynchronous and non-blocking — files are available for download before all stages finish.

Virus Scan

Always first

ClamAV-powered antivirus scan runs synchronously before any other stage. Detected files are quarantined immediately — the file record is created but download is blocked. A NATS event is fired so your webhook can notify admins.

MIME Validation

Always second

Content-based MIME detection (libmagic) runs in parallel with virus scanning. Blocks files whose actual type doesn't match the declared Content-Type — stops extension spoofing at the upload boundary.

OCR

Configurable

Tesseract-based OCR extracts text from images and scanned PDFs. Extracted text is stored as file metadata and sent to the OpenSearch indexer so uploaded documents become full-text searchable without client-side preprocessing.

PHI / PII Detection

Configurable

Pattern-based and ML-based scanning flags protected health information and PII before a file's download URL is issued. On detection the file can be quarantined, redacted, or forwarded to a review queue — all via project config.

Thumbnails & Previews

Configurable

Generates responsive image thumbnails (WebP, configurable sizes) and PDF page previews. Results are stored in the same object storage bucket and referenced in the file metadata. The `<FilePreview>` React component picks them up automatically.

AI Embedding

Phase 7

Sentence-transformer or OpenAI embeddings are generated per document chunk and stored in pgvector. Powers semantic search: 'find documents similar to this one' or 'find all documents mentioning adverse reactions' without exact-keyword matches.

Configuration — per project, no code changes

# Enable stages per project via config
project_config = {
  "processing": {
    "stages": [
      "virus_scan",       # always runs
      "mime_validation",  # always runs
      "ocr",
      "phi_detection",
      "thumbnail"
    ],
    "thumbnail_sizes": [128, 512],
    "ocr_languages": ["eng"],
    "phi_action": "quarantine"  # or "redact" | "flag"
  }
}