Documents (upload, search, retrieval)

A per-tenant document store for PDFs, images, and text — upload once, then search and retrieve from the API, MCP, and CLI. Each upload is SHA-256 deduped (an identical file is returned, not re-stored), converted to a markdown rendition (PDF text via PdfPig; text/HTML converted; images are not OCR’d in v1 — they get a metadata stub but are still cataloged and searchable by metadata), chunked, and — when an embedding provider is configured — embedded for semantic search. Embedding is best-effort and never fails the upload; the catalog row shows embedStatus (pending/done/failed/skipped).

Set standard CMS metadata at upload (title, description, author, subject, keywords, language, category, version, source system) and edit it later. Optionally link a document to a data-lake entity row (schema/table/row id).

Caller-supplied markdown. Any upload can include its own markdown rendition — it is stored verbatim and server conversion is skipped. This is how you make an image searchable without OCR: describe it in markdown (an AI assistant can do this by looking at the image) and that description is what search matches. Re-uploading the same file later with markdown upgrades an earlier stub in place; caller markdown is never silently overwritten. Provenance is tracked as markdownSource (caller/converted/stub).

Search is hybrid: metadata + extracted-text matching, plus (on SQL 2025+ tenants with an embedding provider set) a semantic vector search over the embedded chunks — merged and scored, with the best snippet and which method matched.

Configure embeddings (Settings → Embeddings). The Settings → Embeddings (semantic search) card (owner-or-admin) is where you set the provider that powers semantic search — no raw SQL needed. Paste your OpenAI API key (or point the Endpoint at a local Ollama, which needs no key), optionally override the Model and Dimensions (256–4096), and Save; the key is stored encrypted and never shown again (the card shows only a masked …abcd hint, and leaving the key field blank on a later edit keeps the stored one). Hit Test to prove connectivity and see the returned dimensions and latency inline. Changing the model or dimensions invalidates every existing document embedding (stored vectors must all share one dimensionality) — the Test button warns on a mismatch, and you should then click Embed pending chunks, which re-embeds documents left un-embedded (uploaded before a provider was set, or after a model/dimensions change) in batches, looping until none remain.

Reads are gated by data.read; uploads, metadata edits, and deletes by data.ingest; upload/delete/metadata-edit are audited. Over MCP: upload_document, search_documents, get_document, list_documents. Over the CLI: dlake docs upload|list|search|get|rm (--markdown <file> supplies a rendition). Upload cap is 50 MB by default (an administrator-configured limit); the MCP upload_document tool caps tighter at 10 MB.