Documents (upload, search, retrieval)

A per-tenant document store for PDFs, images, and text — upload once, then search and retrieve from the API, MCP, and CLI. Each upload is SHA-256 deduped (an identical file is returned, not re-stored; a deleted identical file is restored — see Delete and restore below), converted to a markdown rendition (PDF text via PdfPig; text/HTML converted; images are not OCR’d in v1 — they get a metadata stub but are still cataloged and searchable by metadata), chunked, and — when an embedding provider is configured — embedded for semantic search. Embedding is best-effort and never fails the upload; the catalog row shows embedStatus (pending/done/failed/skipped).

Set standard CMS metadata at upload (title, description, author, subject, keywords, language, category, version, source system) and edit it later. Optionally link a document to a data-lake entity row (schema/table/row id).

Caller-supplied markdown. Any upload can include its own markdown rendition — it is stored verbatim and server conversion is skipped. This is how you make an image searchable without OCR: describe it in markdown (an AI assistant can do this by looking at the image) and that description is what search matches. Re-uploading the same file later with markdown upgrades an earlier stub in place; caller markdown is never silently overwritten. Provenance is tracked as markdownSource (caller/converted/stub).

Search is hybrid: metadata + extracted-text matching, plus (on SQL 2025+ tenants with an embedding provider set) a semantic vector search over the embedded chunks — merged and scored, with the best snippet and which method matched.

Configure embeddings (Settings → Embeddings). The Settings → Embeddings (semantic search) card (owner-or-admin) is where you set the provider that powers semantic search — no raw SQL needed. Paste your OpenAI API key (or point the Endpoint at a local Ollama, which needs no key), optionally override the Model and Dimensions (256–4096), and Save; the key is stored encrypted and never shown again (the card shows only a masked …abcd hint, and leaving the key field blank on a later edit keeps the stored one). Hit Test to prove connectivity and see the returned dimensions and latency inline. Changing the model or dimensions invalidates every existing document embedding (stored vectors must all share one dimensionality) — the Test button warns on a mismatch, and you should then click Embed pending chunks, which re-embeds documents left un-embedded (uploaded before a provider was set, or after a model/dimensions change) in batches, looping until none remain.

Delete and restore. Deleting a document is a soft delete: it disappears from the list, from search (text and semantic), from tag counts, and from get_document (which answers not found), but nothing is erased. Delete is reversible; content and embeddings are retained — the original file, its markdown, its chunks and their vectors stay in the tenant database, and restore brings the document back to listing and search exactly as it was. There is no purge (hard delete) on any surface. Note that if an embedding provider is configured, the document’s text was sent to that provider when it was embedded; deleting the document here does not recall anything the provider already received. Deleting an already-deleted document succeeds and changes nothing; so does restoring a live one. To see deleted documents, list with include deleted (each row shows whether it is deleted). Re-uploading a deleted file restores it: if the only stored copy of identical content is a deleted one, the upload revives that document instead of storing a second copy, and reports restored: true. A duplicate upload keeps the stored metadata — whether the identical document was live or restored, the title, tags and other metadata sent with the upload are not applied, and the result says so with metadataKept: "stored" and ignoredFields (the fields you sent that were not applied); edit the document afterwards to change them. Over the REST API, restore is a metadata PATCH carrying only isDeleted: false (there is no separate restore address); it cannot be combined with metadata changes in the same call — restore first, then edit — and a PATCH cannot delete. A restore is refused when another live document already holds the same content. Delete and restore are both audited (DOCUMENT_DELETE / DOCUMENT_RESTORE, naming who did it), including a restore caused by a re-upload. Anyone with SQL read access to the platform tables (an owner or Admin running raw SQL) can still see a deleted document’s rows — deleted documents are hidden from the document tools, not from the database.

Reads are gated by data.read; uploads, metadata edits, deletes and restores by data.ingest; upload/delete/restore/metadata-edit are audited. Over MCP: upload_document, search_documents, get_document, list_documents, update_document, delete_document, restore_document. Over the CLI: dlake docs upload|list|search|get|update|rm|restore (--markdown <file> supplies a rendition; --include-deleted on list/get shows deleted documents). Upload cap is 50 MB by default (an administrator-configured limit); the MCP upload_document tool caps tighter at 10 MB, and so does the CLI’s upload, which goes through that tool.