Datasets and RAG

Datasets are tenant-scoped knowledge stores. Deeplinq persists source objects, processes them through durable jobs, stores organization-isolated vectors, and authorizes every dataset before retrieval.

Before starting, configure:

• S3-compatible object storage;
• Weaviate;
• a kind: "embedding" connector compatible with the platform's 1,024 dimension retrieval contract;
• Docling for PDF and office-document extraction;
• an optional reranker connector for cross-encoder relevance scoring.

Create a dataset

curl -H "Authorization: Bearer $DEEPLINQ_TOKEN" \
  -H "Content-Type: application/json" \
  -X POST "$BASE_URL/v1/datasets" \
  -d '{"name":"HR policies","shared":false}'

Private datasets belong to their creator. Shared datasets can be governed at organization scope. Lists use opaque keyset cursors:

GET /v1/datasets?limit=50&cursor=<opaque>

Upload a file

curl -H "Authorization: Bearer $DEEPLINQ_TOKEN" \
  -X POST "$BASE_URL/v1/datasets/$DATASET_ID/files" \
  -F "file=@employee-handbook.pdf"

The source is stored before processing. Keep the document_id from the upload response. Uploads are not deduplicated by filename: posting a file under a name the dataset already holds creates a second document rather than superseding the first. To change a file's contents, replace it by the document_id you kept. A naive incremental sync that re-pushes every changed file through POST …/files will duplicate the corpus on every pass.

Poll:

GET /v1/datasets/{dataset_id}/files

Status moves through:

pending → extracting → embedding → ready
                                  ↘ failed

Each row also carries origin: upload or crawl, and absent for a legacy document with no resource. Only an upload can be replaced. Beside the document's status, which a replacement leaves alone until it activates, a row carries:

• replacement_status: pending, extracting, embedding or failed; null when none is in flight and the last one did not fail.
• replacement_error: why the last replacement failed, else null.
• last_replacement_changes: {replaced_at, type_changes: [{table, column, from, to}]} from the last replacement that activated, null before one.
• readiness_pending: true while the file is ready but its current content has not yet been handed to document extraction. After that, the extraction's rereading and coverage say whether its table has caught up.

Committed ingestion survives restarts and retries with backoff. Inspect job history and progress:

GET /v1/datasets/{dataset_id}/ingest/runs
GET /v1/datasets/{dataset_id}/ingest/runs/{run_id}

A replacement opens its own run with trigger: "replace". A run's superseded_docs counts the documents whose job found a newer ingest operation, such as a replace or a reprocess, owning the document, and so wrote nothing.

Replace a file

curl -H "Authorization: Bearer $DEEPLINQ_TOKEN" \
  -X POST "$BASE_URL/v1/datasets/$DATASET_ID/files/$DOCUMENT_ID/replace" \
  -F "file=@employee-handbook.pdf"

A replacement takes the same multipart file part as an upload, checked the same way (below), and requires write on the dataset. It keeps the document's document_id and filename and answers 202:

{"status": "pending", "document_id": "…"}

The document keeps serving its current file until the replacement activates, in one transaction that leaves it ready. A document whose own ingest failed can be replaced too. Follow it through replacement_status in the file listing.

The upload's checks refuse a replacement exactly as they refuse an upload: 413 file_too_large over the 50 MiB per-file limit, 415 unsupported_type for an extension ingestion does not support, 400 invalid_request for a missing, empty or unreadable file part, and 403 dataset_forbidden without write.

For CSV and XLSX files, every table and column the current file has must also be in the replacement; added sheets and columns are fine, and column order is free. Names are compared case-sensitively. A missing one fails the replacement while the current file keeps serving, and replacement_error names it — the new file has no table "sales_q3"; the current one does, or table "sales_2026" has no column "region" (the new file has "Region"). Column types are not promised, because DuckDB infers them from each file: a changed type is listed in last_replacement_changes, never refused.

Ingest a website

curl -H "Authorization: Bearer $DEEPLINQ_TOKEN" \
  -H "Content-Type: application/json" \
  -X POST "$BASE_URL/v1/datasets/$DATASET_ID/resources" -d '{
  "type":"website",
  "name":"Product documentation",
  "url":"https://docs.example.com",
  "max_pages":50,
  "max_depth":2,
  "rate_limit_ms":250
}'

The crawler is same-host, depth-bounded, page-bounded, robots-aware, and protected against private, loopback, link-local, and metadata destinations. Redirect destinations are revalidated.

Schedule content refresh:

curl -H "Authorization: Bearer $DEEPLINQ_TOKEN" \
  -H "Content-Type: application/json" \
  -X PUT "$BASE_URL/v1/datasets/$DATASET_ID/ingest/schedule" \
  -d '{"interval":"1d"}'

Allowed intervals are 1h, 6h, 12h, 1d, and 7d. Content hashes avoid re-embedding unchanged pages.

One-shot retrieval

Add dataset_ids to an OpenAI-compatible chat request:

curl -H "Authorization: Bearer $DEEPLINQ_TOKEN" \
  -H "Content-Type: application/json" \
  -X POST "$BASE_URL/v1/chat/completions" -d "{
  \"model\":\"gpt-5-mini\",
  \"messages\":[
    {\"role\":\"user\",\"content\":\"What is our parental leave policy?\"}
  ],
  \"dataset_ids\":[\"$DATASET_ID\"]
}"

Deeplinq combines BM25 and vector retrieval, optionally reranks candidates, screens retrieved chunks, and injects authorized context. Retrieval failure is 503; it never silently becomes plain chat.

Governed knowledge agent

When KNOWLEDGE_AGENT_ENABLED=1, a non-streaming request can ask the knowledge agent to search and read evidence iteratively:

{
  "model": "auto",
  "messages": [
    {"role": "user", "content": "Compare our parental and caregiver leave."}
  ],
  "agent_mode": "knowledge",
  "dataset_ids": ["<dataset-id>"],
  "stream": false
}

The normal completion response gains a deeplinq extension containing:

• agent.status: completed, insufficient_evidence, or budget_exhausted;
• citations: dataset, document, filename, and excerpt provenance.

Those three statuses are normal 200 outcomes. Runtime unavailability or a required dependency failure is 503.

The run budget, and who sets it

Every run obeys ten per-run ceilings — datasets, model calls, tool calls, searches, results per search, evidence characters, evidence-read ids, output tokens, wall-clock seconds, and spend in micro-USD. They are a BUDGET, not a content control: they decide how much work one question may cost, never which sources are searched or who may ask. Dataset ACLs and the request's own dataset_ids decide that.

Three layers can set it, and exactly one of them is the parent of any given organization:

An application budget replaces rather than tightens: it may raise a ceiling as freely as lower one, and while its row exists the engine-wide row is not read at all. Deleting it returns the application to the engine-wide budget on the next run. The single ceiling nobody may raise is max_datasets, capped at the 10 selected datasets a knowledge request may name in the first place.

An organization override is field-level: an omitted or null field inherits the parent, and a supplied field must be positive and no greater than the parent's. An organization read reports parent_source and parent_version, so a tenant administrator can see which layer they are tightening.

Every read and successful write returns a strong ETag, and every write requires If-Match — the tag you were last given, or *. The tags are opaque: echo them, never parse them. A missing condition is 428; a stale one is 412. Because an organization's tag covers its selected parent, an application changing its budget invalidates tags its tenants took before the change, and a fresh GET is the recovery.

Share a dataset

Grant read, write, or admin permission to a user, role, or team:

curl -H "Authorization: Bearer $DEEPLINQ_TOKEN" \
  -H "Content-Type: application/json" \
  -X POST "$BASE_URL/v1/datasets/$DATASET_ID/grants" -d '{
  "principal_type":"team",
  "principal_id":"<team-id>",
  "permission":"read"
}'

Team membership is resolved live, so removing a member affects the next access check.

Tabular files

With the DuckDB sidecar configured, CSV and XLSX files are converted to Parquet and expose a schema summary for retrieval. Analytical questions can generate read-only DuckDB SQL. SQL, rows, execution time, memory, and result sizes are bounded; execution is interrupted after 30 seconds.

A durable run reads those tables itself, through the dataset.query engine tool: the task names the dataset with dataset_id at create, or the call names one, and the engine answers a plain-words question — it writes the SQL — an explicit SELECT over one named table, or a typed calculation for a row count, numeric sum or grouped rank. Calculation definitions record their population and execution facts with the captured rows; they do not verify the model's interpretation or prose. See Query a dataset's tables.

Delete safely

DELETE /v1/datasets/{id} returns 202. A durable cleanup job removes vectors, blobs, documents, and metadata. Continue polling your inventory until the dataset disappears; do not assume the response means every backing store has already been erased.

How much has this person uploaded — GET /v1/me/storage

The calling user's own uploaded-file footprint in the organization their token resolved to. No role, no parameters.

curl -H "Authorization: Bearer <that user's access token>" \
     "$BASE_URL/v1/me/storage"

{
  "file_count": 2,
  "used_bytes": 20000000
}

• used_bytes is SOURCE bytes, as uploaded. Chunks, vectors, extracted text and Parquet artifacts occupy more space than this. Label it uploaded-file storage in your product; do not compare it with a disk or provider bill.
• A file belongs to whoever uploaded it. The identity is recorded at upload time and never reassigned — not by sharing a dataset, not by transferring ownership, not by whoever eventually deletes it. Reading a colleague's file in a shared dataset adds nothing to the reader, and a file you uploaded into a container you can no longer open still counts for you. The response carries only the two totals, so it reveals nothing about that container; removing such a file may need an organization administrator. A replaced file stays counted for its original uploader, whoever replaced it, at the new file's size once the replacement activates.
• Deletion is asynchronous, so the number lags it. A file stays counted while its resource is deleting and falls out once cleanup removes the row — which is honest about bytes that are still there.
• Every ingestion status counts, failed included: a document that could not be extracted still holds the bytes that were uploaded.
• Crawled pages are excluded. A website resource produces many documents and no upload, so a crawl never lands on a person's total.
• This is an observation, not a quota. Uploads behave identically at any total. If your product sells a storage allowance, hold the allowance on your side and compare it with this number.