Datasets and RAG

Datasets are tenant-scoped knowledge stores. Deeplinq persists source objects, processes them through durable jobs, stores organization-isolated vectors, and authorizes every dataset before retrieval.

Before starting, configure:

• S3-compatible object storage;
• Weaviate;
• a kind: "embedding" connector compatible with the platform's 1,024 dimension retrieval contract;
• Docling for PDF and office-document extraction;
• an optional reranker connector for cross-encoder relevance scoring.

Create a dataset

curl -H "Authorization: Bearer $DEEPLINQ_TOKEN" \
  -H "Content-Type: application/json" \
  -X POST "$BASE_URL/v1/datasets" \
  -d '{"name":"HR policies","shared":false}'

Private datasets belong to their creator. Shared datasets can be governed at organization scope. Lists use opaque keyset cursors:

GET /v1/datasets?limit=50&cursor=<opaque>

Upload a file

curl -H "Authorization: Bearer $DEEPLINQ_TOKEN" \
  -X POST "$BASE_URL/v1/datasets/$DATASET_ID/files" \
  -F "file=@employee-handbook.pdf"

The source is stored before processing. Keep the document_id from the upload response. There is no PUT replace and no deduplication by filename: re-uploading a file under the same name creates a second document rather than superseding the first. To replace a file, sequence DELETE /v1/datasets/{dataset_id}/files/{fileId} — fileId is the document_id you kept — then POST the new upload. A naive incremental sync that just re-pushes every changed file without deleting the old document first will duplicate the corpus on every pass.

Poll:

GET /v1/datasets/{dataset_id}/files

Status moves through:

pending → extracting → embedding → ready
                                  ↘ failed

Committed ingestion survives restarts and retries with backoff. Inspect job history and progress:

GET /v1/datasets/{dataset_id}/ingest/runs
GET /v1/datasets/{dataset_id}/ingest/runs/{run_id}

Ingest a website

curl -H "Authorization: Bearer $DEEPLINQ_TOKEN" \
  -H "Content-Type: application/json" \
  -X POST "$BASE_URL/v1/datasets/$DATASET_ID/resources" -d '{
  "type":"website",
  "name":"Product documentation",
  "url":"https://docs.example.com",
  "max_pages":50,
  "max_depth":2,
  "rate_limit_ms":250
}'

The crawler is same-host, depth-bounded, page-bounded, robots-aware, and protected against private, loopback, link-local, and metadata destinations. Redirect destinations are revalidated.

Schedule content refresh:

curl -H "Authorization: Bearer $DEEPLINQ_TOKEN" \
  -H "Content-Type: application/json" \
  -X PUT "$BASE_URL/v1/datasets/$DATASET_ID/ingest/schedule" \
  -d '{"interval":"1d"}'

Allowed intervals are 1h, 6h, 12h, 1d, and 7d. Content hashes avoid re-embedding unchanged pages.

One-shot retrieval

Add dataset_ids to an OpenAI-compatible chat request:

curl -H "Authorization: Bearer $DEEPLINQ_TOKEN" \
  -H "Content-Type: application/json" \
  -X POST "$BASE_URL/v1/chat/completions" -d "{
  \"model\":\"gpt-5-mini\",
  \"messages\":[
    {\"role\":\"user\",\"content\":\"What is our parental leave policy?\"}
  ],
  \"dataset_ids\":[\"$DATASET_ID\"]
}"

Deeplinq combines BM25 and vector retrieval, optionally reranks candidates, screens retrieved chunks, and injects authorized context. Retrieval failure is 503; it never silently becomes plain chat.

Governed knowledge agent

When KNOWLEDGE_AGENT_ENABLED=1, a non-streaming request can ask the knowledge agent to search and read evidence iteratively:

{
  "model": "auto",
  "messages": [
    {"role": "user", "content": "Compare our parental and caregiver leave."}
  ],
  "agent_mode": "knowledge",
  "dataset_ids": ["<dataset-id>"],
  "stream": false
}

The normal completion response gains a deeplinq extension containing:

• agent.status: completed, insufficient_evidence, or budget_exhausted;
• citations: dataset, document, filename, and excerpt provenance.

Those three statuses are normal 200 outcomes. Runtime unavailability or a required dependency failure is 503.

The run budget, and who sets it

Every run obeys ten per-run ceilings — datasets, model calls, tool calls, searches, results per search, evidence characters, evidence-read ids, output tokens, wall-clock seconds, and spend in micro-USD. They are a BUDGET, not a content control: they decide how much work one question may cost, never which sources are searched or who may ask. Dataset ACLs and the request's own dataset_ids decide that.

Three layers can set it, and exactly one of them is the parent of any given organization:

An application budget replaces rather than tightens: it may raise a ceiling as freely as lower one, and while its row exists the engine-wide row is not read at all. Deleting it returns the application to the engine-wide budget on the next run. The single ceiling nobody may raise is max_datasets, capped at the 10 selected datasets a knowledge request may name in the first place.

An organization override is field-level: an omitted or null field inherits the parent, and a supplied field must be positive and no greater than the parent's. An organization read reports parent_source and parent_version, so a tenant administrator can see which layer they are tightening.

Every read and successful write returns a strong ETag, and every write requires If-Match — the tag you were last given, or *. The tags are opaque: echo them, never parse them. A missing condition is 428; a stale one is 412. Because an organization's tag covers its selected parent, an application changing its budget invalidates tags its tenants took before the change, and a fresh GET is the recovery.

Share a dataset

Grant read, write, or admin permission to a user, role, or team:

curl -H "Authorization: Bearer $DEEPLINQ_TOKEN" \
  -H "Content-Type: application/json" \
  -X POST "$BASE_URL/v1/datasets/$DATASET_ID/grants" -d '{
  "principal_type":"team",
  "principal_id":"<team-id>",
  "permission":"read"
}'

Team membership is resolved live, so removing a member affects the next access check.

Tabular files

With the DuckDB sidecar configured, CSV and XLSX files are converted to Parquet and expose a schema summary for retrieval. Analytical questions can generate read-only DuckDB SQL. SQL, rows, execution time, memory, and result sizes are bounded; execution is interrupted after 30 seconds.

A durable run reads those tables itself, through the dataset.query engine tool: the task names the dataset with dataset_id at create, or the call names one, and the engine answers a plain-words question — it writes the SQL — or an explicit SELECT over one named table. See Query a dataset's tables.

Delete safely

DELETE /v1/datasets/{id} returns 202. A durable cleanup job removes vectors, blobs, documents, and metadata. Continue polling your inventory until the dataset disappears; do not assume the response means every backing store has already been erased.

How much has this person uploaded — GET /v1/me/storage

The calling user's own uploaded-file footprint in the organization their token resolved to. No role, no parameters.

curl -H "Authorization: Bearer <that user's access token>" \
     "$BASE_URL/v1/me/storage"

{
  "file_count": 2,
  "used_bytes": 20000000
}

• used_bytes is SOURCE bytes, as uploaded. Chunks, vectors, extracted text and Parquet artifacts occupy more space than this. Label it uploaded-file storage in your product; do not compare it with a disk or provider bill.
• A file belongs to whoever uploaded it. The identity is recorded at upload time and never reassigned — not by sharing a dataset, not by transferring ownership, not by whoever eventually deletes it. Reading a colleague's file in a shared dataset adds nothing to the reader, and a file you uploaded into a container you can no longer open still counts for you. The response carries only the two totals, so it reveals nothing about that container; removing such a file may need an organization administrator.
• Deletion is asynchronous, so the number lags it. A file stays counted while its resource is deleting and falls out once cleanup removes the row — which is honest about bytes that are still there.
• Every ingestion status counts, failed included: a document that could not be extracted still holds the bytes that were uploaded.
• Crawled pages are excluded. A website resource produces many documents and no upload, so a crawl never lands on a person's total.
• This is an observation, not a quota. Uploads behave identically at any total. If your product sells a storage allowance, hold the allowance on your side and compare it with this number.