Query: filter, sort, group and total Blocks

A query (DataQuery, @skaile/forge-editor/core) asks the Block index for the Blocks of one or several Block types: which Documents, which Properties, in which order, grouped how, totalled how. A query is data. It is a JSON object that a view stores in a Typed block's meta, that the client sends over HTTP, and that the document chat builds in a tool call. One evaluator, evaluateQuery, answers all of them.

The query

{
  "v": 1,
  "source": { "kind": "blocks", "blockType": "task", "scope": { "documentIds": ["plan"] } },
  "filter": { "and": [
    { "field": "status", "op": "eq", "value": "done" },
    { "field": "points", "op": "gte", "value": 3 }
  ] },
  "search": { "q": "angebot" },
  "sorts": [{ "field": "due", "dir": "asc", "nulls": "last" }],
  "group": { "field": "$basket", "hideEmpty": true },
  "aggregates": [{ "fn": "sum", "field": "points", "as": "total" }],
  "select": ["status", "points", "due"],
  "page": { "limit": 50 }
}

• source.blockType is one Block type id or a list. A Typed block's type is its blockType; any other Block has its node type (paragraph, heading, …). source.scope narrows further: documentIds, tags (a row needs one of them), basket. subtreeOf is not applied: the index does not know the nesting, and the answer says so with a warning.
• filter is a condition { field, op, value? } or { and: [...] } / { or: [...] }, nested as deep as needed. An empty and holds, an empty or does not.
• search.q uses the words rule of the workspace search: every word must match, as a whole word or as the start of one. Without a sorts entry that decides, the best match comes first.
• sorts: the first entry that tells two rows apart wins. A row without a value sorts last; nulls: 'first' turns that around.
• group partitions the page into groups. The groups follow the order of the property's options (a status group todo, doing, done as declared), else the order of first appearance; the group without a value (key: null) is last unless hideEmpty.
• aggregates (count, sum, avg, min, max) are computed over all matching rows, not just the page, and answered under as.
• select lists the property keys and pseudo-fields to put in DataRow.values. Absent: every property of the Block.
• page.limit is at most MAX_QUERY_ROWS (1000). nextCursor of an answer goes into page.cursor of the next request. The cursor is the base64 of the row offset, so a page is not stable if Blocks change between two calls.

Fields

A field is a property key of the Block types in the source, or a pseudo-field:

Property keys must not start with $; propertyDefinitionSchema refuses them.

Operators per property kind

Which operators, sort key, group key and aggregates apply is the property kind's entry in PROPERTY_KINDS (core/property-kinds.ts); the query does not branch on the type itself.

Dates: on compares the calendar day in env.timeZone (a plain YYYY-MM-DD is that day whatever the time zone), before and after compare the instant (a plain day counts as its start in UTC). Text sorts by the collation of env.locale, numerically where it holds numbers (item 2 before item 10).

env (QueryEnv) carries locale, timeZone, now and me. The evaluator is pure: the same rows, query and env give the same answer.

Which Property definitions apply

evaluateQuery(rows, defs, …) types a field by defs. On the server, defs are the properties capability of the host's Block types (EditorServerOptions.blockTypes) that the query's source names, merged, the first definition of a key winning (queryDefinitionsOf in @skaile/forge-editor/server). A property that no Block type declares is not an error: the evaluator infers a kind from the values of the rows (a boolean is a checkbox, a number a number, a date-like text a date, other text text, a list of addresses a reference-list, a list of text a multiselect; the first value of a key decides). A key that no row has is an unknown-field. So a host with no blockTypes configured still answers queries over task Blocks, just without option order or status groups.

An error issue means no rows: the answer has rows: [], total: 0 and the issues. A warning (a select entry nobody has, subtreeOf) is answered along with the rows.

The port, the route and the client

BlockIndex.query(query, scope, env, definitions?) answers a query over the Blocks of the Documents in scope. source.scope.documentIds narrows the scope and never widens it (narrowScope). The memory adapter runs evaluateQuery over its rows. The drizzle adapter loads the rows of the source's Block types per Document with one SQL statement and evaluates in JS; a database-specific adapter may push parts of the filter into SQL as long as it gives the answer evaluateQuery gives (the BlockIndex contract in test/support/block-index-contract.ts checks this). A custom BlockIndex must implement query; the quickest way is evaluateQuery over byDocument.

POST /api/editor/query takes { query, env? } (QueryRequestBody; env has locale and timeZone, defaulting to the first Accept-Language tag Intl knows and UTC). It answers a DataResult for the Documents the user may read (readableDocumentsOf). 401 without a user, 400 for a body that is no query, 422 when the query has an error issue (the error's data lists the issues), 501 without the index port.

const result = await backend.index.query({
  v: 1,
  source: { kind: 'blocks', blockType: 'task' },
  filter: { field: 'status', op: 'eq', value: 'done' },
  group: { field: '$basket' },
})
for (const group of result.groups ?? []) console.log(group.key, group.rows.length)

EditorBackend.index.query?(query, { env?, signal? }) is optional: a backend without it leaves data views out. It rejects with BackendError (422, data = the issues) for an invalid query. The HTTP backend sends the browser's time zone unless env says otherwise.

The document chat

find_blocks (createWorkspaceTools()) gains two inputs:

• blockType lists the Blocks of that type (all of them without a filter);
• filter is a FilterNode over their Properties, and needs blockType.

With query as well, only the hits of the search come back. Each hit then carries its values. A filter that does not fit the type comes back to the model as a sentence (Ungültige Abfrage: … naming the field and the operators the type takes), so it can correct the call. Without blockType the tool works as before: a search by words.