Capabilities / Connected data

How Unity processes queries and large datasets

Understand DuckDB, source calculations, consistent text rules, large inputs, result limits, paging, and read progress.

X4 Tech SolutionsUpdated November 2, 20264 min read

Unity calculates report results with read-only DuckDB SQL over the report's saved model: it reads the source outputs it needs and finishes the calculation in its own engine.

A query can process millions of rows and return a few numbers. Those input rows are different from the rows displayed in a chart, returned by an inspection tool, or loaded into a table window.

Where calculations run

ScenarioExecution
A calculation over a SQL Server sourceUnity's engine completes it; SQL Server can first reduce the outputs it sends (below).
Text comparisons, grouping, or sortingUnity's engine applies its consistent text rules. Source collation is not inspected or cached for these reads.
A join across different databases, files, or API outputsUnity reads the referenced outputs and computes the combined result.
An imported CSVDuckDB queries the stored typed Parquet dataset.
Snowflake or Databricks entities in new report SQLThe source supplies its saved entity-query output; Unity completes the report calculation.

Source entity queries use their connector's native dialect or API request descriptor. Report calculations use named model entities and DuckDB SQL. See Semantic models and Query inspection.

Reduce work at the source

SQL Server can do useful work first. Proven source reductions can select only needed columns, apply compatible restrictions, or return partial sums and counts that Unity combines. If a reduction cannot be proved safe, Unity reads the full entity output instead.

This can reduce data transfer, but it does not make every query fast. Joins, distinct counts, median calculations, wide text, and sorting can require more work than a simple sum.

Understand input and output limits

QuantityMeaning
Source-input rowsNo fixed 250,000-row cap. Available memory, staging/spill disk, and execution safety budgets bound the work.
A complete report-query responseBounded by the endpoint's row and byte limits. An oversized complete result fails explicitly.
SQL Playground previewDefaults to 1,000 returned rows; removing that preview limit still leaves runtime output limits.
Non-paged HTML datasetMust return a complete result of at most 10,000 rows.
Matrix hierarchyIts query computes every level and the grand total from their own rows. Any number of grouped rows: the complete result stays on the server and the matrix reads the rows it shows.
Detail tableRetrieves windows rather than sending its entire result to the browser.
CSV uploadDefault file-size limit is 2 GB; there is no fixed row-count cap. The dialog shows the configured limit.

For example, SELECT SUM(amount) FROM orders can calculate over millions of orders while returning one row. A table window showing 200 orders does not contain the data needed to sum all orders in browser JavaScript. Use a separate aggregate query for that total.

Page through large results

Native and HTML tables can reuse a temporary ordered result for later windows. It belongs to the same account, report, model revision, SQL, and parameters. Changed filters or sorting require a different read; expired or unavailable results are read again. QueryReportData reads every page of a table, matrix, paged dataset or trial query from one retained result, kept until ten minutes after its last page, so its pages agree.

The retained result is a snapshot from when it was built. Refresh to request current data. It is not a permanent import or a transaction shared by every visual in the report. An authored LIMIT remains part of the analytical query.

Follow a slow calculation

Long browser reads continue as background jobs after the initial wait. Loading progress can show rows processed and a total when known. A source can be computing before it has any rows to send, so an unknown total or a pause before the first row is not itself an error.

Use Cancel to stop a read and refresh to start another. Healthy work can continue beyond a minute, within configured resources and the safety ceiling. The default read ceiling is 15 minutes; a streaming read with no progress for two minutes can stop earlier. Jobs with nobody polling them are also stopped. See Background work.

Compare results at the same scope

Match the report model, measure, row grain, filters, text rules, and source time before comparing a chart, table, HTML visual, or chat answer. Inspection samples and table windows are partial results, while old chat answers retain their original data.

Large row counts alone do not establish a deployment's capacity. The source plan, row width, available memory/disk, and simultaneous work determine how much the backend can process. A resource error is reported rather than quietly returning a partial aggregate.

Questions about your environment? Contact us.