Native Lowering & Final Plans

Native lowering converts a completed DataFusion ExecutionPlan tree into the query engine's QueryPlan IR. When the converted plan wins final selection, it replaces the original plan with a wrapper node that executes the QueryPlan on the native engine instead.

It runs after DataFusion physical planning has finished, so it translates decisions DataFusion already made — aggregate mode, join shape, scan pushdown, partitioning, output order — rather than replanning them. The two plan models differ in one important way: ExecutionPlan is an open trait, so DataFusion can produce any operator shape, while QueryPlan is a closed, DataFusion-free set of nodes owned by the engine. Lowering handles only the shapes it can map exactly and declines the rest with a structured reason (step 1).

To DataFusion, all of this is ordinary machinery: three PhysicalOptimizerRules and, when selection succeeds, an ExecutionPlan wrapper node in the completed plan.

The planning path is:

DataFusion ExecutionPlan
  -> normalize and validate the completed physical shape
  -> lower one supported candidate to QueryPlan
  -> optimize IR, apply cost selection, and check execution capability
  -> NexusNativeExec or NexusNativeComposedExec
  -> capability proof and, when needed, a QueryAdmissionTicket

Runtime admission and device-memory accounting begin later. See Admission & Memory Governance.

Session wiring

with_nexus_native installs three physical optimizer rules around DataFusion's own rules:

Rule	Position	Responsibility
datafusion_nexus_native_count_only	Before aggregate_statistics	Capture safe zero-column count shapes before DataFusion replaces them with statistics.
datafusion_nexus_native	After the other physical optimizer rules	Make the final native selection and retain its structured diagnostics. This public name is also the deduplication key.
datafusion_nexus_final_plan_requirement	After native selection, before SanityCheckPlan	Enforce optional requirements against the exact completed plan.

The entry points are src/session.rs for rule ordering and NexusNativePhysicalOptimizerRule::optimize in src/planner/native.rs for the rewrite. The rule is idempotent: an admission wrapper, NexusNativeExec, NexusNativeComposedExec, or a certified mixed plan is not lowered again.

Lowering pipeline

1. Recognize a physical candidate

src/native/normalize.rs removes only known transparent wrappers and builds a NormalizedNativePlan for an exact supported shape. Operator, expression, type, source, and ordering checks stay in src/native/validate.rs, src/native/expression_contract/, and the source-specific lowering code.

Normalization is deliberately narrower than DataFusion's plan model. A miss returns a stable CapabilityReason; it does not guess at an equivalent plan.

2. Build engine-owned IR

src/native/lowering/ recursively translates the normalized tree with QueryPlanBuilder. The result is an append-only DAG of engine PlanNodes with an explicit ResultSink. The host sink preserves the DataFusion output schema, including schema metadata.

Declared ordering is part of the replacement contract. Lowering must prove the native plan reproduces it or append a supported native sort. Otherwise the candidate remains on DataFusion with NativeOrderedOutputUnsupported.

3. Select an executable native plan

replace_lowered_root_or_keep_plan in src/planner/native/attempt.rs accepts only one lowered plan with one selected candidate. It then:

1. runs the engine-owned native optimizer;
2. rechecks declared output ordering;
3. applies the engine-owned native cost decision; and
4. builds capability diagnostics and rejects unsupported bounded execution before creating a wrapper.

Every non-selection returns the original executable DataFusion plan. A native-capable plan declined by cost is distinct from an unsupported plan.

4. Choose the DataFusion wrapper

The selected candidate is wrapped in one of two ExecutionPlan nodes:

• NexusNativeExec for plans with ordinary sources.
• NexusNativeComposedExec for plans that read named GPU-resident sources. Its children are the explicit GPU producers, and the wrapper checks that the required and provided input names match exactly. This is GPU-to-GPU composition, not arbitrary relational fragment extraction.

The planner also attaches admission evidence. Each selected fragment passes engine preflight and receives one QueryPlanCapabilityProof. When the session config is set up for planner-side admission (an admission source and no QueryServiceHandle), the proofs are combined into one QueryAdmissionTicket bound to the plan. The ticket records what the query needs; device and grant acquisition wait until execution. Two cases create no ticket: configs that already hold a QueryServiceHandle, because execution uses that handle's context, and plans with a cuGraph root. Graph roots get no planner-time ticket because their lakehouse footer metadata may still be resolving — a ticket minted now would freeze incomplete cardinality evidence. The root mints its own ticket once that metadata is bound to the exact plan.

Selection, fallback, and mixed execution

PlanningReport is the public planning evidence. Assert its structured outcomes rather than formatted plan text:

Outcome	Meaning
Selected	The candidate became native IR and passed final selection.
NotSupported	A CapabilityReason identifies the unsupported semantic, source, ordering, or execution contract.
NotSelectedByCost	The candidate is supported, but the native cost decision retained DataFusion and records its evidence.

The installed rule instance retains only its most recent completed lowering attempt and exposes it through last_planning_diagnostics. The bundle pairs the report with native optimizer, source, and capability evidence; it is diagnostic state, not a planning history.

Relational selection is whole-candidate by default. There is one narrow mixed exception in src/planner/native/mixed.rs: when whole-plan lowering misses on an unsupported expression, Nexus may certify a final CPU ILIKE filter over a single local-Parquet source and replace only that source with a native host-output leaf. The certificate checks the exact shape, source, schema, and physical properties, then rebuilds the CPU filter from the new child. Do not generalize this into a subtree extractor; a new mixed boundary needs its own complete property and ownership proof.

Native execution boundary

NexusNativeExec and NexusNativeComposedExec are ordinary DataFusion ExecutionPlans with one bounded output partition. At execution they acquire the admitted native context, run the engine's observed execution entry point, and expose Arrow RecordBatches to the caller. The composed form first turns its producer results into the GpuResidentInputMap consumed by the native plan, avoiding an intermediate host materialization.

The runtime observation snapshot is the source for engine QueryMetrics, DataFusion metrics, and native report projections. Once native execution has started, a failure is returned as a structured error; the query is not replayed through the retained CPU plan.

Baseline preservation and final-plan requirements

NotSupported and NotSelectedByCost are planning outcomes, not errors. The original DataFusion plan remains executable unless the caller configured a completed-plan requirement.

classify_final_plan walks the exact result and reports DataFusion, Native, or Mixed. FinalPlanRequirement::NoDataFusionCpu rejects a completed plan that would execute DataFusion CPU operators. A feature-disabled cuGraph terminal is exempt because it executes neither CPU nor GPU work. An unsatisfied requirement returns FinalPlanRequirementUnsatisfied; Rejected is the terminal planning outcome, not another physical node.

Extending native lowering

Keep changes at the owner of the missing contract:

Change	Extend
Recognize another DataFusion physical shape	src/native/normalize.rs, src/native/validate.rs, and the existing module in src/native/lowering/
Add or change native IR semantics	crates/nexus-query-engine/src/plan/, then its optimizer, capability, and execution owners
Add an explicit GPU producer or graph island	The existing GpuResidentProducer or GraphExecutionPlan boundary in src/native_exec/, then compose through the current wrappers
Add a missing cuDF or cuGraph primitive	components/cudf or components/cugraph; do not emulate it in the DataFusion adapter
Add a missing cuVS primitive	components/cuvs; preserve runtime binding, tensor ownership, and completion contracts
Change selection evidence	src/report.rs, src/capability.rs, and the existing public report projections and contract tests

Extend tests/native_planning/lowering.rs for lowering and report behavior, tests/native_exec_tests.rs for selection, fallback, mixed, and wrapper contracts, and the engine's native_contract_tests for DataFusion-independent behavior. A useful focused loop is:

cargo nextest run -p datafusion-nexus --all-features --test native_planning_tests
cargo nextest run -p nexus-query-engine --all-features --test native_contract_tests