feat(optimizer): logical rewrite Filter(row_number() = 1) over PARTITION BY → Aggregate(FIRST_VALUE(... ORDER BY)) by saadtajwar · Pull Request #23824 · apache/datafusion

saadtajwar · 2026-07-22T23:49:13Z

Which issue does this PR close?

Closes Logical rewrite: Filter(row_number() = 1) over PARTITION BY → Aggregate(FIRST_VALUE(... ORDER BY)) #23603

Rationale for this change

DF plans the common "top-1 per group" pattern (ROW_NUMBER() OVER (PARTITION BY ... ORDER BY ...) WHERE rn = 1) as a full window sort followed by a filter - this is expensive on wide payloads because it materializes and ranks every row before discarding most.

This PR adds a logical optimizer rule that recognizes the pattern and rewrites it to GROUP BY partition keys with first_value(col ORDER BY ...), which can pick the winning row per group without a full sort. The rule is gated behind optimizer.enable_row_number_to_aggregate and defaults to off until nested-type GroupsAccumulator support lands in #23601/#23628, since the rewritten aggregate only becomes memory-efficient once that fast path exists.

What changes are included in this PR?

Adds ReplaceFilterTop1, a new logical optimizer rule that matches Filter(rn = 1 | rn <= 1 | rn < 2) over a single row_number() window with a non-empty PARTITION BY, and rewrites it to Projection -> Aggregate(first_value(...) ORDER BY ...) -> child, preserving the filter's output schema. It supports the direct Filter -> Window shape and Filter -> Projection -> Window when the projection passes the row_number column through unchanged.

A new config option datafusion.optimizer.enable_row_number_to_aggregate was added, and config docs and information_schema.slt are updated accordingly.

Are these changes tested?

Yes

Are there any user-facing changes?

Yes, users can opt in with SET datafusion.optimizer.enable_row_number_to_aggregate = true. The new setting is documented in docs/source/user-guide/configs.md and appears in information_schema

saadtajwar · 2026-07-22T23:49:52Z

+        /// efficient once the columnar nested-type `GroupsAccumulator` fast
+        /// path is available; enabling it without that support can route wide
+        /// payloads onto the slow per-group accumulator path.
+        pub enable_row_number_to_aggregate: bool, default = false


I assume we wanted a config for this - and can change the default to true and remove the "disabled by default..." comment when #23601 is closed out - but if not happy to remove!

saadtajwar · 2026-07-22T23:50:42Z

+    let (name, op, val) = match (&**left, &**right) {
+        (
+            Expr::Column(Column { name, .. }),
+            Expr::Literal(ScalarValue::UInt64(Some(val)), _),


When I was testing it looked like the type was always UInt64 - is that a valid assumption? Or should I match on all int types?

saadtajwar · 2026-07-22T23:51:14Z

+
+/// Recognize either `Filter -> Window` or `Filter -> Projection -> Window`,
+/// where the window is a single `row_number()` with a non-empty `PARTITION BY`.
+fn validate_window_input(input: &Arc<LogicalPlan>) -> Option<WindowTop1<'_>> {


I was a bit unsure about this function name/the general structure and return values 😅 happy to receive any feedback here!

saadtajwar · 2026-07-22T23:53:11Z

+}
+
+/// Unalias expression reference
+fn unalias(mut expr: &Expr) -> &Expr {


BTW, looks like this unalias pattern is used in push_down_filter & eliminate_outer_join (although those two are recursive) - happy to consolidate this to a helper in a follow-up if we want!

saadtajwar · 2026-07-22T23:53:57Z

+/// - window has a `PARTITION BY` clause
+/// - gated behind the `optimizer.enable_row_number_to_aggregate` config option (off by default)
+#[derive(Default, Debug)]
+pub struct ReplaceFilterTop1 {}


Happy to rename this optimizer rule, I recognize that having a number in here might be a bit awkward (recurring theme throughout this PR of me not being good at naming things haha)

github-actions · 2026-07-22T23:58:16Z

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details

     Cloning apache/main
    Building datafusion v54.1.0 (current)
       Built [ 108.488s] (current)
     Parsing datafusion v54.1.0 (current)
      Parsed [   0.038s] (current)
    Building datafusion v54.1.0 (baseline)
       Built [ 107.981s] (baseline)
     Parsing datafusion v54.1.0 (baseline)
      Parsed [   0.039s] (baseline)
    Checking datafusion v54.1.0 -> v54.1.0 (no change; assume patch)
     Checked [   0.889s] 223 checks: 223 pass, 30 skip
     Summary no semver update required
    Finished [ 219.758s] datafusion
    Building datafusion-common v54.1.0 (current)
       Built [  35.373s] (current)
     Parsing datafusion-common v54.1.0 (current)
      Parsed [   0.064s] (current)
    Building datafusion-common v54.1.0 (baseline)
       Built [  34.642s] (baseline)
     Parsing datafusion-common v54.1.0 (baseline)
      Parsed [   0.065s] (baseline)
    Checking datafusion-common v54.1.0 -> v54.1.0 (no change; assume patch)
     Checked [   1.046s] 223 checks: 222 pass, 1 fail, 0 warn, 30 skip

--- failure constructible_struct_adds_field: externally-constructible struct adds field ---

Description:
A pub struct constructible with a struct literal has a new pub field. Existing struct literals must be updated to include the new field.
        ref: https://doc.rust-lang.org/reference/expressions/struct-expr.html
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.49.0/src/lints/constructible_struct_adds_field.ron

Failed in:
  field OptimizerOptions.enable_row_number_to_aggregate in /home/runner/work/datafusion/datafusion/datafusion/common/src/config.rs:1360

     Summary semver requires new major version: 1 major and 0 minor checks failed
    Finished [  72.769s] datafusion-common
    Building datafusion-optimizer v54.1.0 (current)
       Built [  26.934s] (current)
     Parsing datafusion-optimizer v54.1.0 (current)
      Parsed [   0.032s] (current)
    Building datafusion-optimizer v54.1.0 (baseline)
       Built [  26.986s] (baseline)
     Parsing datafusion-optimizer v54.1.0 (baseline)
      Parsed [   0.033s] (baseline)
    Checking datafusion-optimizer v54.1.0 -> v54.1.0 (no change; assume patch)
     Checked [   0.215s] 223 checks: 223 pass, 30 skip
     Summary no semver update required
    Finished [  55.125s] datafusion-optimizer
    Building datafusion-sqllogictest v54.1.0 (current)
       Built [ 191.008s] (current)
     Parsing datafusion-sqllogictest v54.1.0 (current)
      Parsed [   0.022s] (current)
    Building datafusion-sqllogictest v54.1.0 (baseline)
       Built [ 192.213s] (baseline)
     Parsing datafusion-sqllogictest v54.1.0 (baseline)
      Parsed [   0.025s] (baseline)
    Checking datafusion-sqllogictest v54.1.0 -> v54.1.0 (no change; assume patch)
     Checked [   0.119s] 223 checks: 223 pass, 30 skip
     Summary no semver update required
    Finished [ 386.340s] datafusion-sqllogictest

codecov-commenter · 2026-07-23T17:01:25Z

Codecov Report

❌ Patch coverage is 76.26667% with 89 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.74%. Comparing base (8abc085) to head (45c0461).

Files with missing lines	Patch %	Lines
datafusion/optimizer/src/replace_filter_top1.rs	76.20%	48 Missing and 41 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #23824      +/-   ##
==========================================
- Coverage   80.75%   80.74%   -0.01%     
==========================================
  Files        1089     1090       +1     
  Lines      369164   369539     +375     
  Branches   369164   369539     +375     
==========================================
+ Hits       298127   298402     +275     
- Misses      53282    53338      +56     
- Partials    17755    17799      +44

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

saadtajwar · 2026-07-23T17:18:05Z

@zhuqi-lucas & @tohuya6 - ready for review for #23603! Looking forward to the feedback! 😁 🎉

saadtajwar · 2026-07-23T17:19:45Z

Another note: I'll make a follow-up issue for this comment from @2010YOUY01 and address in a subsequent PR!

saadtajwar added 9 commits July 21, 2026 19:42

feat: logical rewrite for filter on rn=1 to aggregate first_value

9293195

helper functions for predicate and window input validation

7549d87

finished rewrite

7e741ec

starting on tests

eb4bead

added more tests

c5994d5

added docs

ee28107

adding more comments

271d291

more comments

02a4c34

removing debug stuff

daa3858

github-actions Bot added documentation Improvements or additions to documentation optimizer Optimizer rules sqllogictest SQL Logic Tests (.slt) common Related to common crate labels Jul 22, 2026

saadtajwar commented Jul 22, 2026

View reviewed changes

github-actions Bot added the auto detected api change Auto detected API change label Jul 22, 2026

fixing CI bugs;

9ae64e2

github-actions Bot added the core Core DataFusion crate label Jul 23, 2026

config change

3ac1253

Merge branch 'main' into saadtajwar/logical-rewrite-filter-top1

e4b7e19

saadtajwar marked this pull request as ready for review July 23, 2026 17:17

saadtajwar mentioned this pull request Jul 23, 2026

Logical rewrite: Filter(row_number() = 1) over PARTITION BY → Aggregate(FIRST_VALUE(... ORDER BY)) #23603

Open

saadtajwar changed the title ~~feat(optimizer): logical rewrite Filter(row_number() = 1) over PARTITION BY → Aggregate(FIRST_VALUE(... ORDER BY)) (DRAFT)~~ feat(optimizer): logical rewrite Filter(row_number() = 1) over PARTITION BY → Aggregate(FIRST_VALUE(... ORDER BY)) Jul 23, 2026

tohuya6 suggested changes Jul 23, 2026

View reviewed changes

Comment thread datafusion/optimizer/src/replace_filter_top1.rs Outdated

saadtajwar and others added 2 commits July 23, 2026 15:02

making access of config function registry safer

6d4beb5

Merge branch 'main' into saadtajwar/logical-rewrite-filter-top1

45c0461

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(optimizer): logical rewrite Filter(row_number() = 1) over PARTITION BY → Aggregate(FIRST_VALUE(... ORDER BY))#23824

feat(optimizer): logical rewrite Filter(row_number() = 1) over PARTITION BY → Aggregate(FIRST_VALUE(... ORDER BY))#23824
saadtajwar wants to merge 14 commits into
apache:mainfrom
saadtajwar:saadtajwar/logical-rewrite-filter-top1

saadtajwar commented Jul 22, 2026 •

edited

Loading

Uh oh!

saadtajwar Jul 22, 2026 •

edited

Loading

Uh oh!

saadtajwar Jul 22, 2026

Uh oh!

saadtajwar Jul 22, 2026

Uh oh!

saadtajwar Jul 22, 2026

Uh oh!

saadtajwar Jul 22, 2026

Uh oh!

github-actions Bot commented Jul 22, 2026 •

edited

Loading

Uh oh!

codecov-commenter commented Jul 23, 2026 •

edited

Loading

Uh oh!

saadtajwar commented Jul 23, 2026

Uh oh!

saadtajwar commented Jul 23, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

saadtajwar commented Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

saadtajwar Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

saadtajwar Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

saadtajwar Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

saadtajwar Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

saadtajwar Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov-commenter commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

saadtajwar commented Jul 23, 2026

Uh oh!

saadtajwar commented Jul 23, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

saadtajwar commented Jul 22, 2026 •

edited

Loading

saadtajwar Jul 22, 2026 •

edited

Loading

github-actions Bot commented Jul 22, 2026 •

edited

Loading

codecov-commenter commented Jul 23, 2026 •

edited

Loading