[CALCITE-5406] Support the SELECT DISTINCT ON statement for PostgreSQL dialect by xuzifu666 · Pull Request #4933 · apache/calcite

xuzifu666 · 2026-05-11T11:44:09Z

jira: https://issues.apache.org/jira/browse/CALCITE-5406

cjj2010 · 2026-05-12T03:24:54Z

"DISTINCT ON" is a unique syntax of PGSQL. Would it be more appropriate to implement the parsing support for this function on the babel parser

xuzifu666 · 2026-05-12T03:43:42Z

"DISTINCT ON" is a unique syntax of PGSQL. Would it be more appropriate to implement the parsing support for this function on the babel parser

Babel's only extensibility is:

Configuring additional keywords, join types, and binary operators in config.fmpp;

Adding custom parser methods (such as PostgreSQL's BEGIN/COMMIT) to includes/*.ftl.

Babel itself does not have a separate syntax file; all SELECT-related syntax must be defined in coreParser.jj.

Consistent with QUALIFY Implementation

QUALIFY is a syntax specific to Snowflake, which Calcite implements in the core:

The <QUALIFY> token is defined in core.Parser.jj.

The SqlSelect class adds a qualify field in the core.

The Validator/SqlToRelConverter logic is in the core; DISTINCT ON uses the exact same pattern.

Conformance Controls Semantics, Not Syntax

The current Parser unconditionally parses DISTINCT ON, but semantic validation is controlled by SqlConformance:

// Using `LENIENT conformance` in testing
`fixture().withConformance(SqlConformanceEnum.LENIENT)`

This is consistent with the handling of other extensions such as QUALIFY, LATERAL, and TABLESAMPLE: the Parser is as lenient as possible, and the Validator decides whether to allow based on the dialect.

so in my view：The current implementation conforms to Calcite's architectural conventions and does not require migration to Babel. For adjustments, finer-grained switches can be added to SqlConformance to control the availability of DISTINCT ON.

By the way: Besides PG, ClickHouse also supports DISTINCT ON.

xiedeyantu · 2026-05-12T03:52:33Z

I attempted to verify this across several databases (though my testing may not have been exhaustive) and indeed found that only PostgreSQL supports this syntax; furthermore, DISTINCT ON does not appear to be part of the standard SQL specification. I would suggest implementing this within Babel (specifically, by adding a configuration parameter to control it); we can also wait to see if anyone else has any better suggestions.

xuzifu666 · 2026-05-12T07:05:45Z

I attempted to verify this across several databases (though my testing may not have been exhaustive) and indeed found that only PostgreSQL supports this syntax; furthermore, DISTINCT ON does not appear to be part of the standard SQL specification. I would suggest implementing this within Babel (specifically, by adding a configuration parameter to control it); we can also wait to see if anyone else has any better suggestions.

Good suggestion, I had add a config to control it in babel.

xiedeyantu · 2026-05-12T07:57:47Z

      "parserImpls.ftl"
    ]
+
+    includeDistinctOn: true


It seems like there’s a better place for this parameter.

cjj2010 · 2026-05-12T09:32:21Z

Would it be better to add a test case similar to "SELECT DISTINCT ON (deptno)" here? I tried executing "SELECT DISTINCT ON (deptno)" and it didn't generate any error

OK, I had added related test cases in SqlParserTest.java.

Perhaps it's a problem with my input method. I intended to add"SELECT DISTINCTON (deptno) empno" and "SELECT DISTINCT ON(deptno) empno". Scenarios like this, where text is written consecutively, should normally result in an error. However, the outcomes of the two situations I executed did not meet my expectations.
https://www.postgresql.org/docs/current/queries-select-lists.html#QUERIES-DISTINCT

As far as I know, distinct on needs to be followed by an order by statement.

As far as I know, distinct on needs to be followed by an order by statement.

Perhaps I should elaborate and add test cases similar to "SELECT DISTINCTON (deptno) empno, ename FROM emp ORDER BY deptno, empno"

sonarqubecloud · 2026-05-13T03:58:17Z

Quality Gate passed

Issues
3 New issues
0 Accepted issues

Measures
0 Security Hotspots
91.2% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

mihaibudiu · 2026-06-04T23:48:17Z

What is the status of this PR?

xuzifu666 · 2026-06-05T02:29:35Z

What is the status of this PR?

Currently I should had addressed all comments, maybe need a further review @mihaibudiu If you're interested, please help review this PR～

mihaibudiu · 2026-06-05T03:09:42Z

+  @BaseMessage("SELECT DISTINCT ON requires an ORDER BY clause")
+  ExInst<SqlValidatorException> distinctOnRequiresOrderBy();
+
+  @BaseMessage("SELECT DISTINCT ON expressions must match initial ORDER BY expressions")


what does "initial" mean here?

This could be ambiguous, so I deleted the word.

mihaibudiu · 2026-06-05T03:11:06Z

+        // not in the SELECT list.
        final SqlValidatorScope selectScope3 =
-            select.isDistinct()
+            (select.isDistinct() && !select.isDistinctOn())


So for distinctOn() isDistinct is true as well?
Or you can have SELECT DISTINCT DISTINCT ON(...)?
If yes, I hope there is a test

And then you won't need an aggregate because the result has only 1 row?

And then you won't need an aggregate because the result has only 1 row?

Yes, that’s the reason. I also added tests for DISTINCT and DISTINCT ON; in fact, this syntax isn't supported and results in an immediate error.

SELECT DISTINCT DISTINCT ON(...) is also not support in PostgreSQL, we can refer this link: https://onecompiler.com/postgresql/44sqp4qfd

mihaibudiu · 2026-06-05T03:12:26Z

 *
 * <ul>
- * <li>0: distinct ({@link SqlLiteral})</li>
+ * <li>0: keywordList ({@link SqlNodeList})</li>


looks like there were some bugs in this comment...

Yes, there is no need. I changed it.

mihaibudiu · 2026-06-05T03:22:38Z

+        .ok();
+
+    // DISTINCT ON with qualified column reference
+    f.withSql("SELECT DISTINCT ON (e.deptno) e.deptno AS x, e.empno "


Does this work if the ORDER BY column is not qualified?
Maybe there is such a test already

It work well, I also added related test in last commit.

mihaibudiu · 2026-06-05T03:24:23Z

    includeIntervalWithoutQualifier: true
    includeStarExclude: true
    includeSelectBy: true
+    includeDistinctOn: true


I think you need to edit the reference.md grammar too, if it's possible to show babel-only extensions

Make sense, done.

mihaibudiu · 2026-06-05T03:27:38Z


    if (select.isDistinct()) {
-      distinctify(bb, true);
+      if (!select.isDistinctOn()) {


apparently it's impossible to have both DISTINCT and DISTINCT ON, so perhaps isDistinct() should return 'false' for DISTINCT ON

You are right, I changed here.

mihaibudiu · 2026-06-12T03:09:03Z

What's going on with this PR?

xuzifu666 · 2026-06-18T06:31:06Z

What's going on with this PR?

Sorry！ I missed this message earlier. I will fix the address related issues soon. Thank you very much for the review. @mihaibudiu

…L dialect

xuzifu666 · 2026-06-18T17:15:10Z

I think I had addressed the comments, PTAL @mihaibudiu

mihaibudiu · 2026-06-18T17:32:30Z

 SELECT * REPLACE list contains unknown column(s): DEPTNO
 !error

+# [CALCITE-5406] Support the SELECT DISTINCT ON statement for PostgreSQL dialect


I think I made this suggestion before: if these were validated on postgres, pleas say so in the comment, to spare future reviewers the need to validate these. I have seen tests with incorrect results in the past.

I sincerely apologize; this taught me a valuable lesson. This time, I conducted rigorous testing and cross-verification using PostgreSQL, and the results matched perfectly. Comment had been added and refer to the result: https://onecompiler.com/postgresql/44sqjg9t6

mihaibudiu · 2026-06-18T17:33:24Z

      fetch = operand;
      break;
+    case 11:
+      hints = (SqlNodeList) operand;


this looks like a bug you are fixing

This is a pre-existing bug. While adding case 12 for DISTINCT ON, I also filled in the missing case 11 (hint handling). The code is now correct.

mihaibudiu · 2026-06-18T17:34:20Z

+    return distinctOn;
+  }
+
+  public void setDistinctOn(@Nullable SqlNodeList distinctOn) {


is this needed?
In general immutable data structures are preferred.

Yes, it is not needed; the corresponding code has been removed.

xuzifu666 · 2026-06-19T03:25:19Z

I’ve addressed the recent relevant comments; please help to review them again. @mihaibudiu

sonarqubecloud · 2026-06-19T03:42:57Z

Quality Gate passed

Issues
2 New issues
0 Accepted issues

Measures
0 Security Hotspots
91.7% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

mihaibudiu · 2026-06-20T00:18:16Z

I have approved

xuzifu666 force-pushed the distinct_on_support branch 2 times, most recently from 99a667c to a5629e3 Compare May 12, 2026 02:39

xuzifu666 changed the title ~~[CALCITE-7517] Support DISTINCT ON clause in SELECT statements~~ [CALCITE-5406] Support the SELECT DISTINCT ON statement for PostgreSQL dialect May 12, 2026

xiedeyantu reviewed May 12, 2026

View reviewed changes

cjj2010 reviewed May 12, 2026

View reviewed changes

mihaibudiu approved these changes Jun 5, 2026

View reviewed changes

[CALCITE-5406] Support the SELECT DISTINCT ON statement for PostgreSQ…

9107937

…L dialect

xuzifu666 force-pushed the distinct_on_support branch from f7012ed to 9107937 Compare June 18, 2026 07:10

xuzifu666 added 3 commits June 19, 2026 00:29

Addressed

ccd5c88

Addressed

48c7981

Addressed

6d77168

xuzifu666 requested a review from mihaibudiu June 18, 2026 17:15

mihaibudiu reviewed Jun 18, 2026

View reviewed changes

xuzifu666 added 3 commits June 19, 2026 10:33

Addressed

4226b41

Addressed

9af6fa4

Addressed

cb7a972

Conversation

xuzifu666 commented May 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cjj2010 commented May 12, 2026

Uh oh!

xuzifu666 commented May 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

xiedeyantu commented May 12, 2026

Uh oh!

xuzifu666 commented May 12, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cjj2010 May 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sonarqubecloud Bot commented May 13, 2026

Quality Gate passed

Uh oh!

mihaibudiu commented Jun 4, 2026

Uh oh!

xuzifu666 commented Jun 5, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

xuzifu666 Jun 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mihaibudiu commented Jun 12, 2026

Uh oh!

xuzifu666 commented Jun 18, 2026

Uh oh!

xuzifu666 commented Jun 18, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

xuzifu666 Jun 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

xuzifu666 commented May 11, 2026 •

edited

Loading

xuzifu666 commented May 12, 2026 •

edited

Loading

cjj2010 May 12, 2026 •

edited

Loading

xuzifu666 Jun 19, 2026 •

edited

Loading

xuzifu666 Jun 19, 2026 •

edited

Loading