Add ANSI_QUOTES and SQL mode validation by JanJakes · Pull Request #452 · WordPress/sqlite-database-integration

JanJakes · 2026-07-15T13:02:02Z

Summary

This PR improves SQL mode handling and adds ANSI_QUOTES and ANSII support. This includes:

SQL mode storage: SQL modes use MySQL's native bitmap and canonical order.
Validation: Named and numeric assignments are validated with MySQL-compatible errors.
ANSI_QUOTES: ANSI_QUOTES is supported by the driver, standalone parser, and native lexer.
ANSI Composite ANSI SQL mode is expanded into its component modes.
Quoted identifiers: Fixed backslash behavior in quoted identifiers.
Native builds: Lexer constants are regenerated, and the generator's monorepo path is fixed.

Why

The ANSI_QUOTES SQL mode support keeps reappearing in our recent performance optimization and parser experiments. This PR extracts and completes that functionality.

Additionally, the driver stored arbitrary mode names, accepted invalid assignments, always treated double quotes as string delimiters, did not expand did not expand composite SQL modes, and applied string escaping to quoted identifiers. The standalone parser and native lexer lacked equivalent quoted-identifier handling.

github-actions · 2026-07-15T13:02:55Z

🤖 Lexer benchmark

Changes to lexer-related files were detected and triggered a benchmark:

Config	Base (QPS)	This PR (QPS)	Speedup
no JIT	73,936	73,693	1.00×
tracing JIT	150,578	156,031	1.04×

Note: Hosted runners are noisy, and absolute numbers vary. Treat the results with caution and verify them locally.

To reproduce locally:

cd packages/mysql-on-sqlite && composer run bench-lexer

Map supported SQL mode names to their native MySQL bit values and use the bitmap as the driver's internal representation. Serialize active modes by bit position so duplicates collapse and @@sql_mode follows MySQL's canonical order.

Validate SQL mode names and numeric masks before changing the session state. Report invalid values with MySQL's SQLSTATE 42000 and error 1231, while retaining MySQL's handling for empty list components and incorrect value types.

When ANSI_QUOTES is active, MySQL treats a double-quoted sequence as a quoted identifier rather than a string literal. Emulate this in the lexer by emitting a backtick-quoted identifier token for double-quoted text when the mode is set, mirroring how NO_BACKSLASH_ESCAPES already alters tokenization. The driver already forwards its active SQL modes to the lexer, so no further wiring is needed. See: https://dev.mysql.com/doc/refman/8.4/en/sql-mode.html#sqlmode_ansi_quotes

MySQL's composite ANSI mode is shorthand for a set of component modes. Expand it when sql_mode is set and store the resulting list, so that "@@sql_mode" and individual mode checks reflect the components: REAL_AS_FLOAT, PIPES_AS_CONCAT, ANSI_QUOTES, IGNORE_SPACE, ONLY_FULL_GROUP_BY REAL_AS_FLOAT and ONLY_FULL_GROUP_BY are stored but not yet respected by the emulation. The driver is the source of truth for SQL modes, but the lexer also recognizes the composite ANSI mode directly so it stays correct when used standalone, applying the components that affect tokenization: PIPES_AS_CONCAT, IGNORE_SPACE, and ANSI_QUOTES. See: https://dev.mysql.com/doc/refman/8.4/en/sql-mode.html#sqlmode_ansi

MySQL does not process backslash escape sequences inside quoted identifiers; the bounding quote is escaped only by doubling it. The lexer was applying string-literal backslash escaping to backtick identifiers (and, with the new ANSI_QUOTES support, to double-quoted identifiers), both when scanning for the closing quote and when unquoting the value. As a result, an identifier like `a\nb` resolved to "a<newline>b", and `a\` (a trailing backslash) failed to tokenize because the backslash was treated as escaping the closing quote. Restrict backslash escaping to string literals so identifiers preserve backslashes verbatim. See: https://dev.mysql.com/doc/refman/8.4/en/identifiers.html

brandonpayton

Hi @JanJakes, I read and reviewed this and left some notes and questions. I'm not confident enough in how everything works to say one way or the other today. But everything looks reasonable to me. Thanks!

brandonpayton · 2026-08-03T15:28:17Z

+			'a"b',
+			self::first_token( 'SELECT "a""b"', 'BACK_TICK_QUOTED_ID', array( 'ANSI_QUOTES' ) )->get_value()


How does this reduce a""b to a"b? I think this is probably expected ANSI SQL behavior, but is there any standard we can link to somewhere that says this explicitly?

IIUC, the actual ANSI SQL standard is not publicly available, but is available for purchase. So I was trying to think what kind of doc we could refer to as an explanation.

Also, if this seems silly to you, please ignore it. I just had a question about the quote handling and thought it would be cool if we showed why it is what it is.

@brandonpayton I'm not sure if this is part of the ANSI standard (AI says it actually is), but quoted identifiers in MySQL escape the quote character by doubling it (backslashes don't work in identifiers). So if you want to express a table name like myˋtable in a query, you'd need to write SELECT * FROM ˋmyˋˋtableˋ.

With the ANSI_QUOTES mode enabled, double quotes behave like backticks, so for a table name like my"table, you'd write SELECT * FROM "my""table" (or SELECT * FROM ˋmy"tableˋ).

I guess the MySQL docs are something we could link, as they say:

If the character to be included within the identifier is the same as that used to quote the identifier itself, then you need to double the character.

I'll review how this is documented in the code.

brandonpayton · 2026-08-03T15:50:03Z

+				if ( null !== $value_node->get_first_child_token( WP_MySQL_Lexer::DEFAULT_SYMBOL ) ) {
+					$sql_modes = $this->get_default_sql_modes();
+				} elseif ( is_string( $value ) ) {
+					$sql_modes = explode( ',', rtrim( $value, ' ' ) );


Naive question:
Why just rtrim() and not trim()?

This just mirrors MySQL behavior. For some reason, MySQL trims only the trailing space, so 'ANSI_QUOTES ' is interpreted as 'ANSI_QUOTES', but ' ANSI_QUOTES' is invalid. Good question, though; I need to document this better.

brandonpayton · 2026-08-03T15:53:40Z

+	}
+
+	public function testSqlModesAcceptNumericBitmap() {
+		$this->assertQuery( 'SET sql_mode = 4294967299' );


Nit: It might be nice to compose the number of specific bits so a human can better confirm how the expected mode corresponds to the assigned value. Same for other places mode numbers are used in these tests.

brandonpayton · 2026-08-03T15:57:42Z

+		$this->assertQuery( 'SET sql_mode = 16' );
+		$this->assertQuery( 'SELECT @@sql_mode AS mode;' );
+		$this->assertSame( ',', $this->last_result[0]->mode );
+		$this->assertFalse( $this->engine->is_sql_mode_active( 'NOT_USED' ) );


What does this do?

I think this sets the mode to:

'ONLY_FULL_GROUP_BY' => 1 << 5,

Why would $this->last_result[0]->mode equal ','?

Ah, that's an edge case that the agent covered in tests. 16 is 1 << 4, which is 'NOT_USED'. In MySQL 5.7, it's literally named ,, under MySQL 8, it's named NOT_USED. I'll document it better in the test.

brandonpayton · 2026-08-03T16:02:04Z

+
+		$this->assertQuery( 'SELECT @@sql_mode AS mode;' );
+		$this->assertSame(
+			'ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION',


Is there a reasonable to way to assert against the actual default bitmap rather than a copy? Is there a reason it would not be a good idea?

I think the explicit value is preferable here. The test is verifying MySQL default, including the canonical order. This ensures we don't accidentally change the default to something else.

brandonpayton · 2026-08-03T16:07:36Z

+		$this->assertQuery( 'CREATE TABLE "my ""tbl""" ("my col" INTEGER);' );
+		$this->assertQuery( 'INSERT INTO "my ""tbl""" ("my col") VALUES (42);' );
+		$this->assertQuery( 'SELECT "my col" FROM "my ""tbl""";' );


What identifier does ""tbl""" represent if two double quotes are combined to make one double quote?

It represents the table name my "tbl" (quotes within identifiers are escaped by doubling them; see also #452 (comment)). I'll see if I can use a better identifier and/or document it better.

JanJakes force-pushed the ansi-quotes branch from 3295335 to b846015 Compare July 15, 2026 17:12

JanJakes changed the title ~~Add ANSI_QUOTES SQL mode support~~ Add ANSI_QUOTES and SQL mode validation Jul 15, 2026

JanJakes force-pushed the ansi-quotes branch 11 times, most recently from b8d6367 to 3f11622 Compare July 17, 2026 07:50

Store SQL modes as a bitmap

e33a05d

Map supported SQL mode names to their native MySQL bit values and use the bitmap as the driver's internal representation. Serialize active modes by bit position so duplicates collapse and @@sql_mode follows MySQL's canonical order.

JanJakes force-pushed the ansi-quotes branch from 3f11622 to 2acb4f5 Compare July 17, 2026 08:49

JanJakes added 4 commits July 17, 2026 11:05

Reject unsupported SQL modes

951b55d

Validate SQL mode names and numeric masks before changing the session state. Report invalid values with MySQL's SQLSTATE 42000 and error 1231, while retaining MySQL's handling for empty list components and incorrect value types.

JanJakes force-pushed the ansi-quotes branch from 2acb4f5 to 647bb25 Compare July 17, 2026 09:07

JanJakes marked this pull request as ready for review July 17, 2026 12:27

JanJakes requested a review from brandonpayton July 17, 2026 12:28

JanJakes mentioned this pull request Jul 28, 2026

Fix MySQL lexer edge cases #460

Closed

brandonpayton approved these changes Aug 3, 2026

View reviewed changes

Navigation Menu

Uh oh!

Add ANSI_QUOTES and SQL mode validation - #452

Add ANSI_QUOTES and SQL mode validation#452
JanJakes wants to merge 5 commits into
trunkfrom
ansi-quotes

JanJakes commented Jul 15, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jul 15, 2026 •

edited

Loading

Uh oh!

brandonpayton left a comment

Uh oh!

brandonpayton Aug 3, 2026

Uh oh!

JanJakes Aug 4, 2026

Uh oh!

brandonpayton Aug 3, 2026

Uh oh!

JanJakes Aug 4, 2026

Uh oh!

brandonpayton Aug 3, 2026

Uh oh!

brandonpayton Aug 3, 2026

Uh oh!

JanJakes Aug 4, 2026

Uh oh!

brandonpayton Aug 3, 2026

Uh oh!

JanJakes Aug 4, 2026

Uh oh!

brandonpayton Aug 3, 2026

Uh oh!

JanJakes Aug 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		'a"b',
		self::first_token( 'SELECT "a""b"', 'BACK_TICK_QUOTED_ID', array( 'ANSI_QUOTES' ) )->get_value()

Uh oh!

Conversation

JanJakes commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Why

Uh oh!

github-actions Bot commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🤖 Lexer benchmark

Uh oh!

brandonpayton left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

JanJakes commented Jul 15, 2026 •

edited

Loading

github-actions Bot commented Jul 15, 2026 •

edited

Loading