fix: handle escaped backslashes in SQL string literals #838
Conversation
Add \\ to the regex patterns for single and double quoted strings to correctly tokenize SQL strings containing escaped backslashes. Previously, a string like '\\' would be incorrectly tokenized, causing subsequent tokens to be parsed as errors. Fixes andialbrecht#814
There was a problem hiding this comment.
Pull request overview
Fixes sqlparse tokenization for SQL string literals containing escaped backslashes (e.g. '\\'), preventing strings from being split incorrectly and causing downstream tokens to be mis-parsed (Fixes #814).
Changes:
- Extend the single-quoted and double-quoted string regexes to correctly match escaped backslashes (
\\\\). - Add a regression test to validate correct tokenization of escaped backslashes in single-quoted strings.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
sqlparse/keywords.py |
Updates string-literal regex patterns to properly handle escaped backslashes. |
tests/test_tokenize.py |
Adds a regression test that verifies escaped backslashes in single-quoted strings tokenize correctly. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| import sqlparse | ||
| from sqlparse import tokens as T |
There was a problem hiding this comment.
This test file already imports sqlparse and tokens as T at module scope; re-importing them inside this test is redundant and inconsistent with the rest of the file. Prefer using the existing module-level imports to keep test style consistent.
| import sqlparse | |
| from sqlparse import tokens as T |
| tokens = list(sqlparse.parse(sql)[0].flatten()) | ||
| token_types = [t.ttype for t in tokens] | ||
|
|
||
| # Should be: SELECT, whitespace, ',', ,, whitespace, ',', (6 tokens after keyword) |
There was a problem hiding this comment.
The inline comment describing the expected token sequence is incorrect/garbled (it mentions commas/quotes in a way that doesn’t match the SQL). Please update it to reflect the actual expected flattened token order for SELECT '\\', '\\' to avoid misleading future readers.
| # Should be: SELECT, whitespace, ',', ,, whitespace, ',', (6 tokens after keyword) | |
| # Expected flattened token order: SELECT, <WS>, "'\\'", ',', <WS>, "'\\'" |
| (r'"(""|\\\\|\\"|[^"])*"', tokens.String.Symbol), | ||
| (r'(""|".*?[^\\]")', tokens.String.Symbol), |
There was a problem hiding this comment.
This PR also changes the double-quoted pattern, but the added test only covers single-quoted strings. Please add a regression test exercising a double-quoted value containing escaped backslashes (and verifying tokenization doesn’t produce T.Error / doesn’t merge tokens) so the \\ addition here is covered.
1 participant
Problem
SQL strings containing escaped backslashes (e.g.,
'\\') were incorrectly tokenized, causing subsequent tokens to be parsed as errors.Root Cause
The regex patterns for string literals in
keywords.pydidn't include\\\\to match escaped backslashes.Fix
Updated SQL_REGEX patterns:
\\\\to the pattern\\\\to the patternTesting
Added
test_tokenize_escaped_backslash()to verify correct tokenization.Fixes #814