gh-133390: Support basic completion for sqlite3 command-line interface #133393

tanloong · 2025-05-04T19:05:22Z

This adds tab-completion for the 147 SQLite Keywords. A whitespace is appended to completion candidates for users' convenience and to mimic the behavior of sqlite3 tool.

Issue: Support completion for sqlite3 command-line interface #133390

python-cla-bot · 2025-05-04T19:05:26Z

All commit authors signed the Contributor License Agreement.

bedevere-app · 2025-05-04T19:05:28Z

Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply the skip news label instead.

StanFromIreland

This needs a whatsnew entry, (maybe a note in the docs?), a NEWS entry and tests.

This is quite limited completion so I guess it could work.

Lib/sqlite3/__main__.py

picnixz · 2025-05-04T20:52:23Z

I think we need a separate nodule for this because if we want to have a smarter completion later the code is better to be isolated, so something like sqlite3._completer.

If possible, can we have an autogenerated list of keywords? and those keywords could also be stored in a (private) module-level list.

Lib/sqlite3/__main__.py

Misc/NEWS.d/next/Library/2025-05-05-03-14-08.gh-issue-133390.AuTggn.rst

picnixz · 2025-05-04T20:54:37Z

Please add tests

tanloong · 2025-05-04T21:08:09Z

Yes, tests is coming. I will try to find a way to autogenerate list of keywords.

StanFromIreland · 2025-05-04T21:22:36Z

You can mark the pr as a draft.

Lib/sqlite3/_completer.py

Doc/whatsnew/3.14.rst

tanloong · 2025-05-05T05:08:34Z

Added tests and moved keyword list to module level.

Keywords seems not easy to auto-generate, the SQLite document says they can be accessed by sqlite3_keyword_count() and sqlite3_keyword_name() using C, but I am bad at C and didn't figure out how to call these two at Python level. I think these two functions are not exposed to Python?

StanFromIreland

With #133447 in mind the keywords list would be quite handy (I think hardcoding is fine, it doesn't change often anyway). Maybe it could be moved elsewhere, what do you think Benedikt?

Doc/whatsnew/3.14.rst

Lib/sqlite3/_completer.py

tanloong · 2025-05-05T18:36:16Z

Hi @picnixz, as Stan mentioned, beta freeze is around the corner, but I really hope this can be shipped in 3.14, do you think there is a chance to merge it before beta freeze?

picnixz · 2025-05-10T14:46:21Z

Please, do not force push. See https://devguide.python.org/getting-started/pull-request-lifecycle/.

picnixz · 2025-05-10T14:47:30Z

Lib/sqlite3/_completer.py

+from contextlib import contextmanager
+
+
+_keywords = ("ABORT", "ACTION", "ADD", "AFTER", "ALL", "ALTER", "ALWAYS",


We're anyway in a private module, so name this variable KEYWORDS. Ideally, I really want it coming from C but we can do it in a follow-up PR.

I moved KEYWORDS to C in the latest commit. I am not confident with my C code and look forward to your review. Thank you very much!

picnixz · 2025-05-10T15:27:35Z

Lib/test/test_sqlite3/test_cli.py

+        script = "from sqlite3.__main__ import main; main()"
+        input = b"zzzz\t;\n.quit\n"
+        output = run_pty(script, input, env={"NO_COLOR": "1"})
+        output = re.sub(rb"\x1b\[[0-9;]*[mK]", b"", output)


Let's find a way to avoid this. If this is a bug in run_pty, we should fix it (do we have some FORCE_NO_COLOR envvar perhaps?)

cc @hugovk as the colorization expert

See: #133393 (comment)

I figure it out. run_pty() has no bug. The problem is with GNU Readline, which needs special settings to turn off color.

Using env={"NO_COLOR":"1"} in subprocess does work, but it only stops coloring within what Python can control, like the sqlite> prompt. To disable color in GNU Readline itself, we need to run readline.parse_and_bind("set colored-completion-prefix off").

Oh interesting. Maybe that's something to consider when we add colors?

cc @hugovk

tanloong · 2025-05-10T15:28:16Z

I have pushed the update.

Please, do not force push. See https://devguide.python.org/getting-started/pull-request-lifecycle/.

Sorry, I will pay attention to this from now on.

Lib/test/test_sqlite3/test_cli.py

StanFromIreland · 2025-05-10T15:59:42Z

Lib/test/test_sqlite3/test_cli.py

+    def setUpClass(cls):
+        readline = import_module("readline")
+        if readline.backend == "editline":
+            raise unittest.SkipTest("libedit readline is not supported")


Suggested change

raise unittest.SkipTest("libedit readline is not supported")

self.unittest.SkipTest("libedit readline is not supported")

You mean unittest.TestCase.skipTest(). There are many uses of it in the codebase, but there are also quite a few uses of raise unittest.SkipTest(), for example in test_pdb.py#L4759 which is used under exactly the same occasion (I actually copied that block from pdb to here). I think raising is OK?

It's incorrect to use self.skipTest. We're in a class method so we can only raise the exception. skipTest is a regular method, not a class method.

StanFromIreland · 2025-05-10T16:00:31Z

Lib/test/test_sqlite3/test_cli.py

+            raise unittest.SkipTest("libedit readline is not supported")
+
+    def test_keyword_completion(self):
+        script = "from sqlite3.__main__ import main; main()"


Why not just use run_cli from the interactiveconsole tests above?

With run_cli() both the output and err contain no completion candidates. I failed with run_cli() so I switched to run_pty() when writing the tests.

Then where are the completion candidates sent? The commands should be executed.

StanFromIreland · 2025-05-10T16:17:06Z

I've opened tanloong#1 instead of more suggestions here. It greatly simplifies your tests.

…d back" This reverts commit 9d03730.

"NO_COLOR" only affects color for the `sqlite>` prompt, it does not help for disabling color of completion prefix.

4-space indents; 79 line width; outermost curly braces in column 1 in function definition; blank line after local variable declaration;

tanloong · 2025-05-11T10:23:56Z

The failed test is unrelated to this PR.

tanloong · 2025-05-12T09:01:53Z

Hi @picnixz , I have found way to avoid colored output in run_pty()(explained here) and moved KEYWORDS to C. Could you take a look at this when you have some spare time?

picnixz · 2025-05-15T19:42:58Z

Doc/whatsnew/3.15.rst

+sqlite3
+-------
+
+* Support keyword completion in the :mod:`sqlite3` command-line interface.
+  (Contributed by Tan Long in :gh:`133393`.)


Let's move this under the "Improved modules" section instead of "New features". The "New features" is reserved for PEP-sized items.

picnixz · 2025-05-15T19:43:58Z

Lib/sqlite3/_completer.py

+    if state == 0:
+        _completion_matches = [c + " " for c in SQLITE_KEYWORDS if c.startswith(text.upper())]


Suggested change

if state == 0:

_completion_matches = [c + " " for c in SQLITE_KEYWORDS if c.startswith(text.upper())]

if state == 0:

text_upper = text.upper()

_completion_matches = [c for c in SQLITE_KEYWORDS if c.startswith(text_upper)]

Otherwise, I think text.upper() is going to be called for each c. Then you can do return _completion_matches[state] + " " instead (no need to store the extra whitespace) as it would create a new string while here you can just create a new reference to an existing string.

picnixz · 2025-05-15T19:45:31Z

Lib/test/test_sqlite3/test_cli.py

@@ -200,5 +205,74 @@ def test_color(self):
            self.assertIn('\x1b[1;35mOperationalError (SQLITE_ERROR)\x1b[0m: '
                          '\x1b[35mnear "sel": syntax error\x1b[0m', err)

+@requires_subprocess()


Suggested change

@requires_subprocess()

@requires_subprocess()

Two blank lines to separate classes.

picnixz · 2025-05-15T19:47:36Z

Lib/test/test_sqlite3/test_cli.py

+    def setUpClass(cls):
+        readline = import_module("readline")
+        if readline.backend == "editline":
+            raise unittest.SkipTest("libedit readline is not supported")


It's incorrect to use self.skipTest. We're in a class method so we can only raise the exception. skipTest is a regular method, not a class method.

picnixz · 2025-05-15T19:49:00Z

Lib/test/test_sqlite3/test_cli.py

+        script = textwrap.dedent("""
+            import readline
+            readline.parse_and_bind("set colored-completion-prefix off")
+            from sqlite3.__main__ import main; main()
+        """)


You can have a helper method:

def write_input(self, input): script = textwrap.dedent(""" import readline readline.parse_and_bind("set colored-completion-prefix off") from sqlite3.__main__ import main; main() """) return run_pty(script, input)

picnixz · 2025-05-15T19:52:14Z

Lib/test/test_sqlite3/test_cli.py

+        output = run_pty(script, input, env={"NO_COLOR": "1"})
+        output_lines = output.decode().splitlines()
+        candidates = []
+        for line in output_lines[-2::-1]:


Why are we iterating in reverse order and then sorting the candidates in reverse order as well?

picnixz · 2025-05-15T19:53:21Z

Lib/test/test_sqlite3/test_cli.py

+            if line.startswith(self.PS1):
+                break
+            candidates.append(line.strip())
+        self.assertEqual(sorted(candidates, reverse=True), candidates)


I'm confused here. Are we checking that candidates are already sorted? or maybe you wanted to check against SQLITE_KEYWORDS?

picnixz · 2025-05-15T19:53:34Z

Lib/test/test_sqlite3/test_cli.py

 import unittest

+from _sqlite3 import SQLITE_KEYWORDS


When are we using it?

picnixz · 2025-05-15T19:54:05Z

Lib/test/test_sqlite3/test_cli.py

+
 if __name__ == "__main__":


Suggested change

if __name__ == "__main__":

if __name__ == "__main__":

picnixz · 2025-05-15T19:54:57Z

Modules/_sqlite/module.c

+add_sequence_constants(PyObject *module)
+{
+    PyObject *kwd;
+    const char *_keywords[] = {


When I said "we need it in C", I meant, we need the list of keywords given by sqlite3 directly, not by hardcoding it in C. Hardcoding it in C doesn't help.

bedevere-app · 2025-05-15T19:55:06Z

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

tanloong requested review from berkerpeksag and erlend-aasland as code owners May 4, 2025 19:05

bedevere-app bot mentioned this pull request May 4, 2025

Support completion for sqlite3 command-line interface #133390

Open

bedevere-app bot added the awaiting review label May 4, 2025

StanFromIreland reviewed May 4, 2025

View reviewed changes

Lib/sqlite3/__main__.py Outdated Show resolved Hide resolved

picnixz reviewed May 4, 2025

View reviewed changes

Lib/sqlite3/__main__.py Outdated Show resolved Hide resolved

Lib/sqlite3/__main__.py Outdated Show resolved Hide resolved

Misc/NEWS.d/next/Library/2025-05-05-03-14-08.gh-issue-133390.AuTggn.rst Outdated Show resolved Hide resolved

tanloong marked this pull request as draft May 4, 2025 21:23

bedevere-app bot removed the awaiting review label May 4, 2025

StanFromIreland reviewed May 4, 2025

View reviewed changes

Lib/sqlite3/_completer.py Outdated Show resolved Hide resolved

Lib/sqlite3/_completer.py Show resolved Hide resolved

Doc/whatsnew/3.14.rst Outdated Show resolved Hide resolved

tanloong force-pushed the sqlite3-cli-completion branch 8 times, most recently from 782a599 to 5005d85 Compare May 5, 2025 04:47

tanloong marked this pull request as ready for review May 5, 2025 05:08

bedevere-app bot added the awaiting review label May 5, 2025

StanFromIreland mentioned this pull request May 5, 2025

Colorize sqlite3 CLI #133447

Open

StanFromIreland reviewed May 5, 2025

View reviewed changes

Doc/whatsnew/3.14.rst Outdated Show resolved Hide resolved

Lib/sqlite3/_completer.py Outdated Show resolved Hide resolved

tanloong force-pushed the sqlite3-cli-completion branch from 15019ea to 22652ae Compare May 5, 2025 18:20

picnixz removed request for hugovk and ezio-melotti May 10, 2025 14:45

picnixz reviewed May 10, 2025

View reviewed changes

tanloong added 2 commits May 10, 2025 23:20

Address Bénédikt's review

70f46e9

Remove color handling of output; If CI fails might need to add back

9d03730

picnixz reviewed May 10, 2025

View reviewed changes

Fix run_pty() doesn't return and test hangs

bfcff38

StanFromIreland reviewed May 10, 2025

View reviewed changes

Lib/test/test_sqlite3/test_cli.py Show resolved Hide resolved

StanFromIreland reviewed May 10, 2025

View reviewed changes

Revert "Remove color handling of output; If CI fails might need to ad…

805d997

…d back" This reverts commit 9d03730.

tanloong mentioned this pull request May 10, 2025

Test suggestion tanloong/cpython#1

Closed

tanloong added 7 commits May 11, 2025 01:29

Turn off colored-completion-prefix for readline

276b4a7

No need to pass "NO_COLOR" to run_pty()

09eeac8

"NO_COLOR" only affects color for the `sqlite>` prompt, it does not help for disabling color of completion prefix.

Flip name

fc57d71

Triggering completion on Ubuntu requires 2 tabs

c508069

Move KEYWORDS to C

231b9e7

Improve style of C code

121b069

4-space indents; 79 line width; outermost curly braces in column 1 in function definition; blank line after local variable declaration;

Improve tests

90a86cf

ZeroIntensity added the topic-sqlite3 label May 11, 2025

picnixz self-requested a review May 15, 2025 12:21

picnixz requested changes May 15, 2025

View reviewed changes

bedevere-app bot added awaiting changes and removed awaiting review labels May 15, 2025

		from contextlib import contextmanager


		_keywords = ("ABORT", "ACTION", "ADD", "AFTER", "ALL", "ALTER", "ALWAYS",

	raise unittest.SkipTest("libedit readline is not supported")
	self.unittest.SkipTest("libedit readline is not supported")

		if state == 0:
		_completion_matches = [c + " " for c in SQLITE_KEYWORDS if c.startswith(text.upper())]

gh-133390: Support basic completion for sqlite3 command-line interface #133393

Are you sure you want to change the base?

gh-133390: Support basic completion for sqlite3 command-line interface #133393

Conversation

tanloong commented May 4, 2025 • edited by bedevere-app bot Loading

python-cla-bot bot commented May 4, 2025 • edited Loading

bedevere-app bot commented May 4, 2025

StanFromIreland left a comment

Choose a reason for hiding this comment

picnixz commented May 4, 2025

picnixz commented May 4, 2025

tanloong commented May 4, 2025

StanFromIreland commented May 4, 2025

tanloong commented May 5, 2025 • edited Loading

StanFromIreland left a comment

Choose a reason for hiding this comment

tanloong commented May 5, 2025 • edited Loading

picnixz commented May 10, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

picnixz May 10, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

picnixz May 15, 2025 • edited Loading

Choose a reason for hiding this comment

tanloong commented May 10, 2025

Choose a reason for hiding this comment

tanloong May 11, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tanloong May 10, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

StanFromIreland commented May 10, 2025

tanloong commented May 11, 2025

tanloong commented May 12, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bedevere-app bot commented May 15, 2025

tanloong commented May 4, 2025 •

edited by bedevere-app bot

Loading

python-cla-bot bot commented May 4, 2025 •

edited

Loading

tanloong commented May 5, 2025 •

edited

Loading

tanloong commented May 5, 2025 •

edited

Loading

picnixz May 10, 2025 •

edited

Loading

picnixz May 15, 2025 •

edited

Loading

tanloong May 11, 2025 •

edited

Loading

tanloong May 10, 2025 •

edited

Loading