Readline issues on new install of FreeBSD 13.1 #2585

dcontiveros · 2023-01-06T19:40:16Z

Too many issues will kill our team's development velocity, drastically.
Make sure you have checked all steps below.

Prerequisite

Make sure your problem is not listed in the common build problems.
Make sure no duplicated issue has already been reported in the pyenv issues. You should look for closed issues, too.
Make sure you are not asking us to help solving your specific issue.
- GitHub issues is opened mainly for development purposes. If you want to ask someone to help solving your problem, go to some community site like Gitter, StackOverflow, etc.
Make sure your problem is not derived from packaging (e.g. Homebrew).
- Please refer to the package documentation for the installation issues, etc.
Make sure your problem is not derived from plugins.
- This repository is maintaining pyenv and the default python-build plugin only. Please refrain from reporting issues of other plugins here.

Description

Platform information (e.g. Ubuntu Linux 16.04): FreeBSD 13.1-RELEASE-p3
OS architecture (e.g. amd64): amd64
pyenv version: pyenv 2.3.9
Python version: Python 3.9.16 (installed via pkg)
C Compiler information (e.g. gcc 7.3): FreeBSD clang version 13.0.0
Please attach the debug trace of the failing command as a gist:
- Run env PYENV_DEBUG=1 <faulty command> 2>&1 | tee trace.log and attach trace.log. E.g. if you have a problem with installing Python, run env PYENV_DEBUG=1 pyenv install -v <version> 2>&1 | tee trace.log (note the -v option to pyenv install).

Recently provisioned a remote VPS on FreeBSD 13.1 and wanted to give pyenv a whirl. Seems I'm hit with readline issues. I attempted to install readline libs, but what I believe to be happening is that pyenv make cannot resolve readline. Here is what I performed:

installed pyenv via bash script
Ran:

pyenv install 3.11.1

Errors reported:

pyenv: /home/blogger/.pyenv/versions/3.11.1 already exists
continue with installation? (y/N) y
Downloading Python-3.11.1.tar.xz...
-> https://www.python.org/ftp/python/3.11.1/Python-3.11.1.tar.xz
Installing Python-3.11.1...
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'readline'
WARNING: The Python readline extension was not compiled. Missing the GNU readline lib?
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/blogger/.pyenv/versions/3.11.1/lib/python3.11/sqlite3/__init__.py", line 57, in <module>
    from sqlite3.dbapi2 import *
  File "/home/blogger/.pyenv/versions/3.11.1/lib/python3.11/sqlite3/dbapi2.py", line 27, in <module>
    from _sqlite3 import *
ModuleNotFoundError: No module named '_sqlite3'
WARNING: The Python sqlite3 extension was not compiled. Missing the SQLite3 lib?
Installed Python-3.11.1 to /home/blogger/.pyenv/versions/3.11.1

Packages installed:

# pkg info | grep -i read
readline-8.2.0                 Library for editing command lines as they are typed
tclreadline-2.3.8              GNU readline for TCL

What I believe is happening is that GNU readline requirement is not being read properly as the output shows:

checking how to link readline libs... none

Debug log attached.

trace.log

The text was updated successfully, but these errors were encountered:

native-api · 2023-01-06T19:46:51Z

Could you please also attach config.log from the build directory?

Do you have Readline development headers and lib files installed? Through what means? (I dunno if BSD have package managers. I know they have "ports" which are kinda that and may need additional compiler flags to link against.)

dcontiveros · 2023-01-06T20:05:59Z

The config.log file does not appear to be created:

[~/.pyenv] $ find . | grep config.log (returns no results)

I made sure to install readline libs using pkg command as stated in my first update. Just to be sure, I logged on and verified:

# pkg info readline
readline-8.2.0
Name           : readline
Version        : 8.2.0
Installed on   : Fri Jan  6 12:31:28 2023 EST
Origin         : devel/readline
Architecture   : FreeBSD:13:amd64
Prefix         : /usr/local
Categories     : devel
Licenses       : GPLv3
Maintainer     : [email protected]
WWW            : https://tiswww.case.edu/php/chet/readline/rltop.html
Comment        : Library for editing command lines as they are typed
Options        :
        BRACKETEDPASTE : on
        DOCS           : on
Shared Libs provided:
        libreadline.so.8
        libhistory.so.8
Annotations    :
        FreeBSD_version: 1301000
        cpe            : cpe:2.3:a:gnu:readline:8.2:::::freebsd13:x64
        repo_type      : binary
        repository     : FreeBSD
Flat size      : 1.78MiB
Description    :
The GNU Readline library provides a set of functions for use by applications
that allow users to edit command lines as they are typed in. Both Emacs and vi
editing modes are available. The Readline library includes additional functions
to maintain a list of previously-entered command lines, to recall and perhaps
reedit those lines, and perform csh-like history expansion on previous commands.

WWW: https://tiswww.case.edu/php/chet/readline/rltop.html

# find /usr/local/ | grep readline
/usr/local/share/licenses/readline-8.2.0
/usr/local/share/licenses/readline-8.2.0/catalog.mk
/usr/local/share/licenses/readline-8.2.0/LICENSE
/usr/local/share/licenses/readline-8.2.0/GPLv3
/usr/local/share/licenses/tclreadline-2.3.8
/usr/local/share/licenses/tclreadline-2.3.8/catalog.mk
/usr/local/share/licenses/tclreadline-2.3.8/LICENSE
/usr/local/share/licenses/tclreadline-2.3.8/BSD3CLAUSE
/usr/local/share/doc/readline
/usr/local/share/doc/readline/CHANGES
/usr/local/share/doc/readline/INSTALL
/usr/local/share/doc/readline/README
/usr/local/share/readline
/usr/local/share/readline/excallback.c
/usr/local/share/readline/fileman.c
/usr/local/share/readline/histexamp.c
/usr/local/share/readline/manexamp.c
/usr/local/share/readline/rl-fgets.c
/usr/local/share/readline/rl.c
/usr/local/share/readline/rlbasic.c
/usr/local/share/readline/rlcat.c
/usr/local/share/readline/rlevent.c
/usr/local/share/readline/rlptytest.c
/usr/local/share/readline/rltest.c
/usr/local/share/readline/rlversion.c
/usr/local/share/info/readline.info
/usr/local/share/vim/vim90/indent/readline.vim
/usr/local/share/vim/vim90/ftplugin/readline.vim
/usr/local/share/vim/vim90/syntax/readline.vim
/usr/local/include/readline
/usr/local/include/readline/chardefs.h
/usr/local/include/readline/history.h
/usr/local/include/readline/keymaps.h
/usr/local/include/readline/readline.h
/usr/local/include/readline/rlconf.h
/usr/local/include/readline/rlstdc.h
/usr/local/include/readline/rltypedefs.h
/usr/local/include/readline/tilde.h
/usr/local/include/editline/readline.h
/usr/local/include/tclreadline.h
/usr/local/lib/libreadline.a
/usr/local/lib/libreadline.so
/usr/local/lib/libreadline.so.8
/usr/local/lib/libreadline.so.8.2
/usr/local/lib/python3.9/lib-dynload/readline.cpython-39.so
/usr/local/lib/python3.9/lib2to3/fixes/__pycache__/fix_xreadlines.cpython-39.opt-1.pyc
/usr/local/lib/python3.9/lib2to3/fixes/__pycache__/fix_xreadlines.cpython-39.opt-2.pyc
/usr/local/lib/python3.9/lib2to3/fixes/__pycache__/fix_xreadlines.cpython-39.pyc
/usr/local/lib/python3.9/lib2to3/fixes/fix_xreadlines.py
/usr/local/lib/python3.9/test/__pycache__/test_readline.cpython-39.opt-1.pyc
/usr/local/lib/python3.9/test/__pycache__/test_readline.cpython-39.opt-2.pyc
/usr/local/lib/python3.9/test/__pycache__/test_readline.cpython-39.pyc
/usr/local/lib/python3.9/test/test_readline.py
/usr/local/lib/libtclreadline.so
/usr/local/lib/libtclreadline-2.3.8.so
/usr/local/lib/tclreadline2.3.8
/usr/local/lib/tclreadline2.3.8/pkgIndex.tcl
/usr/local/lib/tclreadline2.3.8/tclreadlineCompleter.tcl
/usr/local/lib/tclreadline2.3.8/tclreadlineInit.tcl
/usr/local/lib/tclreadline2.3.8/tclreadlineSetup.tcl
/usr/local/libdata/pkgconfig/readline.pc
/usr/local/man/man3/readline.3.gz
/usr/local/man/man3/libssh2_knownhost_readline.3.gz
/usr/local/man/mann/tclreadline.n.gz

native-api · 2023-01-07T07:34:03Z

config.log is in the build directory under /tmp. In the log, it's /tmp/python-build.20230106142056.35755.

dcontiveros · 2023-01-07T16:35:37Z

Log attached.
python-build.20230107112237.54254.log

native-api · 2023-01-07T16:37:46Z

I said config.log, not the build log.
It should be under the build directory, e.g. /tmp/python-build.20230106142056.35755/config.log.

dcontiveros · 2023-01-07T16:38:31Z

I removed all files from /tmp and this is the only log that is being generated.

native-api · 2023-01-07T16:42:33Z

Oh, that's right. Your build succeeds, so the build path is being removed.

Please run pyenv install with -k to keep it.

dcontiveros · 2023-01-07T16:57:18Z

It appears config.log was generated in another location with the -k switch:

[~] $ find . | grep config.log
./.pyenv/sources/3.11.1/Python-3.11.1/config.log

Please see attached. I am seeing linker errors, so am assuming would need to append something to either LDFLAGS or CFLAGS. Let me know.

config.log

native-api · 2023-01-07T17:07:39Z

Okay, it looks like unlike Linux, in BSD, the compiler is configured to not look in /usr/local/ by default:

configure:20925: checking how to link readline libs
configure:20952: cc -pthread -o conftest  -I/home/blogger/.pyenv/versions/3.11.1/include  -L/home/blogger/.pyenv/versions/3.11.1/lib -Wl,-rpath=/home/blogger/.pyenv/versions/3.11.1/lib  conftest.c -lreadline -ldl  -lutil >&5
ld: error: unable to find library -lreadline
cc: error: linker command failed with exit code 1 (use -v to see invocation)
configure:20952: $? = 1
configure: failed program was:
<...>
| char readline ();
| int
| main ()
| {
| return readline ();
|   ;
|   return 0;
| }

native-api · 2023-01-07T17:23:41Z

Could you test the codebase from https://github.com/native-api/pyenv/tree/use_freebsd_ports ?

dcontiveros · 2023-01-07T21:42:35Z

Using your code tree the latest python available is 3.9.7. Is there anyway I can test 3.11.1?

native-api · 2023-01-07T21:43:33Z

Use the use_freebsd_ports branch

dcontiveros · 2023-01-07T22:10:09Z

Same error. Looks like the same output:

I rebuilt to regenerate relevant logs. Please see attached.

[~/.pyenv] $ git branch
  master
* use_freebsd_ports

$ pyenv install 3.11.1 -k
Installing Python-3.11.1...
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'readline'
WARNING: The Python readline extension was not compiled. Missing the GNU readline lib?
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/blogger/.pyenv/versions/3.11.1/lib/python3.11/sqlite3/__init__.py", line 57, in <module>
    from sqlite3.dbapi2 import *
  File "/home/blogger/.pyenv/versions/3.11.1/lib/python3.11/sqlite3/dbapi2.py", line 27, in <module>
    from _sqlite3 import *
ModuleNotFoundError: No module named '_sqlite3'
WARNING: The Python sqlite3 extension was not compiled. Missing the SQLite3 lib?
Installed Python-3.11.1 to /home/blogger/.pyenv/versions/3.11.1

config.log

native-api · 2023-01-07T23:17:03Z

Okay, I was able to set up a FreeBSD VM and reproduce the issue.

It turns out, you don't have pkg-config installed.

configure: WARNING: pkg-config is missing. Some dependencies may not be detected correctly.

Which, counterintuitively, is in the pkgconf Ports package.

All ports modules have .pc files that allow to locate them using pkg-config (which Python's Configure uses as well).

We can still add compiler flags for the case when readline package is installed and pkg-config is missing.
I've checked that either way works.

dcontiveros · 2023-01-08T17:53:09Z

Great find. I installed pkgconf, the replacement for pkg-config but still getting a failure. I'm not using any ports and do not even have the ports tree populated, only binary pacckages. I'm still receiving the error. Please see config.log.

config.log

native-api · 2023-01-08T19:34:26Z

I installed pkgconf, the replacement for pkg-config but still getting a failure

That's strange. Make sure that you have /usr/local/libdata/pkgconfig/readline.pc. Try querying pkg-config about readline (e.g. pkg-config --libs readline). I've searched where the search path to /usr/local/libdata/pkgconfig is configured in pkgconf -- and it appears to be a compiled-in default.

Anyway, I've updated the PR's code, so please try with the use_freebsd_ports branch tip.

If the error persists, please attach the debug log as well as config.log.

dcontiveros · 2023-01-08T20:16:05Z

Ok this seems to have gotten me past readline:

[H: blog] [U: blogger] [~/.pyenv] $ git branch
  master
* use_freebsd_ports

[H: blog] [U: blogger] [~/.pyenv] $ git log --pretty=format:'%h' -n 1
08835d22

[H: blog] [U: blogger] [~/.pyenv] $ find /usr/local/libdata/pkgconfig/ | grep readline
/usr/local/libdata/pkgconfig/readline.pc

[H: blog] [U: blogger] [~/.pyenv] $ pyenv install 3.11.1 -k
Installing Python-3.11.1...
/home/blogger/.pyenv/plugins/python-build/bin/python-build: line 1425: dev/null: No such file or directory
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/blogger/.pyenv/versions/3.11.1/lib/python3.11/sqlite3/__init__.py", line 57, in <module>
    from sqlite3.dbapi2 import *
  File "/home/blogger/.pyenv/versions/3.11.1/lib/python3.11/sqlite3/dbapi2.py", line 27, in <module>
    from _sqlite3 import *
ModuleNotFoundError: No module named '_sqlite3'
WARNING: The Python sqlite3 extension was not compiled. Missing the SQLite3 lib?
Installed Python-3.11.1 to /home/blogger/.pyenv/versions/3.11.1

[H: blog] [U: blogger] [~/.pyenv] $ pyenv shell 3.11.1
[H: blog] [U: blogger] [~/.pyenv] $ python3.11
Python 3.11.1 (main, Jan  8 2023, 15:13:27) [Clang 13.0.0 ([email protected]:llvm/llvm-project.git llvmorg-13.0.0-0-gd7b669b3a3 on freebsd13
Type "help", "copyright", "credits" or "license" for more information.
>>> import readline
>>>

Let me get sqlite up before we close this. Would love a complete build. Thanks!

dcontiveros · 2023-01-08T20:20:45Z

I'm wondering if there needs to be some pkgconf optimizations to look for libraries in /usr/local/lib:

[H: blog] [U: blogger] [~/.pyenv] $ pkgconf --libs readline
-L/usr/local/lib -lreadline
[H: blog] [U: blogger] [~/.pyenv] $ pkgconf --libs sqlite
-L/usr/local/lib -lsqlite
[H: blog] [U: blogger] [~/.pyenv] $ pkg-config --libs readline
-L/usr/local/lib -lreadline
[H: blog] [U: blogger] [~/.pyenv] $ pkg-config --libs sqlite
-L/usr/local/lib -lsqlite

Is there a way to get the build script to use pkgconf/pkg-config by default on FreeBSD?

native-api · 2023-01-08T20:23:37Z

CPython's Configure is supposed to already be using pkg-config to locate libraries.

dcontiveros · 2023-01-08T20:32:37Z

Ok troubleshooting this further, with the pkgconf/pkg-config commands it seems the build script expects sqlite3. The version I had installed was sqlite, which defaults to 2 on FreeBSD.

$ pyenv install 3.11.1 -k
Installing Python-3.11.1...
/home/blogger/.pyenv/plugins/python-build/bin/python-build: line 1425: dev/null: No such file or directory
Installed Python-3.11.1 to /home/blogger/.pyenv/versions/3.11.1

$ pkgconf --libs readline
-L/usr/local/lib -lreadline

$ pkgconf --libs sqlite3
-L/usr/local/lib -lsqlite3

$ pyenv shell 3.11.1

$ python3.11
Python 3.11.1 (main, Jan  8 2023, 15:28:35) [Clang 13.0.0 ([email protected]:llvm/llvm-project.git llvmorg-13.0.0-0-gd7b669b3a3 on freebsd13
Type "help", "copyright", "credits" or "license" for more information.
>>> import sqlite
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'sqlite'
>>> import sqlite3
>>> import readline

I think we are good here. Believe we can close this. Definitely appreciate your help. One last question, will these changes be merged into master pyenv?

dcontiveros · 2023-01-10T15:23:46Z

I grabbed latest merge today, 2369ac2a089e1afaf61753ee1d4bc08b80c1ace7 and I receive the same readline error, although sqlite is fine. I cannot compare the tree with master since it has been deleted. Can we double check?
config.log

dcontiveros · 2023-01-10T16:09:32Z

Testing with 2.3.10

dcontiveros · 2023-01-10T16:11:22Z

Same readline error. Please reopen.

native-api · 2023-01-13T10:18:24Z

Okay, it turns out that -- surprisingly enough -- Python's Configure does NOT use pkg-config to locate Readline.
And the PR's code only worked because my invocation for command -v pkg-config was broken! So when I fixed it when merging...

dcontiveros · 2023-01-13T14:17:37Z

I have been testing on a new VM and it does appear that the crux of this issue is python not using pkg-config for readLine.

Interestingly enough, specifying libedit does allow it to build, as well as setting flags manually. I believe the only way around this would be to just set the flags in the previous fix if FreeBSD is detected. I did read up on this and it does appear that you are correct in that tool chains don't look at /usr/local.

What really bothers me about this is that there IS a readline.pc file available in PKG_CONFIG_PATH. Think only this could be fixed upstream.

native-api · 2023-01-13T18:22:55Z

I've tested that https://github.com/native-api/pyenv/tree/fix_freebsd fixes the probem for me (the URL means https://github.com/native-api/pyenv repo, fix_freebsd branch). Could you confirm?

native-api · 2023-01-13T19:05:19Z

What really bothers me about this is that there IS a readline.pc file available in PKG_CONFIG_PATH. Think only this could be fixed upstream.

Yes, this is an upstream problem.
They already fixed it in 3.12.0a1.

dcontiveros · 2023-01-13T19:05:28Z

Ok this branch works. I have also confirmed pkg info -e lib returns 0, so it should pass that test.

native-api mentioned this issue Jan 7, 2023

Fix using dependencies from Ports in BSD with no pkg-config #2586

Merged

5 tasks

native-api closed this as completed in #2586 Jan 9, 2023

native-api reopened this Jan 13, 2023

native-api mentioned this issue Jan 13, 2023

Fix using dependencies from Ports in FreeBSD that are not searched with pkg-config #2593

Merged

5 tasks

native-api closed this as completed in #2593 Jan 13, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Readline issues on new install of FreeBSD 13.1 #2585

Readline issues on new install of FreeBSD 13.1 #2585

dcontiveros commented Jan 6, 2023 •

edited

Loading

native-api commented Jan 6, 2023

dcontiveros commented Jan 6, 2023 •

edited

Loading

native-api commented Jan 7, 2023

dcontiveros commented Jan 7, 2023

native-api commented Jan 7, 2023

dcontiveros commented Jan 7, 2023

native-api commented Jan 7, 2023 •

edited

Loading

dcontiveros commented Jan 7, 2023

native-api commented Jan 7, 2023

native-api commented Jan 7, 2023

dcontiveros commented Jan 7, 2023

native-api commented Jan 7, 2023

dcontiveros commented Jan 7, 2023

native-api commented Jan 7, 2023 •

edited

Loading

dcontiveros commented Jan 8, 2023 •

edited

Loading

native-api commented Jan 8, 2023 •

edited

Loading

dcontiveros commented Jan 8, 2023

dcontiveros commented Jan 8, 2023

native-api commented Jan 8, 2023

dcontiveros commented Jan 8, 2023

dcontiveros commented Jan 10, 2023

dcontiveros commented Jan 10, 2023

dcontiveros commented Jan 10, 2023

native-api commented Jan 13, 2023

dcontiveros commented Jan 13, 2023

native-api commented Jan 13, 2023

native-api commented Jan 13, 2023

dcontiveros commented Jan 13, 2023

Readline issues on new install of FreeBSD 13.1 #2585

Readline issues on new install of FreeBSD 13.1 #2585

Comments

dcontiveros commented Jan 6, 2023 • edited Loading

Prerequisite

Description

native-api commented Jan 6, 2023

dcontiveros commented Jan 6, 2023 • edited Loading

native-api commented Jan 7, 2023

dcontiveros commented Jan 7, 2023

native-api commented Jan 7, 2023

dcontiveros commented Jan 7, 2023

native-api commented Jan 7, 2023 • edited Loading

dcontiveros commented Jan 7, 2023

native-api commented Jan 7, 2023

native-api commented Jan 7, 2023

dcontiveros commented Jan 7, 2023

native-api commented Jan 7, 2023

dcontiveros commented Jan 7, 2023

native-api commented Jan 7, 2023 • edited Loading

dcontiveros commented Jan 8, 2023 • edited Loading

native-api commented Jan 8, 2023 • edited Loading

dcontiveros commented Jan 8, 2023

dcontiveros commented Jan 8, 2023

native-api commented Jan 8, 2023

dcontiveros commented Jan 8, 2023

dcontiveros commented Jan 10, 2023

dcontiveros commented Jan 10, 2023

dcontiveros commented Jan 10, 2023

native-api commented Jan 13, 2023

dcontiveros commented Jan 13, 2023

native-api commented Jan 13, 2023

native-api commented Jan 13, 2023

dcontiveros commented Jan 13, 2023

dcontiveros commented Jan 6, 2023 •

edited

Loading

dcontiveros commented Jan 6, 2023 •

edited

Loading

native-api commented Jan 7, 2023 •

edited

Loading

native-api commented Jan 7, 2023 •

edited

Loading

dcontiveros commented Jan 8, 2023 •

edited

Loading

native-api commented Jan 8, 2023 •

edited

Loading