ZeroNSd segfault in Alpine docker

[mcondarelli]

Disclaimer: I know nothing about Rust, Rust programming, Cargo, Rust and such; this could well be entirely my fault, please be lenient.

I was trying to setup this in a container (FROM rust:alpine), but standard zeronsd start -d <domain> <network-id> simply exited with an uninformative "Segmentation fault" error.

I then made a VirualBox VM with the same content as Docker Container, compiled with debugging symbols and fired-up rust-gdb --args /home/mcondarelli/zeronsd/target/debug/zeronsd start ...
Sure enough I got the same error.

alpine:~# ./start.zeronsd.sh 
GNU gdb (GDB) 10.2
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-alpine-linux-musl".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /home/mcondarelli/zeronsd/target/debug/zeronsd...
(gdb) r
Starting program: /home/mcondarelli/zeronsd/target/debug/zeronsd start -d <domain> <network-id>
[New LWP 12918]

Thread 1 "zeronsd" received signal SIGSEGV, Segmentation fault.
0x0000000000000000 in ?? ()
(gdb) bt
#0  0x0000000000000000 in ?? ()
#1  0x00007ffff76ffc7a in openssl::ssl::SslMethod::tls () at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/openssl-0.10.35/src/ssl/mod.rs:324
#2  0x00007ffff76f53c2 in native_tls::imp::TlsConnector::new (builder=0x7ffffffedcc0) at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/native-tls-0.2.7/src/imp/openssl.rs:257
#3  0x00007ffff76f6c32 in native_tls::TlsConnectorBuilder::build (self=0x7ffffffedcc0) at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/native-tls-0.2.7/src/lib.rs:433
#4  0x00007ffff75e26e0 in reqwest::connect::Connector::new_default_tls<core::option::Option<std::net::ip::IpAddr>> (http=..., tls=..., proxies=Arc(strong=2, weak=0) = {...}, user_agent=..., 
    local_addr=<error reading variable: Cannot access memory at address 0x0>, nodelay=true) at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/reqwest-0.11.4/src/connect.rs:220
#5  0x00007ffff75b57ed in reqwest::async_impl::client::ClientBuilder::build (self=...) at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/reqwest-0.11.4/src/async_impl/client.rs:253
#6  0x00007ffff75b654d in reqwest::async_impl::client::Client::new () at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/reqwest-0.11.4/src/async_impl/client.rs:1088
#7  0x00007ffff75235cc in zerotier_central_api::apis::configuration::{{impl}}::default () at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/zerotier-central-api-1.0.2/src/apis/configuration.rs:45
#8  0x00007ffff70603a2 in zeronsd::utils::central_config (token=<error reading variable: Cannot access memory at address 0x5>) at src/utils.rs:21
#9  0x00007ffff6fc0f36 in zeronsd::start (args=0x7ffff8002e38) at src/main.rs:60
#10 0x00007ffff6fc4856 in zeronsd::main () at src/main.rs:211
(gdb) 

An attempt to follow initialization seems to crash when stepping into /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/native-tls-0.2.7/src/imp/openssl.rs#94 ..> ONCE.call_once(openssl_probe::init_ssl_cert_env_vars);.

What am I doing so wrong?

[erikh]

What does your script have it it? Also some details about how you built it would be good. Sent from ProtonMail mobile

…

-------- Original Message --------

On Aug 31, 2021, 7:50 AM, Mauro Condarelli wrote: Disclaimer: I know nothing about Rust, Rust programming, Cargo, Rust and such; this could well be entirely my fault, please be lenient. I was trying to setup this in a container (FROM rust:alpine), but standard zeronsd start -d <domain> <network-id> simply exited with an uninformative "Segmentation fault" error. I then made a VirualBox VM with the same content as Docker Container, compiled with debugging symbols and fired-up rust-gdb --args /home/mcondarelli/zeronsd/target/debug/zeronsd start ... Sure enough I got the same error. alpine:~# ./start.zeronsd.sh GNU gdb (GDB) 10 . 2 Copyright (C) 2021 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type " show copying " and " show warranty " for details. This GDB was configured as " x86_64-alpine-linux-musl " . Type " show configuration " for configuration details. For bug reporting instructions, please see: <https://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type " help " . Type " apropos word " to search for commands related to " word " ... Reading symbols from /home/mcondarelli/zeronsd/ target /debug/zeronsd... (gdb) r Starting program : /home/mcondarelli/zeronsd/ target /debug/zeronsd start -d <domain> <network-id> [New LWP 12918 ] Thread 1 " zeronsd " received signal SIGSEGV, Segmentation fault. 0x0000000000000000 in ?? () (gdb) bt # 0 0x0000000000000000 in ?? () # 1 0x00007ffff76ffc7a in openssl::ssl::SslMethod::tls () at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/openssl-0.10.35/src/ssl/mod.rs:324 # 2 0x00007ffff76f53c2 in native_tls::imp::TlsConnector::new (builder=0x7ffffffedcc0) at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/native-tls-0.2.7/src/imp/openssl.rs:257 # 3 0x00007ffff76f6c32 in native_tls::TlsConnectorBuilder::build (self=0x7ffffffedcc0) at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/native-tls-0.2.7/src/lib.rs:433 # 4 0x00007ffff75e26e0 in reqwest::connect::Connector::new_default_tls<core::option::Option<std::net::ip::IpAddr>> (http=..., tls=..., proxies=Arc(strong=2, weak=0) = {...}, user_agent=..., local_addr=<error reading variable : Cannot access memory at address 0x0 >, nodelay=true) at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/reqwest- 0 . 11 . 4 /src/connect.rs: 220 # 5 0x00007ffff75b57ed in reqwest::async_impl::client::ClientBuilder::build (self=...) at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/reqwest-0.11.4/src/async_impl/client.rs:253 # 6 0x00007ffff75b654d in reqwest::async_impl::client::Client::new () at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/reqwest-0.11.4/src/async_impl/client.rs:1088 # 7 0x00007ffff75235cc in zerotier_central_api::apis::configuration::{{impl}}::default () at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/zerotier-central-api-1.0.2/src/apis/configuration.rs:45 # 8 0x00007ffff70603a2 in zeronsd::utils::central_config (token=<error reading variable: Cannot access memory at address 0x5>) at src/utils.rs:21 # 9 0x00007ffff6fc0f36 in zeronsd::start (args=0x7ffff8002e38) at src/main.rs:60 # 10 0x00007ffff6fc4856 in zeronsd::main () at src/main.rs:211 (gdb) An attempt to follow initialization seems to crash when stepping into /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/native-tls-0.2.7/src/imp/openssl.rs#94 ..> ONCE.call_once(openssl_probe::init_ssl_cert_env_vars);. What am I doing so wrong? — You are receiving this because you are subscribed to this thread. Reply to this email directly, [view it on GitHub](#106), or [unsubscribe](https://github.com/notifications/unsubscribe-auth/AAAET24PHKN5Y3IDSOGRCXDT7TT25ANCNFSM5DEJQPPQ). Triage notifications on the go with GitHub Mobile for [iOS](https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675) or [Android](https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub).

[mcondarelli]

Thanks erikh,
my script is very basic, just set a few Env variables and start zeronsd:

alpine:~# cat start.zeronsd.sh 
#!/bin/sh

export RUSTUP_HOME=/usr/local/rustup
export CARGO_HOME=/usr/local/cargo
export PATH=/usr/local/cargo/bin:$PATH \
export RUST_VERSION=1.54.0

export ZEROTIER_CENTRAL_TOKEN="..."
zeronsd start -d condarelli.it ...

Also build is absolutely basic; first I tried the standard:

cargo install --path .

then I tried the debug/unoptimized way:

cargo build

of course then I changed slightly the script to start the right executable:

...
rust-gdb --args /home/mcondarelli/zeronsd/target/debug/zeronsd start -d condarelli.it ...

The whole scripting stuff is because I normally compile as a "normal user", but zeronsd needs to be "root" to access ZeroTier-One authtoken and other stuff (at least this is how I understood, if I'm wrong please correct me).

[erikh]

thanks for reporting. this is likely due to libc. strong suggestion to just not use alpine (or any musl-based distribution) until I have some time to settle this. ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

…

On Tuesday, August 31st, 2021 at 4:02 PM, Mauro Condarelli ***@***.***> wrote: Thanks erikh, my script is very basic, just set a few Env variables and start zeronsd: alpine:~# cat start.zeronsd.sh #!/bin/sh export RUSTUP_HOME=/usr/local/rustup export CARGO_HOME=/usr/local/cargo export PATH=/usr/local/cargo/bin:$PATH \ export RUST_VERSION=1.54.0 export ZEROTIER_CENTRAL_TOKEN="..." zeronsd start -d condarelli.it ... Also build is absolutely basic; first I tried the standard: cargo install --path . then I tried the debug/unoptimized way: cargo build of course then I changed slightly the script to start the right executable: ... rust-gdb --args /home/mcondarelli/zeronsd/target/debug/zeronsd start -d condarelli.it ... The whole scripting stuff is because I normally compile as a "normal user", but zeronsd needs to be "root" to access ZeroTier-One authtoken and other stuff (at least this is how I understood, if I'm wrong please correct me). — You are receiving this because you commented. Reply to this email directly, [view it on GitHub](#106 (comment)), or [unsubscribe](https://github.com/notifications/unsubscribe-auth/AAAET25BQRG7CYDBQATIYL3T7VNQDANCNFSM5DEJQPPQ). Triage notifications on the go with GitHub Mobile for [iOS](https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675) or [Android](https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub).

[erikh]

Just to be clear, this is not your failure or fault. This is an incompatibility with some library (likely openssl or libc) and our code.

zeronsd needs to run as root to bind on port 53, which is used for DNS.

[erikh]

after some sleuthing I think I figured out what the problem might be. alexcrichton/openssl-probe#19

If you could do me a favor, ensure one of these paths exist on your system where zeronsd is running? This will help me report / patch the issue.

https://github.com/alexcrichton/openssl-probe/blob/master/src/lib.rs#L24

[mcondarelli]

Hi,
thanks for the fast answer.
I am unsure about what you are looking for, anyhow:

alpine:~# ls -l /var/ssl
ls: /var/ssl: No such file or directory
alpine:~# ls -l /usr/share/ssl
ls: /usr/share/ssl: No such file or directory
alpine:~# ls -l /usr/local/ssl
ls: /usr/local/ssl: No such file or directory
alpine:~# ls -l /usr/local/etc/ssl
ls: /usr/local/etc/ssl: No such file or directory
alpine:~# ls -l /usr/local/openssl
ls: /usr/local/openssl: No such file or directory
alpine:~# ls -l /usr/local/etc/openssl
ls: /usr/local/etc/openssl: No such file or directory
alpine:~# ls -l /usr/local/share
total 16
drwxr-xr-x    2 root     root          4096 Aug 30 22:22 ca-certificates
drwxr-sr-x    3 root     root          4096 Aug 30 22:17 doc
drwxr-sr-x    3 root     root          4096 Aug 30 22:17 man
drwxr-sr-x    3 root     root          4096 Aug 30 22:17 systemtap
alpine:~# ls -l /usr/lib/ssl
ls: /usr/lib/ssl: No such file or directory
alpine:~# ls -l /usr/ssl
ls: /usr/ssl: No such file or directory
alpine:~# ls -l /etc/openssl
ls: /etc/openssl: No such file or directory
alpine:~# ls -l /etc/pki/ca-trust/extracted/pem
ls: /etc/pki/ca-trust/extracted/pem: No such file or directory
alpine:~# ls -l /etc/pki/tls
ls: /etc/pki/tls: No such file or directory
alpine:~# ls -l /etc/ssl
total 60
lrwxrwxrwx    1 root     root            25 Aug 30 22:01 cert.pem -> certs/ca-certificates.crt
drwxr-xr-x    2 root     root         20480 Aug 30 22:22 certs
-rw-r--r--    1 root     root           412 Aug 24 17:52 ct_log_list.cnf
-rw-r--r--    1 root     root           412 Aug 24 17:52 ct_log_list.cnf.dist
drwxr-xr-x    2 root     root          4096 Aug 30 22:01 misc
-rw-r--r--    1 root     root         10909 Aug 24 17:52 openssl.cnf
-rw-r--r--    1 root     root         10909 Aug 24 17:52 openssl.cnf.dist
drwxr-xr-x    2 root     root          4096 Aug 30 22:01 private
alpine:~# ls -l /data/data/com.termux/files/usr/etc/tls
ls: /data/data/com.termux/files/usr/etc/tls: No such file or directory
alpine:~# ls -l /boot/system/data/ssl
ls: /boot/system/data/ssl: No such file or directory
alpine:~# 

As you see two of those paths actually exist.
Note the first one (/usr/local/share/ca-certificates/) is empty, while the second (/etc/ssl/certs) has the usual assortment of root certificates.
In /etc/ssl/certs there's a bunch of symlinks to /usr/local/share/ca-certificates/mozilla which contains the actual data.
I don't know if this is relevant.

[erikh]

I'm trying to suss out what is causing the null ptr dereference in your trace. It seems related to the listening address at a second peek, so maybe I jumped the gun on that report, but still, rust should largely prevent this from happening before the project is compiled, so it is almost definitely happening somewhere in the FFI layer. You have the requisite paths and this is probably the wrong tree to be barking up.

that said, I cannot build with alpine at all. can you share the build instructions you used? I am adapting the project like so:

FROM rust:alpine as buildenv

ARG IS_LOCAL=0
ARG VERSION=main
ARG IS_TAG=0

COPY . /code
WORKDIR /code
RUN apk add openssl pkgconfig
RUN sh cargo-docker.sh

FROM alpine:latest

#RUN apt-get update -qq && apt-get install libssl1.1 ca-certificates -y && apt-get autoclean -y && apt-get clean -y
COPY --from=buildenv /usr/local/cargo/bin/zeronsd /usr/bin/zeronsd

ENTRYPOINT ["/usr/bin/zeronsd"]

Compare with our root level dockerfile for reference. It is not building because pkg-config cannot find the headers. I am not an alpine user.

[mcondarelli]

I am able to build using the attached script (test-it.sh)

test.tar.gz

After that I can get to the segfault with:

mcon@cinderella:/tmp/t$ docker exec -ti zt1 /bin/sh
/ # export RUSTUP_HOME=/usr/local/rustup
/ # export CARGO_HOME=/usr/local/cargo
/ # export PATH=/usr/local/cargo/bin:$PATH
/ # export RUST_VERSION=1.54.0
/ # export ZEROTIER_CENTRAL_TOKEN="..."
/ # zeronsd start -d condarelli.it <network-id>
Segmentation fault (core dumped)
/ # 

[erikh]

thanks for preparing this; I'm looking into it now.

[mcondarelli]

Note: as expected redoing (roughly) the same with Debian-bullseye as base seems to work (at least it doesn't segafault): INFO - Welcome to ZeroNS!

[erikh]

yes, our standard zerotier/zeronsd container is built atop debian.

[erikh]

so, I'll have to look into the alpine specific need later. it's 4am here. :) I need to get some rest.

I'm busy with another project, so until I come up for air this may be stalled. Are you ok with another solution for now?

[mcondarelli]

Yup!
No problem.
I need to cross check everything else works in bullseye, but I can live with a slightly fatter container.
If you are interested in digging into this (alpine is a fairly popular base for Docker images) I am willing to help testing.
Thanks for all the good work.

FYI: I'm trying to setup a container with a complete solution for Zerotier/controller/controller-gui and dns.
stay tuned!
;)