Skip to content

Conversation

Cyan4973
Copy link
Contributor

@Cyan4973 Cyan4973 commented Apr 28, 2022

fix segfault error identified by @eli-schwartz when running zstreamtest with MALLOC_PERTURB_.

The quicker solution is to use calloc instead of malloc to ensure proper initialization.
I don't anticipate any speed impact, as these memory segments are relatively small.
On the other hand, calloc is bound to result in safer initialization.

Tested using MALLOC_PERTURB_=124, as in #3119. No failure after this patch.

fix segfault error when running zstreamtest with MALLOC_PERTURB_
assumption is that, worst case, for platform that don't support it, it will have no impact
@eli-schwartz
Copy link
Contributor

FWIW, Meson actually generates a totally random number for you with the python code: env['MALLOC_PERTURB_'] = str(random.randint(1, 255))

This is also relatively easy to do in a bash shell (due to the presence of $RANDOM) but I'm not sure how to do it with a generic /bin/sh...

@Cyan4973 Cyan4973 marked this pull request as draft April 28, 2022 16:03
@Cyan4973
Copy link
Contributor Author

Extended tests seem to show that failures are still possible.
It seems more rare, but still not zero.
I would appreciate @eli-schwartz if you could have a try at this branch on your test setup.

@eli-schwartz
Copy link
Contributor

eli-schwartz commented Apr 28, 2022

I tried this branch, and got the following results: https://github.com/mesonbuild/wrapdb/runs/6216706988?check_suite_focus=true

[86/92] Linking target subprojects/zstd-fix3119/build/meson/programs/zstd.exe
FAILED: subprojects/zstd-fix3119/build/meson/programs/zstd.exe subprojects/zstd-fix3119/build/meson/programs/zstd.pdb 
"link"  /MACHINE:x64 /OUT:subprojects/zstd-fix3119/build/meson/programs/zstd.exe subprojects/zstd-fix3119/build/meson/programs/subprojects_zstd-fix3119_build_meson_programs_.._.._.._build_VS2010_zstd_zstd.rc_zstd.res subprojects/zstd-fix3119/build/meson/programs/zstd.exe.p/.._.._.._programs_zstdcli.c.obj subprojects/zstd-fix3119/build/meson/programs/zstd.exe.p/.._.._.._programs_util.c.obj subprojects/zstd-fix3119/build/meson/programs/zstd.exe.p/.._.._.._programs_timefn.c.obj subprojects/zstd-fix3119/build/meson/programs/zstd.exe.p/.._.._.._programs_fileio.c.obj subprojects/zstd-fix3119/build/meson/programs/zstd.exe.p/.._.._.._programs_fileio_asyncio.c.obj subprojects/zstd-fix3119/build/meson/programs/zstd.exe.p/.._.._.._programs_benchfn.c.obj subprojects/zstd-fix3119/build/meson/programs/zstd.exe.p/.._.._.._programs_benchzstd.c.obj subprojects/zstd-fix3119/build/meson/programs/zstd.exe.p/.._.._.._programs_datagen.c.obj subprojects/zstd-fix3119/build/meson/programs/zstd.exe.p/.._.._.._programs_dibio.c.obj subprojects/zstd-fix3119/build/meson/programs/zstd.exe.p/.._.._.._programs_zstdcli_trace.c.obj subprojects/zstd-fix3119/build/meson/programs/zstd.exe.p/.._.._.._lib_common_xxhash.c.obj subprojects/zstd-fix3119/build/meson/programs/zstd.exe.p/.._.._.._lib_common_pool.c.obj subprojects/zstd-fix3119/build/meson/programs/zstd.exe.p/.._.._.._lib_common_zstd_common.c.obj subprojects/zstd-fix3119/build/meson/programs/zstd.exe.p/.._.._.._lib_common_error_private.c.obj "/nologo" "/release" "/nologo" "/DEBUG" "/PDB:subprojects\zstd-fix3119\build/meson/programs\zstd.pdb" "subprojects\zstd-fix3119\build/meson/lib\zstd.lib" "/SUBSYSTEM:CONSOLE" "kernel32.lib" "user32.lib" "gdi32.lib" "winspool.lib" "shell32.lib" "ole32.lib" "oleaut32.lib" "uuid.lib" "comdlg32.lib" "advapi32.lib"
.._.._.._lib_common_pool.c.obj : error LNK2019: unresolved external symbol ZSTD_pthread_create referenced in function POOL_create_advanced
.._.._.._lib_common_pool.c.obj : error LNK2019: unresolved external symbol ZSTD_pthread_join referenced in function POOL_join
subprojects\zstd-fix3119\build\meson\programs\zstd.exe : fatal error LNK1120: 2 unresolved externals
[87/92] Linking target subprojects/zstd-fix3119/build/meson/programs/zstd-frugal.exe
FAILED: subprojects/zstd-fix3119/build/meson/programs/zstd-frugal.exe subprojects/zstd-fix3119/build/meson/programs/zstd-frugal.pdb 
"link"  /MACHINE:x64 /OUT:subprojects/zstd-fix3119/build/meson/programs/zstd-frugal.exe subprojects/zstd-fix3119/build/meson/programs/zstd-frugal.exe.p/.._.._.._programs_zstdcli.c.obj subprojects/zstd-fix3119/build/meson/programs/zstd-frugal.exe.p/.._.._.._programs_timefn.c.obj subprojects/zstd-fix3119/build/meson/programs/zstd-frugal.exe.p/.._.._.._programs_util.c.obj subprojects/zstd-fix3119/build/meson/programs/zstd-frugal.exe.p/.._.._.._programs_fileio.c.obj subprojects/zstd-fix3119/build/meson/programs/zstd-frugal.exe.p/.._.._.._programs_fileio_asyncio.c.obj subprojects/zstd-fix3119/build/meson/programs/zstd-frugal.exe.p/.._.._.._lib_common_pool.c.obj subprojects/zstd-fix3119/build/meson/programs/zstd-frugal.exe.p/.._.._.._lib_common_zstd_common.c.obj subprojects/zstd-fix3119/build/meson/programs/zstd-frugal.exe.p/.._.._.._lib_common_error_private.c.obj "/nologo" "/release" "/nologo" "/DEBUG" "/PDB:subprojects\zstd-fix3119\build/meson/programs\zstd-frugal.pdb" "subprojects\zstd-fix3119\build/meson/lib\zstd.lib" "/SUBSYSTEM:CONSOLE" "kernel32.lib" "user32.lib" "gdi32.lib" "winspool.lib" "shell32.lib" "ole32.lib" "oleaut32.lib" "uuid.lib" "comdlg32.lib" "advapi32.lib"
.._.._.._lib_common_pool.c.obj : error LNK2019: unresolved external symbol ZSTD_pthread_create referenced in function POOL_create_advanced
.._.._.._lib_common_pool.c.obj : error LNK2019: unresolved external symbol ZSTD_pthread_join referenced in function POOL_join
subprojects\zstd-fix3119\build\meson\programs\zstd-frugal.exe : fatal error LNK1120: 2 unresolved externals
[88/92] Linking target subprojects/zstd-fix3119/build/meson/tests/checkTag.exe
ninja: build stopped: subcommand failed.

It didn't get as far as the tests.

@Cyan4973
Copy link
Contributor Author

Strange, this PR should have no impact on such kind of linking error...

@eli-schwartz
Copy link
Contributor

eli-schwartz commented Apr 28, 2022

I haven't tried the dev branch, only the latest release and then this PR branch. So it could be a preexisting issue. All I know is I can't currently compile this PR to test it.

I agree that it doesn't seem like calloc/malloc should affect whether pthread is linkable...

@eli-schwartz
Copy link
Contributor

Alright, I tried the dev branch at https://github.com/mesonbuild/wrapdb/runs/6216706988?check_suite_focus=true

It is still broken there too. Looking around a bit, I think this goes back to the issue in #2976

@Cyan4973
Copy link
Contributor Author

Ideally, we should have a meson build test able to catch such issue.
It seems this PR #2976 managed to pass CI tests.

@eli-schwartz
Copy link
Contributor

This PR when applied to the latest dev branch compiles correctly, but zstreamtest still dies of SIGinvalid. Github Actions log: https://github.com/mesonbuild/wrapdb/runs/6632846325?check_suite_focus=true

@Cyan4973 Cyan4973 mentioned this pull request Oct 13, 2022
@Cyan4973
Copy link
Contributor Author

The first part of the PR, replacing malloc() by calloc(), looks correct, but doesn't fix the intended issue.
This is more likely a race condition, which requires having a deeper look at the pthread translation layer for Windows.

@Cyan4973 Cyan4973 closed this Oct 13, 2022
@Cyan4973 Cyan4973 deleted the fix3119 branch February 10, 2023 00:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants