recipes: add scipy support #2370

mzakharo · 2020-12-04T20:15:52Z

Addressing tickets #874, #1767, #2061, #2267

This is a port of scipy cross-compile scripts from OpenEmbedded/Poky layer meta-scipy

Once scipy is available, scikit-learn can be made possible

inclement

Very nice, thanks. Looks good to me, barring minor comments.

pythonforandroid/recipes/lapack/__init__.py

pythonforandroid/recipes/numpy/__init__.py

pythonforandroid/recipes/scipy/__init__.py

inclement · 2020-12-19T12:55:12Z

pythonforandroid/recipes/scipy/__init__.py

+    def build_compiled_components(self, arch):
+        self.setup_extra_args = ['-j', str(cpu_count())]
+        super().build_compiled_components(arch)
+        self.setup_extra_args = []


Why the need to reset this to the empty list?

This is copy-paste from the original numpy/__init__.py

Thoughts on whether this is needed?

mzakharo · 2020-12-23T01:12:31Z

some OSX related nonsense did not pass in Travis. This failure is not related to the last commit.

AndreMiras

Looking good enough to me, thanks for the hard work.
@inclement feel free to merge if you think we're good to go

AndreMiras · 2021-01-05T01:25:34Z

Thanks for your patience, maintainers have been rather busy lately. Merging as the last comment is rather minor and might be addressed in a follow up if needed

adin234 · 2021-01-05T14:28:04Z

I tried this and I also built an ndk toolchain with fortran but I encounter this error. @mzakharo

CMake Error at /usr/share/cmake-3.13/Modules/CMakeTestCCompiler.cmake:52 (message):
  The C compiler

    "/opt/android-ndk/toolchains/llvm/prebuilt/linux-x86_64/bin/clang"

  is not able to compile a simple test program.

  It fails with the following output:

    Change Dir: /home/user/hostcwd/.buildozer/android/platform/build-armeabi-v7a/build/other_builds/lapack/armeabi-v7a__ndk_target_21/lapack/build/CMakeFiles/CMakeTmp

    Run Build Command:"/usr/bin/make" "cmTC_b9ac9/fast"
    make -j8 -f CMakeFiles/cmTC_b9ac9.dir/build.make CMakeFiles/cmTC_b9ac9.dir/build
    make[1]: Entering directory '/home/user/hostcwd/.buildozer/android/platform/build-armeabi-v7a/build/other_builds/lapack/armeabi-v7a__ndk_target_21/lapack/build/CMakeFiles/CMakeTmp'
    Building C object CMakeFiles/cmTC_b9ac9.dir/testCCompiler.c.o
    /opt/android-ndk/toolchains/llvm/prebuilt/linux-x86_64/bin/clang --target=armv7-none-linux-androideabi --gcc-toolchain=/opt/android-ndk/toolchains//prebuilt/linux-x86_64 --sysroot=/opt/android-ndk/sysroot  -isystem /opt/android-ndk/sysroot/usr/include -isystem /opt/android-ndk/sysroot/usr/include/arm-linux-androideabi  -target armv7a-linux-androideabi21 -fomit-frame-pointer -march=armv7-a -mfloat-abi=softfp -mfpu=vfp -mthumb -fPIC    -march=armv7-a -mthumb -mfpu=vfpv3-d16 -mfloat-abi=softfp -funwind-tables -no-canonical-prefixes -D__ANDROID_API__=28 -fexceptions  -g -fPIE   -o CMakeFiles/cmTC_b9ac9.dir/testCCompiler.c.o   -c /home/user/hostcwd/.buildozer/android/platform/build-armeabi-v7a/build/other_builds/lapack/armeabi-v7a__ndk_target_21/lapack/build/CMakeFiles/CMakeTmp/testCCompiler.c
    In file included from <built-in>:372:
    <command line>:1:9: warning: '__ANDROID_API__' macro redefined [-Wmacro-redefined]
    #define __ANDROID_API__ 28
            ^
    <built-in>:366:9: note: previous definition is here
    #define __ANDROID_API__ 21
            ^
    1 warning generated.
    Linking C executable cmTC_b9ac9
    /usr/bin/cmake -E cmake_link_script CMakeFiles/cmTC_b9ac9.dir/link.txt --verbose=1
    /opt/android-ndk/toolchains/llvm/prebuilt/linux-x86_64/bin/clang --target=armv7-none-linux-androideabi --gcc-toolchain=/opt/android-ndk/toolchains//prebuilt/linux-x86_64 --sysroot=/opt/android-ndk/platforms/android-28/arch-arm -target armv7a-linux-androideabi21 -fomit-frame-pointer -march=armv7-a -mfloat-abi=softfp -mfpu=vfp -mthumb -fPIC    -march=armv7-a -mthumb -mfpu=vfpv3-d16 -mfloat-abi=softfp -funwind-tables -no-canonical-prefixes -D__ANDROID_API__=28 -fexceptions  -g  -Wl,--fix-cortex-a8  -fPIE -pie -Wl,--gc-sections -Wl,-z,nocopyreloc   -L/home/user/hostcwd/.buildozer/android/platform/build-armeabi-v7a/build/libs_collections/myapp/armeabi-v7a -L/opt/android-ndk/sources/cxx-stl/llvm-libc++/libs/armeabi-v7a  CMakeFiles/cmTC_b9ac9.dir/testCCompiler.c.o  -o cmTC_b9ac9
    /usr/bin/ld: unrecognised emulation mode: armelf_linux_eabi
    Supported emulations: elf_x86_64 elf32_x86_64 elf_i386 elf_iamcu elf_l1om elf_k1om i386pep i386pe
    clang: error: linker command failed with exit code 1 (use -v to see invocation)
    make[1]: *** [CMakeFiles/cmTC_b9ac9.dir/build.make:87: cmTC_b9ac9] Error 1
    make[1]: Leaving directory '/home/user/hostcwd/.buildozer/android/platform/build-armeabi-v7a/build/other_builds/lapack/armeabi-v7a__ndk_target_21/lapack/build/CMakeFiles/CMakeTmp'
    make: *** [Makefile:121: cmTC_b9ac9/fast] Error 2




  CMake will not be able to correctly generate this project.
Call Stack (most recent call first):
  CMakeLists.txt:3 (project)

mzakharo · 2021-01-05T15:01:42Z

@adin234 -> I have not seen this ANDROID_API warning before. Try my project and buildozer file here. (just add scipy to requirements list,). it works for me and NDK r19c, with pre-built fortran image here. Please note I only tried android.api = 27 when building this, so maybe more work needs to be done to get this running on lower API levels.

adin234 · 2021-01-05T15:13:31Z

@mzakharo i think the issue is somewhere in the gcc toolchain. for some reason, it is using this

--gcc-toolchain=/opt/android-ndk/toolchains//prebuilt/linux-x86_64

as you can see, there seems to be something missing between toolchains and prebuilt

mzakharo · 2021-01-05T15:20:42Z

That API warning aside, real issue seesm to be this
/usr/bin/ld: unrecognised emulation mode: armelf_linux_eabi
I doubt it should be using your system's /usr/bin/ld -> maybe you have some environemnt variables set in your build console that confuse cmake?

adin234 · 2021-01-05T15:30:11Z

@mzakharo i agree it might be a cmake error but i can't seem to see where it could've went wrong. I am using your prebuilt fortran image. i also tried using arm64-v8a as my android.arch but it is still using the same gcc toolchain (the seemingly incomplete one). I also tried running it in the kivy docker image to isolate issues caused by my environment but it has the same error

--gcc-toolchain=/opt/android-ndk/toolchains//prebuilt/linux-x86_64

mzakharo · 2021-01-05T15:34:51Z

@adin234 - Since I am unable to reproduce your issue, you might have to do this a different way: clone lapack into a standalone folder, and manually set all the global variables in your console to point to correct NDK folders, until cmake is able to correctly configure, identify proper CC/LD binaries, and compile the lapack project. Once that works, you can try and see what's different in the p4a build flow. Good luck!

adin234 · 2021-01-05T17:57:04Z

@mzakharo don't you have any special configutation in your environment variables? have you tried building it under a fresh venv or docker?

adin234 · 2021-01-06T03:04:18Z

@mzakharo i also looked at your buildozer.spec and noticed that you don't have scipy or lapack.

adin234 · 2021-01-06T03:15:02Z

seems like it is a known issue with CMake. @mzakharo what CMake version are you using?

adin234 · 2021-01-06T04:54:26Z

I was able to get past it by using CMake 3.19.2 but encountered another error

/opt/android-ndk/toolchains/llvm/prebuilt/linux-x86_64/bin/../lib/gcc/arm-linux-androideabi/4.9.x/../../../../arm-linux-androideabi/bin/ld: error: cannot find -lunwind

@mzakharo can you give us the ndk version you are using too?

adin234 · 2021-01-06T07:23:52Z

seems like we can only build it on arm64-v8a

mzakharo · 2021-01-06T14:40:43Z

@adin234 -> thanks for reporting this. I added comments to lapack __init__ with the cmake issues you found. Try the pull request #2402 for armeabi-v7a arch fix.

* add scipy support * flake8 * remove ccache check * remove lapack, scipy from ci due to missing gfortran in CI machines * better recipe names, imports

will702 · 2021-06-06T15:24:56Z

hello @mzakharo what should i do after this . Build failed: arm-linux-androideabi-gfortran not found. See https://github.com/mzakharo/android-gfortran

Command failed: /usr/bin/python3 -m pythonforandroid.toolchain create --dist_name=myapp --bootstrap=sdl2 --requirements=python3,kivy==2.0.0rc1,numpy,plyer,filetype,scikit-fuzzy,matplotlib,scipy,networkx,decorator,opencv==4.5.2 --arch armeabi-v7a --copy-libs --color=always --storage-dir="/content/melon/.buildozer/android/platform/build-armeabi-v7a" --ndk-api=21

ENVIRONMENT:

CUDNN_VERSION = '8.0.4.30'

PYDEVD_USE_FRAME_EVAL = 'NO'

LD_LIBRARY_PATH = '/usr/local/nvidia/lib:/usr/local/nvidia/lib64'

CLOUDSDK_PYTHON = 'python3'

LANG = 'en_US.UTF-8'

HOSTNAME = '341155cc34b0'

OLDPWD = '/'

CLOUDSDK_CONFIG = '/content/.config'

NVIDIA_VISIBLE_DEVICES = 'all'

DATALAB_SETTINGS_OVERRIDES = '{"kernelManagerProxyPort":6000,"kernelManagerProxyHost":"172.28.0.3","jupyterArgs":["--ip=\"172.28.0.2\""],"debugAdapterMultiplexerPath":"/usr/local/bin/dap_multiplexer"}'

ENV = '/root/.bashrc'

PAGER = 'cat'

NCCL_VERSION = '2.7.8'

TF_FORCE_GPU_ALLOW_GROWTH = 'true'

JPY_PARENT_PID = '46'

NO_GCE_CHECK = 'True'

PWD = '/content/melon'

HOME = '/root'

LAST_FORCED_REBUILD = '20210528'

CLICOLOR = '1'

DEBIAN_FRONTEND = 'noninteractive'

LIBRARY_PATH = '/usr/local/cuda/lib64/stubs'

GCE_METADATA_TIMEOUT = '0'

GLIBCPP_FORCE_NEW = '1'

TBE_CREDS_ADDR = '172.28.0.1:8008'

TERM = 'xterm-color'

SHELL = '/bin/bash'

GCS_READ_CACHE_BLOCK_SIZE_MB = '16'

PYTHONWARNINGS = 'ignore:::pip._internal.cli.base_command'

MPLBACKEND = 'module://ipykernel.pylab.backend_inline'

CUDA_VERSION = '11.0.3'

NVIDIA_DRIVER_CAPABILITIES = 'compute,utility'

SHLVL = '1'

PYTHONPATH = '/env/python'

NVIDIA_REQUIRE_CUDA = ('cuda>=11.0 brand=tesla,driver>=418,driver<419 '

'brand=tesla,driver>=440,driver<441 brand=tesla,driver>=450,driver<451')

COLAB_GPU = '0'

GLIBCXX_FORCE_NEW = '1'

PATH = '/root/.buildozer/android/platform/apache-ant-1.9.4/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/tools/node/bin:/tools/google-cloud-sdk/bin:/opt/bin'

LD_PRELOAD = '/usr/lib/x86_64-linux-gnu/libtcmalloc.so.4'

GIT_PAGER = 'cat'

_ = '/usr/local/bin/buildozer'

PACKAGES_PATH = '/root/.buildozer/android/packages'

ANDROIDSDK = '/root/.buildozer/android/platform/android-sdk'

ANDROIDNDK = '/root/.buildozer/android/platform/android-ndk-r19c'

ANDROIDAPI = '29'

ANDROIDMINAPI = '21'

Buildozer failed to execute the last command

The error might be hidden in the log above this error

Please read the full log, and search for it before

raising an issue with buildozer itself.

mriscoc · 2025-03-25T18:54:44Z

How could I update the recipe to support the latest version of SciPy? I have opened this issue: #3116

mzakharo added 4 commits December 4, 2020 15:14

add scipy support

41a7797

flake8

a9714a0

remove ccache check

f915ea1

remove lapack, scipy from ci due to missing gfortran in CI machines

bad1c5a

inclement reviewed Dec 19, 2020

View reviewed changes

better recipe names, imports

7ed787a

AndreMiras approved these changes Dec 28, 2020

View reviewed changes

AndreMiras merged commit 7f4652a into kivy:develop Jan 5, 2021

mzakharo deleted the scipy branch January 6, 2021 14:11

misl6 mentioned this pull request Jul 10, 2022

scipy module fails loading for 32 bit and 64 bit APK builds. #2061

Closed

misl6 mentioned this pull request Jul 22, 2023

scipy with kivy #2861

Closed

Julian-O mentioned this pull request Oct 25, 2023

scipy/_lib/_ccallback_c.so is 64-bit instead of 32-bit kivy/buildozer#880

Closed

Julian-O mentioned this pull request Nov 20, 2023

Custom recipe for scipy fails with permission issue #2267

Closed

recipes: add scipy support #2370

recipes: add scipy support #2370

Uh oh!

Conversation

mzakharo commented Dec 4, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

inclement left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

inclement Dec 19, 2020

Choose a reason for hiding this comment

Uh oh!

mzakharo Dec 19, 2020

Choose a reason for hiding this comment

Uh oh!

mzakharo commented Dec 23, 2020

Uh oh!

AndreMiras left a comment

Choose a reason for hiding this comment

Uh oh!

AndreMiras commented Jan 5, 2021

Uh oh!

adin234 commented Jan 5, 2021

Uh oh!

mzakharo commented Jan 5, 2021

Uh oh!

adin234 commented Jan 5, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mzakharo commented Jan 5, 2021

Uh oh!

adin234 commented Jan 5, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mzakharo commented Jan 5, 2021

Uh oh!

adin234 commented Jan 5, 2021

Uh oh!

adin234 commented Jan 6, 2021

Uh oh!

adin234 commented Jan 6, 2021

Uh oh!

adin234 commented Jan 6, 2021

Uh oh!

adin234 commented Jan 6, 2021

Uh oh!

mzakharo commented Jan 6, 2021

Uh oh!

will702 commented Jun 6, 2021

ENVIRONMENT:

CUDNN_VERSION = '8.0.4.30'

PYDEVD_USE_FRAME_EVAL = 'NO'

LD_LIBRARY_PATH = '/usr/local/nvidia/lib:/usr/local/nvidia/lib64'

CLOUDSDK_PYTHON = 'python3'

LANG = 'en_US.UTF-8'

HOSTNAME = '341155cc34b0'

OLDPWD = '/'

CLOUDSDK_CONFIG = '/content/.config'

NVIDIA_VISIBLE_DEVICES = 'all'

DATALAB_SETTINGS_OVERRIDES = '{"kernelManagerProxyPort":6000,"kernelManagerProxyHost":"172.28.0.3","jupyterArgs":["--ip=\"172.28.0.2\""],"debugAdapterMultiplexerPath":"/usr/local/bin/dap_multiplexer"}'

ENV = '/root/.bashrc'

PAGER = 'cat'

NCCL_VERSION = '2.7.8'

TF_FORCE_GPU_ALLOW_GROWTH = 'true'

JPY_PARENT_PID = '46'

NO_GCE_CHECK = 'True'

PWD = '/content/melon'

HOME = '/root'

LAST_FORCED_REBUILD = '20210528'

CLICOLOR = '1'

DEBIAN_FRONTEND = 'noninteractive'

LIBRARY_PATH = '/usr/local/cuda/lib64/stubs'

GCE_METADATA_TIMEOUT = '0'

GLIBCPP_FORCE_NEW = '1'

TBE_CREDS_ADDR = '172.28.0.1:8008'

mzakharo commented Dec 4, 2020 •

edited

Loading

adin234 commented Jan 5, 2021 •

edited

Loading

adin234 commented Jan 5, 2021 •

edited

Loading