ENH: Require Lapack 3.4.0 #8670

insertinterestingnamehere · 2018-04-03T18:56:56Z

This is a reboot of #6051.

It bumps the required LAPACK version to 3.4.0 and updates the Cython BLAS/LAPACK wrappers accordingly.

insertinterestingnamehere · 2018-04-03T18:58:58Z

Thus far this just implements this for Windows and Linux. I'm putting this up now to see how the CI tests go. I haven't removed the various custom accelerate wrappers or updated the CI scripts yet, so, for the moment, failures on OSX are expected.

ilayn · 2018-04-03T20:24:49Z

Most of the errors are about missing qr_insertwhich calls ?qptr and ?rptr routines. But the rest is surprisingly happy (apart from PEP8 details) !

rgommers · 2018-04-04T03:38:46Z

Nice and compact:)

Could possibly still be included in 1.1.x if it gets merged within about one week.

perimosocordiae · 2018-04-04T13:42:04Z

scipy/linalg/_generate_pyx.py

-from LAPACK 3.1.0 to 3.6.0. It also provides some of the
+in LAPACK 3.4.0 except for ``zcgesv`` since its interface is not consistent
+from LAPACK 3.4.0 to 3.6.0. It also provides some of the
+from LAPACK 3.4.0 to 3.6.0. It also provides some of the


Duplicate line

Good catch. Thanks.

perimosocordiae · 2018-04-04T13:46:07Z

scipy/linalg/_generate_pyx.py

    return name

+def fix_line_length(subroutine):


Seems like this would be simpler if we integrated the line-wrapping into the code that generates the subroutine (fort_subroutine_wrapper). It would at least remove the need to do fiddly parsing here.

Also do we really need line wrapping? Nobody is editing this file anyways

The line wrapping is needed to get around the maximum column limitation in Fortran 77. Including it in the original code generation is a good idea though.

Okay, so after looking at this a bit more, it turns out that the xblas functions were the only things hitting this corner case, so we could just pull this logic out entirely for LAPACK 3.4.0. That said, I don't really want to deal with this breaking again when doing some future update, so I'm pushing a version that handles the max line length issue by just vertically stacking the arguments in the generated Fortran subroutines. That greatly simplifies the logic for generating the wrapper but should also avoid potential trouble with line length later.

Sounds like a better solution indeed.

perimosocordiae · 2018-04-04T13:47:19Z

scipy/linalg/_generate_pyx.py

@@ -689,7 +727,8 @@ def make_all(blas_signature_file="cython_blas_signatures.txt",

    comments = ["This file was generated by _generate_pyx.py.\n",
                "Do not edit this file directly.\n"]
-    ccomment = ''.join(['/* ' + line.rstrip() + ' */\n' for line in comments]) + '\n'
+    ccomment = ''.join(['/* ' + line.rstrip() + ' */\n'
+                        for line in comments]) + '\n'


This produces a double newline after the last element. Not sure if that's intended.

Yah, that's not needed. I'll remove it.

On second thought, looking at the generated files, the extra newline does help readability even though it's not strictly necessary, so I'll leave it there.

insertinterestingnamehere · 2018-04-04T15:27:20Z

Yah, I'm amazed that it built and linked at all on OSX. For some reason the unavailable symbols aren't detected till load time. It must only be listing the first missing symbol it finds though, since it's only showing one missing symbol but many new routines were added between 3.2.1 and 3.4.0.

insertinterestingnamehere · 2018-04-04T15:28:59Z

1.1.x is definitely my goal, though that really depends on how deep the OSX rabbithole is.

perimosocordiae · 2018-04-05T13:36:16Z

scipy/linalg/_cython_signature_generator.py

@@ -58,7 +62,7 @@ def sigs_from_dir(directory, outfile, manual_wrappers=None, exclusions=None):
        exclusions += [get_sig_name(l) for l in manual_wrappers.split('\n')]
    signatures = []
    for filename in files:
-        name = filename.split('\\')[-1][:-2]
+        name = os.path.basename(filename)[:-2]


Is the [:-2] stripping a file extension? If so, use splitext.

perimosocordiae · 2018-04-05T13:44:40Z

scipy/linalg/_generate_pyx.py

-        return name + dims[name]
+    if ('x' in name or 'y' in name):
+        return name + special_cases.get(funcname[1:], dict()).get(name, '(n)')
+    name += special_cases.get(funcname[1:], dims).get(name, dims.get(name, ''))


This is a bit hairy. How about:

special_cases = defaultdict(dict, ladiv=...)

and then in process_fortran_name:

special = special_cases[funcname[1:]] if 'x' in name or 'y' in name: suffix = special.get(name, '(n)') else: suffix = special.get(name, '') return name + suffix

perimosocordiae · 2018-04-05T13:45:07Z

scipy/linalg/_generate_pyx.py

+    included = ['zladiv', 'zdotu', 'zdotc']
+    excluded = ['chla_transtype', 'clanhf', 'slansf']
+    if (name[0] in 'cs' and name not in excluded) or name in included:
+        return "w" + name


Use single-quoted 'w' for consistency.

perimosocordiae · 2018-04-05T13:48:31Z

scipy/linalg/_generate_pyx.py

@@ -645,7 +661,8 @@ def split_signature(sig):

 def filter_lines(lines):
    lines = [line.strip() for line in lines
-                              if line != '\n' and line[0] != '#']
+                              if line not in ['\n', '\r\n']
+                              and line[0] != '#']


I'd split this into two passes:

lines = [line for line in map(str.strip, lines) if line and not line.startswith('#')]

insertinterestingnamehere · 2018-04-20T01:15:38Z

WRT the OSX CI tests, for some reason I was unable to get the build to pick up any LAPACK other than the one numpy is built with. Specifying a different BLAS in the site.cfg file didn't help and I'm not aware of any other way to do that.

Given that limitation, for the CI tests, I do have a script to download an openblas binary and then build both numpy and scipy against that, but it ended up being easier to just drop the whole thing into a miniconda install and then use the openblas linked numpy available there. Any preference in general? The miniconda route requires less actual compilation, but it does download a bit more. It would also make it easier to test against MKL if we wanted to add that.

matthew-brett · 2018-04-20T08:41:12Z

In case it is at all useful, here is how I built Scipy with OpenBLAS a while back : https://github.com/matthew-brett/scipy-wheels/tree/with-openblas-on-osx . The mechanism could be simplified a bit now because of changes in our wheel building.

I have some vague memory that you have to put the site.cfg in your home directory, maybe because it is numpy doing the processing:

https://github.com/scipy/scipy/blob/master/site.cfg.example#L7

but please correct me if I'm wrong about that.

insertinterestingnamehere · 2018-04-21T18:54:59Z

Okay, I managed to get scipy to link against something other than what was used to build numpy. I think the issue I ran up against was just a peculiarity of how I had set things up to test locally. The expected workflow works in the CI environment, but not with a numpy pulled in from the default conda channel. Either way, it's working now.

Now that it's linking against the newer BLAS, I've found that the gfortran ABI changed between gcc 6 and gcc 7, so I shouldn't have the downloaded libopenblas load libgfortran ABI version 4 since it was built with ABI version 3. Would it be preferable to use a newer binary or to install an older gcc/gfortran from homebrew for the CI tests?

ilayn · 2018-04-21T19:33:58Z

OSX build didn't like the solution apparently.

insertinterestingnamehere · 2018-04-21T20:22:19Z

Sorry I wasn't clear. Currently it succeeds in building against the new library, but the test runs fail because it is looking for a different version of libgfortran than is provided by the latest homebrew. I could just use install_name_tool to change libopenblas to use the newer version, but the reason the gfortran devs changed the shared object version number was because of ABI incompatibilities in libgfortran between gfortran 7 and previous versions. To get around this, I need to either pin the CI tests to use an older gfortran so the older version of libgfortran is available or I can use a binary of OpenBLAS that was built with the newer gfortran.

So, does anyone have a binary of OpenBLAS built with gcc7 handy? If so, great. Either way, I'll look in to using an older gfortran from homebrew for the time being.

introduced after LAPACK 3.2.1. Functions introduced after that version that would otherwise need ABI wrappers for clapack ABI compatibility are not supported by Accelerate at all. That's the only place where the clapack abi is used, so ABI wrappers for these routines are not needed.

For detection of BLAS/LAPACK, use our own copies lapack_opt_info and blas_opt_info which ignore OSX Accelerate.

with Accelerate on OSX.

iterative solver code.

LAPACK f2py wrappers.

BLAS/LAPACK wrappers.

BLAS/LAPACK API.

no longer supported.

The top level setup.py file now checks that a BLAS/LAPACK installation has been found, so it is no longer necessary to check for that when linking against BLAS and LAPACK in the lower level setup.py files.

notes.

no longer supported.

insertinterestingnamehere · 2018-06-01T03:54:05Z

To what extent do we want to discuss the site.cfg machinery in the docs here? Should that be included at all in the top-level INSTALL file? What about the OSX-specific page in the docs? The Windows page does not mention it, but the Linux one does. It's not really an OS-specific detail though.

rgommers · 2018-06-01T04:22:35Z

doc/source/building/macosx.rst

+------------------------
+
+You will also need to install a library providing the BLAS and LAPACK
+interfaces. ATLAS, OpenBLAS, and MKL all work. OpenBLAS can be installed


How about changing the order to "OpenBLAS, MKL and ATLAS"? ATLAS is the worst of the three, and the most difficult to obtain.

Another sentence to add is something like "if you're using numpy from the Anaconda default channel, it will use MKL, and SciPy will then be built against that same MKL install".

matthew-brett · 2018-06-01T21:02:38Z

While I'm here - I wanted to point to https://github.com/MacPython/numpy-wheels/blob/master/on_openblas.rst - it describes the Travis-CI / Appveyor build machinery for OpenBLAS, which is a bit disorganized at the moment.

ilayn · 2018-06-01T21:46:01Z

I think build instructions can be developed independent from this PR. As we have done with Building on Windows we can make another for OSX environment separately to be more focused on building.

insertinterestingnamehere · 2018-06-01T21:56:17Z

Yah, I'd rather not let documentation concerns hold up getting the other changes in to master. The longer the non-doc changes are in master, the more time we have to see if they cause trouble for anyone.

Should we merge the doc edits here as is? I can remove them if needed.

matthew-brett · 2018-06-01T22:00:14Z

Sorry, my link was just an FYI, not a suggestion for the documentation.

pv

I don't have further comments, looks good to go from my side. I agree polishing the build instructions should be done separately.

No need to remove the documentation changes made here, though, I think.

rgommers · 2018-06-01T22:35:55Z

LGTM too - let's leave doc changes and possibly other little tweaks to future PRs. Merging this so people will start trying it out.

Thanks a lot @insertinterestingnamehere, all!

insertinterestingnamehere mentioned this pull request Apr 3, 2018

MAINT: Increase minimum required LAPACK version to 3.3.1 #6051

Closed

rgommers added enhancement A new feature or improvement scipy.linalg labels Apr 4, 2018

perimosocordiae reviewed Apr 4, 2018

View reviewed changes

insertinterestingnamehere force-pushed the lapack34 branch from a6eb082 to bec027e Compare April 4, 2018 19:27

perimosocordiae reviewed Apr 5, 2018

View reviewed changes

pv added this to the 1.2.0 milestone Apr 12, 2018

insertinterestingnamehere force-pushed the lapack34 branch 7 times, most recently from 242845a to 4bf8f1b Compare April 19, 2018 20:42

insertinterestingnamehere force-pushed the lapack34 branch 5 times, most recently from 6b4acb7 to 33d11f2 Compare April 21, 2018 17:33

insertinterestingnamehere force-pushed the lapack34 branch from 33d11f2 to 7e4d516 Compare April 21, 2018 20:28

insertinterestingnamehere and others added 15 commits May 31, 2018 16:46

MAINT: Update OSX CI tests to download and use OpenBLAS.

5b5e587

MAINT: Raise an error when a user attempts to build with Accelerate.

56557bd

MAINT: Remove accelerate ABI patches.

582250a

BLD: Drop accelerate support on numpy.distutils.system_info level

7155aa8

For detection of BLAS/LAPACK, use our own copies lapack_opt_info and blas_opt_info which ignore OSX Accelerate.

BLD: Provide a more explicit error for users that attempt to build

e77e68f

with Accelerate on OSX.

MAINT: No longer use the dummy accelerate ABI wrappers in ARPACK.

dc535a9

MAINT: No longer use the dummy accelerate ABI wrappers in sparse

d33ed28

iterative solver code.

MAINT: No longer use the dummy accelerate ABI wrappers in the BLAS and

67e2447

LAPACK f2py wrappers.

MAINT: No longer use the dummy accelerate ABI wrappers in the cython

0a58d45

BLAS/LAPACK wrappers.

MAINT: Remove dummy Accelerate ABI wrappers.

4c7a62f

STY: Minor stylistic improvements in generation scrips for Cython

eafeaef

BLAS/LAPACK API.

DOC: Update OSX Installation instructions to reflect that Accelerate is

fb0cdd2

no longer supported.

BLD: Remove redundant error raising code for if no LAPACK/BLAS is found.

57ff063

The top level setup.py file now checks that a BLAS/LAPACK installation has been found, so it is no longer necessary to check for that when linking against BLAS and LAPACK in the lower level setup.py files.

DOC: Mention the increase in the required LAPACK version in the release

80fe62a

notes.

insertinterestingnamehere force-pushed the lapack34 branch from 85f7f74 to 80fe62a Compare May 31, 2018 21:53

DOC: Explicitly mention in OSX install instructions that Accelerate is

d6d6b8b

no longer supported.

insertinterestingnamehere force-pushed the lapack34 branch from 3254b16 to d6d6b8b Compare June 1, 2018 03:18

rgommers reviewed Jun 1, 2018

View reviewed changes

pv approved these changes Jun 1, 2018

View reviewed changes

rgommers merged commit 14de485 into scipy:master Jun 1, 2018

insertinterestingnamehere mentioned this pull request Jun 20, 2018

LAPACK Rectangular Full Packed routines #8212

Closed

larsoner mentioned this pull request Aug 5, 2019

API: Expose _compute_lwork to facilitate low-level LAPACK use #10575

Open

mdhaber mentioned this pull request Sep 29, 2019

ENH: Roadmap Item: Support for newer LAPACK features #10881

Closed

Uh oh!

ENH: Require Lapack 3.4.0 #8670

ENH: Require Lapack 3.4.0 #8670

Uh oh!

Conversation

insertinterestingnamehere commented Apr 3, 2018

Uh oh!

insertinterestingnamehere commented Apr 3, 2018

Uh oh!

ilayn commented Apr 3, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rgommers commented Apr 4, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

insertinterestingnamehere commented Apr 4, 2018

Uh oh!

insertinterestingnamehere commented Apr 4, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

insertinterestingnamehere commented Apr 20, 2018

Uh oh!

matthew-brett commented Apr 20, 2018

Uh oh!

insertinterestingnamehere commented Apr 21, 2018

Uh oh!

ilayn commented Apr 21, 2018

Uh oh!

insertinterestingnamehere commented Apr 21, 2018

Uh oh!

insertinterestingnamehere commented Jun 1, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

matthew-brett commented Jun 1, 2018

Uh oh!

ilayn commented Jun 1, 2018

Uh oh!

insertinterestingnamehere commented Jun 1, 2018

Uh oh!

matthew-brett commented Jun 1, 2018

Uh oh!

pv left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rgommers commented Jun 1, 2018

Uh oh!

Uh oh!

ilayn commented Apr 3, 2018 •

edited

Loading

pv left a comment •

edited

Loading