switch sage to the new directory layout

[ohanar]

To switch to a unified repository we need to include some information that are currently in the spkg tarballs, which requires modifying the sage's directory structure. Since we are switching workflows at the same time, it is an ideal time to rethink the directory structure as a whole.

The proposed significant changes

spkg -> build
devel/sage -> src
devel/ext -> src/ext
sage_scripts -> src/bin
spkg/bin -> src/bin

devel/sagenb -> /dev/null (is now considered a typical upstream python library)
/dev/null -> build/pkgs/* (these are now the spkg repositories)

Full changes are described at http://wiki.sagemath.org/WorkflowSEP.

The script used to create the git repository is consolidate-repos.sh, which depends on configuration.sh, git-filter-branch, and the fast-export directory (all located at https://github.com/sagemath/sage-workflow). They require bash4, git 1.8.x, and python 2.7 to run (at the very least, possibly more).

Depends on #13015
Depends on #14781

CC: @jdemeyer @saraedum

Component: distribution

Author: R. Andrew Ohana

Branch/Commit: u/ohanar/build_system @ cd237e7

Reviewer: Volker Braun

Issue created by migration from https://trac.sagemath.org/ticket/14480

[ohanar]

comment:1

This can really only be reviewed on the new trac instance. For now, the branch field on http://trac.tangentspace.org/ticket/13015 links to the list of changes (to actually see the changes click the view changes button -- since this includes some massive changes, it might take awhile to load the side-by-side diff).

[jdemeyer]

comment:5

Where is the script that you used to automatically create the git repo from the hg repos? That should be reviewed too and kept for reference.

[ohanar]

Description changed:

--- 
+++ 
@@ -14,3 +14,6 @@
 ```
 
 Full changes are described at http://wiki.sagemath.org/WorkflowSEP.
+
+The script used to create the git repository is `consolidate-repos.sh`, which depends on `configuration.sh`, `git-filter-branch`, and the `fast-export` directory (all located at https://github.com/sagemath/sage-workflow). They require bash4, git 1.8.x, and python 2.7 to run (at the very least, possibly more).
+

[ohanar]

Changed dependencies from #13015 to #13015, #14781

[ohanar]

Branch: u/ohanar/build_system

[mezzarobba]

comment:9

A few small issues with this branch. I am not sure which of these should actually be fixed here, so I'm marking the ticket as need_info rather that need_work.

• There remain references to src/sage/misc/hg.py (which was removed as part of 48ff645) in src/sage/doctest/control.py.

sage -t --long src/sage/doctest/control.py
**********************************************************************
File "src/sage/doctest/control.py", line 434, in sage.doctest.control.DocTestController.add_files
Failed example:
    DC.add_files()
Exception raised:
    Traceback (most recent call last):
      File "/home/marc/co/sage/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 486, in _run
        self.execute(example, compiled, test.globs)
      File "/home/marc/co/sage/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 845, in execute
        exec compiled in globs
      File "<doctest sage.doctest.control.DocTestController.add_files[10]>", line 1, in <module>
        DC.add_files()
      File "/home/marc/co/sage/local/lib/python2.7/site-packages/sage/doctest/control.py", line 458, in add_files
        from sage.misc.hg import hg_sage
    ImportError: No module named hg
**********************************************************************
1 item had failures:
   1 of  16 in sage.doctest.control.DocTestController.add_files
    [162 tests, 1 failure, 1.97 s]

• I also get a few more doctest failures such as the one below due to a missing import that seems to be fixed in the current git version of sagenb.

**********************************************************************
File "local/lib/python2.7/site-packages/sagenb-0.10.4-py2.7.egg/sagenb/misc/misc.py", line 44, in sagenb.misc.misc.print_open_msg
Failed example:
    sage.server.misc.print_open_msg('localhost', 8080, True)
Exception raised:
    Traceback (most recent call last):
      File "/home/marc/co/sage/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 486, in _run
        self.execute(example, compiled, test.globs)
      File "/home/marc/co/sage/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 845, in execute
        exec compiled in globs
      File "<doctest sagenb.misc.misc.print_open_msg[0]>", line 1, in <module>
        sage.server.misc.print_open_msg('localhost', Integer(8080), True)
    AttributeError: 'module' object has no attribute 'misc'
**********************************************************************

• Finally, trying to run the notebook produces the following error:

sage: notebook()
The notebook files are stored in: sage_notebook.sagenb
**************************************************
*                                                *
* Open your web browser to http://localhost:8080 *
*                                                *
**************************************************
Executing twistd  --pidfile="sage_notebook.sagenb/sagenb.pid" -ny "sage_notebook.sagenb/twistedconf.tac"
Unhandled Error
Traceback (most recent call last):
  File "/home/marc/co/sage/local/lib/python2.7/site-packages/Twisted-12.3.0-py2.7-linux-x86_64.egg/twisted/application/app.py", line 652, in run
    runApp(config)
  File "/home/marc/co/sage/local/lib/python2.7/site-packages/Twisted-12.3.0-py2.7-linux-x86_64.egg/twisted/scripts/twistd.py", line 23, in runApp
    _SomeApplicationRunner(config).run()
  File "/home/marc/co/sage/local/lib/python2.7/site-packages/Twisted-12.3.0-py2.7-linux-x86_64.egg/twisted/application/app.py", line 386, in run
    self.application = self.createOrGetApplication()
  File "/home/marc/co/sage/local/lib/python2.7/site-packages/Twisted-12.3.0-py2.7-linux-x86_64.egg/twisted/application/app.py", line 451, in createOrGetApplication
    application = getApplication(self.config, passphrase)
--- <exception caught here> ---
  File "/home/marc/co/sage/local/lib/python2.7/site-packages/Twisted-12.3.0-py2.7-linux-x86_64.egg/twisted/application/app.py", line 462, in getApplication
    application = service.loadApplication(filename, style, passphrase)
  File "/home/marc/co/sage/local/lib/python2.7/site-packages/Twisted-12.3.0-py2.7-linux-x86_64.egg/twisted/application/service.py", line 405, in loadApplication
    application = sob.loadValueFromFile(filename, 'application', passphrase)
  File "/home/marc/co/sage/local/lib/python2.7/site-packages/Twisted-12.3.0-py2.7-linux-x86_64.egg/twisted/persisted/sob.py", line 210, in loadValueFromFile
    exec fileObj in d, d
  File "sage_notebook.sagenb/twistedconf.tac", line 15, in <module>
    import base as flask_base
exceptions.ImportError: No module named base

Failed to load application: No module named base

---------------------------------------------------------------------------
error                                     Traceback (most recent call last)
<ipython-input-1-3728cb3d7c7d> in <module>()
----> 1 notebook()

/home/marc/co/sage/local/lib/python2.7/site-packages/sagenb-0.10.4-py2.7.egg/sagenb/notebook/notebook_object.pyc in __call__(self, *args, **kwds)
    221     """
    222     def __call__(self, *args, **kwds):
--> 223         return self.notebook(*args, **kwds)
    224 
    225     notebook = run_notebook.notebook_run

/home/marc/co/sage/local/lib/python2.7/site-packages/sagenb-0.10.4-py2.7.egg/sagenb/notebook/run_notebook.pyc in notebook_run(self, directory, port, interface, port_tries, secure, reset, accounts, openid, server_pool, ulimit, timeout, upload, automatic_login, start_path, fork, quiet, server, profile, subnets, require_login, open_viewer, address)
    626     os.chdir(cwd)
    627     if e == 256:
--> 628         raise socket.error
    629 
    630 def get_admin_passwd():

error: