Metaticket: revamp workflow / switch to git

[kini]

We plan to revamp the development workflow for Sage, involving a few separate (big) changes:

• switch to git 
• consolidate our repositories into a single repository
• simplified development interface

We break down the steps needed for this transition in the following tasks/tickets:

• switch sage to the new directory layout #14480 - switch to new directory layout and update build system
• document the new workflow #14481 - write a new developer manual; see https://github.com/sagemath/git-developer-guide
• create the new development scripts/library #14482 - write development scripts
• Make sure the patchbot works with the new layout (this seems to be okay).
• Make sure the buildbot works with the new layout (VB: works now)
• make sure all optional packages work in the git repository #14962 - Make sure optional spkgs work with the new layout.
• Add editor- and os-specific temporary files to .gitignore #15241 - ignore various temporary files
• "make download" target for sage #15222 - make download to download all tarballs
• Sage should re-build packages when versions change. #15221 - always rebuild packages if the version number changes
• Build git without gettext if necessary, spkg-check sanity #15401 - Build git without gettext if necessary
• Borken git doctests. #15420 - Borken git doctests.
• Enable cython caching #15430 - Enable cython caching
• Cksum uses tabs instead of spaces, breaking sage-spkg's regex #15436 - Cksum uses tabs instead of spaces, breaking sage-spkg's regex
• "make distclean" fails on Solaris #15458 - "make distclean" fails on Solaris
• Allow using Sage without dev/git #15465 - Allow using Sage without git repository
• update to latest upstream git #15468 - update to latest upstream git
• Symmetrica spkg-check fails due to whitespace #15469 - Symmetrica spkg-check fails due to whitespace
• Fixes for the source tarball generation (sage -sdist) #15472 - Fixes for the source tarball generation (sage -sdist)
• Store actual tarball name in pkg metadata. #14968 - Store actual tarball name in pkg metadata.
• Run sage-upgrade *outside* of the Sage shell #15517 - Run sage-upgrade outside of the Sage shell

Instructions for using the new workflow: QuickStartSageGit

Depends on #15517
Depends on #14968
Depends on #15472
Depends on #15468
Depends on #15469
Depends on #15465
Depends on #15458
Depends on #15436
Depends on #15430
Depends on #15420
Depends on #15401
Depends on #14480
Depends on #14481
Depends on #14482
Depends on #14962
Depends on #15241
Depends on #15222
Depends on #15221

CC: @koffie @jdemeyer @roed314 @ohanar @robertwb @burcin @ppurka @saraedum @nexttime @dimpase

Component: misc

Keywords: git workflow, sd51

Branch/Commit: u/hivert/master @ 68680e9

Issue created by migration from https://trac.sagemath.org/ticket/13015

[kini]

Description changed:

--- 
+++ 
@@ -5,6 +5,12 @@
 - simplified development interface
 - etc. (many other ideas)
 
+Help develop scripts and other things related to this project at http://github.com/kini/sage-workflow . We are not making subtickets because this is such a big change that it won't really make sense to get most of this stuff into Sage until after the switchover to git and a single repo, and also because collaboration is faster this way.
+
+Related tickets:
+
+* #13025 and its subtickets, to clean up some SPKGs
+* #13029 and #13031, to use new Cython features
+* #13032, to use ccache
+
 This ticket description needs to be expanded...
-
-Help develop scripts and other things related to this project at http://github.com/kini/sage-workflow . We are not making subtickets because this is such a big change that it won't really make sense to get most of this stuff into Sage until after the switchover to git and a single repo, and also because collaboration is faster this way.

[ohanar]

Dependencies: #9906, #11080

[jdemeyer]

Changed dependencies from #9906, #11080 to #12938, #9906, #11080

[ohanar]

comment:8

Those four spkgs are treated differently in the git layout, so the current (or future) spkg-install scripts are irrelevant.

[ohanar]

Changed dependencies from #12938, #9906, #11080 to #9906, #11080

[ohanar]

Changed dependencies from #9906, #11080 to #9906, #11080, #13123

[ohanar]

Changed dependencies from #9906, #11080, #13123 to #9906, #11080, #13031, #13032, #13123, #13432

[ohanar]

Description changed:

--- 
+++ 
@@ -10,7 +10,6 @@
 Related tickets:
 
 * #13025 and its subtickets, to clean up some SPKGs
-* #13029 and #13031, to use new Cython features
-* #13032, to use ccache
+* #13190 and #13201, switch sage library to setuptools and allow for parallel easy_installs
 
 This ticket description needs to be expanded...

[ohanar]

Changed dependencies from #9906, #11080, #13031, #13032, #13123, #13432 to #9906, #11080, #13031, #13032, #13123, #13432, #13448

[ohanar]

Description changed:

--- 
+++ 
@@ -1,11 +1,13 @@
 We plan to revamp the development workflow for Sage, involving a few separate (big) changes:
 
-- switch to git (switch to using commits and a history graph for collaboration rather than patch files)
-- consolidate our repositories into a single repository
-- simplified development interface
-- etc. (many other ideas)
+* switch to git 
+  * switch to using commits and a history graph for collaboration rather than patch files
+* consolidate our repositories into a single repository
+  * main 4 repos for now, spkgs can be merged in one by one after we transition
+* simplified development interface
+* etc.
 
-Help develop scripts and other things related to this project at http://github.com/kini/sage-workflow . We are not making subtickets because this is such a big change that it won't really make sense to get most of this stuff into Sage until after the switchover to git and a single repo, and also because collaboration is faster this way.
+Help develop scripts and other things related to this project at http://github.com/kini/sage-workflow . A significant portion of the work to transition to git can be done before we actually carry it out (without affecting how sage presently functions). For these portions, we are making tickets and adding them as dependencies.
 
 Related tickets:
 

[robertwb]

Changed dependencies from #9906, #11080, #13031, #13032, #13123, #13432, #13448 to #9906, #11080, #13031, #13032, #13123, #13432, #13448, #14372

[ohanar]

Changed dependencies from #9906, #11080, #13031, #13032, #13123, #13432, #13448, #14372 to #9906, #11080, #13031, #13032, #13123, #13432, #13448, #14372, #14226, #14316, #14331