Stack metadata version 5

Stack metadata is now kept in refs/stacks/<branch> instead of
refs/heads/<branch>.stgit. The rationale:

- StGit's metadata branch is an implementation detail that leaks to
  users whenever git branches are inspected.
- `git log --branches` and other views of the commit DAG can become
  extremely cluttered when the stack metadata branch enters the picture.
  Avoiding refs/heads solves this problem.
- There is a nice symmetry between refs/patches/<branch> and
  refs/heads/<branch>.
- Distributed use (push, pull, clone) of stack metadata is not
  significantly different with the new ref.

And the stack metadata file format is now JSON instead of the former
custom format.

- Can use off-the-shelf JSON parser and serializer from Python standard
  library. Fast, easy, and less surface area for bugs.
- Stack metadata becomes more accessible to third-party tools, scripts,
  and programming languages.
- JSON format allows for additional metadata to be associated with
  each patch and/or the entire stack. This may be useful for advanced
  features such as patch guards.
- The stack.json file remains at least as human-readable as the old meta
  file. Line-oriented diffs also remain effective.

Resolves #65.

Signed-off-by: Peter Grayson <[email protected]>

Author: jpgrayson

completion/stgit.zsh

@@ -709,7 +709,7 @@ __stg_git_describe_branch () {
 __stg_branch_stgit() {
     declare -a stg_branches
     stg_branches=(
-        ${(u)${${${(f)"$(_call_program branchrefs git for-each-ref --format='"%(refname)"' refs/heads 2>/dev/null)"}#refs/heads/}:/*.stgit/}}
+        ${${(f)"$(_call_program branchrefs git for-each-ref --format='"%(refname)"' refs/stacks 2>/dev/null)"}#refs/stacks/}
     )
     local expl
     _wanted -V branches expl "branch" compadd $stg_branches

stgit/commands/branch.py

@@ -404,7 +404,7 @@ def func(parser, options, args):
         branch_names = sorted(
             ref.replace('refs/heads/', '', 1)
             for ref in repository.refs
-            if ref.startswith('refs/heads/') and not ref.endswith('.stgit')
+            if ref.startswith('refs/heads/')
         )
 
         if branch_names:

stgit/lib/log.py

@@ -3,7 +3,7 @@
 A stack log is a Git branch. Each commit contains the complete state (metadata) of the
 stack at the moment it was written; the most recent commit has the most recent state.
 
-For a branch `foo`, the stack state log is stored in the branch `foo.stgit`.
+For a branch `foo`, the stack state log is stored in the ref `refs/stacks/foo`.
 
 Each log entry makes sure to have proper references to everything it needs to make it
 safe against garbage collection. The stack state log can even be pulled from one
@@ -89,17 +89,23 @@
 Format version 4
 ================
 
-The metadata in the `<branch>.stgit` branch is the same as format version 1.
+The metadata in `refs/heads/<branch>.stgit` branch is the same as format version 1.
 
 Format version 4 indicates that, unlike previous format versions used by older versions
 of StGit, the stack log state is *only* contained in the stack log branch and *not* as
 files in the .git/patches directory.
 
+Format version 5
+================
+
+Stack metadata resides in `refs/stacks/<branch>` where the `stack.json` metadata file
+is JSON format instead of the previous custom format.
+
 """
 
+import json
 import re
 
-from stgit import utils
 from stgit.exception import StgException
 from stgit.lib.git import BlobData, CommitData, TreeData
 from stgit.lib.stackupgrade import FORMAT_VERSION
@@ -185,68 +191,47 @@ def from_stack(cls, prev, stack):
             patches={pn: stack.patches[pn] for pn in stack.patchorder.all},
         )
 
-    @staticmethod
-    def _parse_metadata(repo, metadata):
-        """Parse a stack log metadata string."""
-        if not metadata.startswith('Version:'):
-            raise LogParseException('Malformed log metadata')
-        metadata = metadata.splitlines()
-        version_str = utils.strip_prefix('Version:', metadata.pop(0)).strip()
-        try:
-            version = int(version_str)
-        except ValueError:
-            raise LogParseException('Malformed version number: %r' % version_str)
-        if version < FORMAT_VERSION:
-            raise LogException('Log is version %d, which is too old' % version)
-        if version > FORMAT_VERSION:
-            raise LogException('Log is version %d, which is too new' % version)
-        parsed = {}
-        key = None
-        for line in metadata:
-            if line.startswith(' '):
-                assert key is not None
-                parsed[key].append(line.strip())
-            else:
-                key, val = [x.strip() for x in line.split(':', 1)]
-                if val:
-                    parsed[key] = val
-                else:
-                    parsed[key] = []
-        prev = parsed['Previous']
-        if prev == 'None':
-            prev = None
-        else:
-            prev = repo.get_commit(prev)
-        head = repo.get_commit(parsed['Head'])
-        lists = {'Applied': [], 'Unapplied': [], 'Hidden': []}
-        patches = {}
-        for lst in lists:
-            for entry in parsed[lst]:
-                pn, sha1 = [x.strip() for x in entry.split(':')]
-                lists[lst].append(pn)
-                patches[pn] = repo.get_commit(sha1)
-        return (
-            prev,
-            head,
-            lists['Applied'],
-            lists['Unapplied'],
-            lists['Hidden'],
-            patches,
-        )
-
     @classmethod
     def from_commit(cls, repo, commit):
         """Parse a (full or simplified) stack log commit."""
         try:
-            perm, meta_blob = commit.data.tree.data['meta']
+            perm, stack_json_blob = commit.data.tree.data['stack.json']
         except KeyError:
             raise LogParseException('Not a stack log')
 
-        prev, head, applied, unapplied, hidden, patches = cls._parse_metadata(
-            repo, meta_blob.data.bytes.decode('utf-8')
-        )
+        try:
+            stack_json = json.loads(stack_json_blob.data.bytes)
+        except json.JSONDecodeError as e:
+            raise LogParseException(str(e))
+
+        version = stack_json.get('version')
+
+        if version is None:
+            raise LogException('Missing stack metadata version')
+        elif version < FORMAT_VERSION:
+            raise LogException('Log is version %d, which is too old' % version)
+        elif version > FORMAT_VERSION:
+            raise LogException('Log is version %d, which is too new' % version)
+
+        patches = {
+            pn: repo.get_commit(patch_info['oid'])
+            for pn, patch_info in stack_json['patches'].items()
+        }
 
-        return cls(prev, head, applied, unapplied, hidden, patches, commit)
+        if stack_json['prev'] is None:
+            prev = None
+        else:
+            prev = repo.get_commit(stack_json['prev'])
+
+        return cls(
+            prev,
+            repo.get_commit(stack_json['head']),
+            stack_json['applied'],
+            stack_json['unapplied'],
+            stack_json['hidden'],
+            patches,
+            commit,
+        )
 
     def _parents(self, prev_state):
         """Parents this entry needs to be a descendant of all commits it refers to."""
@@ -258,23 +243,18 @@ def _parents(self, prev_state):
             xp -= set(prev_state.patches.values())
         return xp
 
-    def _metadata_blob(self, repo, prev_state):
-        lines = ['Version: %d' % FORMAT_VERSION]
-        lines.append(
-            'Previous: %s' % ('None' if prev_state is None else prev_state.commit.sha1)
+    def _stack_json_blob(self, repo, prev_state):
+        stack_json = dict(
+            version=FORMAT_VERSION,
+            prev=None if prev_state is None else prev_state.commit.sha1,
+            head=self.head.sha1,
+            applied=self.applied,
+            unapplied=self.unapplied,
+            hidden=self.hidden,
+            patches={pn: dict(oid=patch.sha1) for pn, patch in self.patches.items()},
         )
-        lines.append('Head: %s' % self.head.sha1)
-        for patch_list, title in [
-            (self.applied, 'Applied'),
-            (self.unapplied, 'Unapplied'),
-            (self.hidden, 'Hidden'),
-        ]:
-            lines.append('%s:' % title)
-            for pn in patch_list:
-                lines.append('  %s: %s' % (pn, self.patches[pn].sha1))
-        lines.append('')
-        metadata_str = '\n'.join(lines)
-        return repo.commit(BlobData(metadata_str.encode('utf-8')))
+        blob = json.dumps(stack_json, indent=2).encode('utf-8')
+        return repo.commit(BlobData(blob))
 
     def _patch_blob(self, repo, pn, commit, prev_state):
         if prev_state is not None:
@@ -308,7 +288,7 @@ def _tree(self, repo, prev_state):
         return repo.commit(
             TreeData(
                 {
-                    'meta': self._metadata_blob(repo, prev_state),
+                    'stack.json': self._stack_json_blob(repo, prev_state),
                     'patches': self._patches_tree(repo, prev_state),
                 }
             )

stgit/lib/stack.py

@@ -10,7 +10,7 @@
 
 def _stack_state_ref(stack_name):
     """Reference to stack state metadata. A.k.a. the stack's "log"."""
-    return 'refs/heads/%s.stgit' % (stack_name,)
+    return 'refs/stacks/%s' % (stack_name,)
 
 
 def _patch_ref(stack_name, patch_name):

stgit/lib/stackupgrade.py

@@ -1,3 +1,4 @@
+import json
 import os
 import shutil
 
@@ -8,7 +9,7 @@
 from stgit.run import RunException
 
 # The current StGit metadata format version.
-FORMAT_VERSION = 4
+FORMAT_VERSION = 5
 
 
 def _format_version_key(branch):
@@ -47,6 +48,16 @@ def update_to_current_format_version(repository, branch):
     older_format_key = 'branch.%s.stgitformatversion' % branch
 
     def get_meta_file_version():
+        """Get format version from the ``meta`` file in the stack log branch."""
+        new_version = get_stack_json_file_version()
+        if new_version is not None:
+            return new_version
+
+        old_version = get_old_meta_file_version()
+        if old_version is not None:
+            return old_version
+
+    def get_old_meta_file_version():
         """Get format version from the ``meta`` file in the stack log branch."""
         stack_ref = 'refs/heads/%s.stgit:meta' % branch
         try:
@@ -64,6 +75,25 @@ def get_meta_file_version():
         else:
             return None
 
+    def get_stack_json_file_version():
+        stack_ref = 'refs/stacks/%s:stack.json' % branch
+        try:
+            data = (
+                repository.run(['git', 'show', stack_ref])
+                .decoding(None)
+                .discard_stderr()
+                .raw_output()
+            )
+        except RunException:
+            return None
+
+        try:
+            stack_json = json.loads(data)
+        except json.JSONDecodeError:
+            return None
+        else:
+            return stack_json.get('version')
+
     def get_format_version():
         """Return the integer format version number.
 
@@ -243,6 +273,77 @@ def rm_ref(ref):
             pass
         out.info('Upgraded branch %s to format version %d' % (branch, 4))
 
+    # Metadata moves from refs/heads/<branch>.stgit to refs/stacks/<branch>.
+    # Also, metadata file format is JSON instead of custom format.
+    if get_format_version() == 4:
+        old_state_ref = 'refs/heads/%s.stgit' % branch
+        old_state = repository.refs.get(old_state_ref)
+        old_meta = old_state.data.tree.data['meta'][1].data.bytes
+        lines = old_meta.decode('utf-8').splitlines()
+        if not lines[0].startswith('Version: 4'):
+            raise StackException('Malformed metadata (expected version 4)')
+
+        parsed = {}
+        key = None
+        for line in lines:
+            if line.startswith(' '):
+                assert key is not None
+                parsed[key].append(line.strip())
+            else:
+                key, val = [x.strip() for x in line.split(':', 1)]
+                if val:
+                    parsed[key] = val
+                else:
+                    parsed[key] = []
+
+        head = repository.refs.get('refs/heads/%s' % branch)
+
+        new_meta = dict(
+            version=5,
+            prev=parsed['Previous'],
+            head=head.sha1,
+            applied=[],
+            unapplied=[],
+            hidden=[],
+            patches=dict(),
+        )
+
+        if parsed['Head'] != new_meta['head']:
+            raise StackException('Unexpected head mismatch')
+
+        for patch_list_name in ['Applied', 'Unapplied', 'Hidden']:
+            for entry in parsed[patch_list_name]:
+                pn, sha1 = [x.strip() for x in entry.split(':')]
+                new_patch_list_name = patch_list_name.lower()
+                new_meta[new_patch_list_name].append(pn)
+                new_meta['patches'][pn] = dict(oid=sha1)
+
+        meta_bytes = json.dumps(new_meta, indent=2).encode('utf-8')
+
+        tree = repository.commit(
+            TreeData(
+                {
+                    'stack.json': repository.commit(BlobData(meta_bytes)),
+                    'patches': old_state.data.tree.data['patches'],
+                }
+            )
+        )
+
+        repository.refs.set(
+            'refs/stacks/%s' % branch,
+            repository.commit(
+                CommitData(
+                    tree=tree,
+                    message='stack upgrade to version 5',
+                    parents=[head],
+                )
+            ),
+            'stack upgrade to v5',
+        )
+
+        repository.refs.delete(old_state_ref)
+        out.info('Upgraded branch %s to format version %d' % (branch, 5))
+
     # Make sure we're at the latest version.
     fv = get_format_version()
     if fv not in [None, FORMAT_VERSION]:

t/t1000-branch-create.sh

@@ -33,8 +33,8 @@ test_expect_success \
 
 test_expect_success \
     'Check for various bits of new branch' '
-    test_path_is_file .git/refs/heads/new &&
-    test_path_is_file .git/refs/heads/new.stgit &&
+    git show-ref --verify --quiet refs/heads/new &&
+    git show-ref --verify --quiet refs/stacks/new &&
     test_path_is_missing .git/patches/new &&
     test_path_is_missing .git/refs/patches &&
     test "$(git config --get branch.new.remote)" = "origin" &&

t/t1001-branch-rename.sh

@@ -12,14 +12,14 @@ Exercises branch renaming commands.
 
 _assert_branch_exists() {
   git config --get-regexp "branch\\.$1\\." &&
-  test_path_is_file ".git/refs/heads/$1" &&
-  test_path_is_file ".git/refs/heads/$1.stgit"
+  git show-ref --verify --quiet "refs/heads/$1" &&
+  git show-ref --verify --quiet "refs/stacks/$1"
 }
 
 _assert_branch_missing() {
   test_expect_code 1 git config --get-regexp "branch\\.$1\\." &&
-  test_path_is_missing ".git/refs/heads/$1" &&
-  test_path_is_missing ".git/refs/heads/$1.stgit"
+  ! git show-ref --verify --quiet "refs/heads/$1" &&
+  ! git show-ref --verify --quiet "refs/stacks/$1"
 }
 
 _assert_current_branch_name() {

t/t1009-gpg.sh

@@ -10,7 +10,7 @@ test_expect_success GPG \
     git config commit.gpgsign true
     git config user.signingkey ${GIT_COMMITTER_EMAIL}
     stg init &&
-    git verify-commit master.stgit
+    git verify-commit refs/stacks/master
 '
 
 test_expect_success GPG \

t/t2110-pull-stack.sh

@@ -18,7 +18,7 @@ test_expect_success 'Pull master and stack with all applied' '
     test_create_repo cloned &&
     (cd cloned &&
      git config pull.ff only &&
-     git pull -f ../upstream master:master master.stgit:master.stgit &&
+     git pull -f ../upstream master:master refs/stacks/master:refs/stacks/master &&
      [ "$(echo $(stg series --applied --noprefix))" = "patch-1 patch-2 patch-3" ]
     )
 '
@@ -28,7 +28,7 @@ test_expect_success 'Pull master and stack with unapplied patches' '
      stg pop
     ) &&
     (cd cloned &&
-     git pull -f ../upstream master:master master.stgit:master.stgit &&
+     git pull -f ../upstream master:master refs/stacks/master:refs/stacks/master &&
      [ "$(echo $(stg series --applied --noprefix))" = "patch-1 patch-2" ] &&
      [ "$(echo $(stg series --unapplied --noprefix))" = "patch-3" ]
     )