cmd/krel/cmd/ff: Initial commit of Go-based branchff tool #869

justaugustus · 2019-09-03T14:58:33Z

Introducing krel (the Kubernetes Release toolbox), and the first draft of the branch fast-forward subcommand, ff!

Here's the directory structure:

cmd/
├── krel
│   ├── BUILD.bazel
│   ├── cmd
│   │   ├── BUILD.bazel
│   │   ├── ff.go
│   │   └── root.go
│   └── main.go

pkg/
├──  util
    ├── BUILD.bazel
    ├── common.go
    └── git.go

After mulling this over for a bit, I decided we should get at least two things out of this:

A set of common libraries to use for the tooling rewrite (pkg/util/common.go and pkg/util/git.go are the first of these)
Unified CLI tool (via cobra), to take advantage of global flags (like --cleanup and --nomock) and common patterns

This is still pretty rough, but it's at least in a good place to merge the structure in and iterate! :)

Signed-off-by: Stephen Augustus [email protected]

justaugustus · 2019-09-03T15:10:18Z

(...and yes, the finished product will have tests 😅 )

saschagrunert

Hey 👋, generally LGTM, but I have some comments. Regarding the many TODOs still left in the code I'd suggest to bring this in as soon as possible and then we can resolve them together. :)

saschagrunert · 2019-10-23T11:45:37Z

.golangci.yml

@@ -0,0 +1,32 @@
+linters-settings:


I would suggest in adding golangci-lint as build target / verification script somewhere inside the CI.

saschagrunert · 2019-10-23T11:51:05Z

pkg/util/common.go

+}
+*/
+
+func AskYesOrNo(question string) (string, error) {


I would return a bool here to make the usage more straight forward.

saschagrunert · 2019-10-23T11:52:17Z

pkg/util/git.go

@@ -0,0 +1,203 @@
+/*


We may unify this file with the one from pkg/notes/git.go

Agreed here, git related things feel like they would be worthy of their own package

+1! That was exactly what I had in mind. :)

saschagrunert · 2019-10-23T11:53:37Z

pkg/util/git.go

+
+func GetMergeBase(masterRefShort, releaseRefShort string, repo *git.Repository) (string, error) {
+	masterRef := fmt.Sprintf("origin/%s", masterRefShort)
+	releaseRef := fmt.Sprintf("origin/%s", releaseRefShort)


The default remote origin may be a const value somewhere since it's used in many places.

saschagrunert · 2019-10-23T11:55:11Z

pkg/util/git.go

+	}
+
+	if len(res) == 0 {
+		os.Exit(1)


I suggest exit'ing in the main path and return an error here instead.

saschagrunert · 2019-10-23T12:43:44Z

cmd/branchff/cmd/root.go

+		log.Println(err)
+		os.Exit(1)


Suggested change

log.Println(err)

os.Exit(1)

log.Fatal(err)

justaugustus · 2019-10-28T10:09:22Z

@saschagrunert -- sounds like a plan! I'll work on this this week.

justaugustus · 2019-10-29T11:44:41Z

/test pull-release-test

justaugustus · 2019-10-31T11:24:37Z

/assign @saschagrunert @hasheddan
/milestone v1.17

hasheddan

This is awesome @justaugustus, thanks for leading this effort :) A few comments below. Not sure at what state we should deem this mergeable, but I am fine moving forward with something that isn't perfect given we intend to iterate on this more.

hasheddan · 2019-10-31T11:45:34Z

pkg/util/common.go

+)
+
+// Run wraps the exec.Cmd.Run() command and sets the standard output.
+// TODO: Should this take an error code argument/return an error code?


This a great thing to point out and I definitely think Run() should be returning error here. Otherwise, any user of this package is unable to determine whether to exit or continue as they see fit based on the outcome of calling this function. While it doesn't make a huge difference for ff, it could prove difficult to work with if any other tool consumes it.

hasheddan · 2019-10-31T11:47:32Z

pkg/util/git.go

@@ -0,0 +1,203 @@
+/*


Agreed here, git related things feel like they would be worthy of their own package

hasheddan · 2019-10-31T11:49:12Z

pkg/util/git.go

+	// We can then use every Remote functions to retrieve wanted information
+	refs, err := rem.List(&git.ListOptions{})
+	if err != nil {
+		log.Fatal(err)


Would be useful to return an informative error here.

Later: We could use something like wrapping errors with additional context to provide some sort of error stack.

hasheddan · 2019-10-31T11:51:51Z

pkg/util/common.go

+*/
+
+func Ask(question, expectedResponse string) (string, bool, error) {
+	retries := 3


How would you feel about making retries a parameter here?

hasheddan · 2019-10-31T11:53:42Z

cmd/krel/cmd/ff.go

+	}
+
+	log.Printf(
+		`Go look around in %s to make sure things look okay before pushing...


nitpick: it might be more readable to move this to a separate variable outside the main function execution

saschagrunert · 2019-10-31T12:13:11Z

cmd/krel/BUILD.bazel

@@ -0,0 +1,32 @@
+load("@io_bazel_rules_go//go:def.bzl", "go_binary", "go_library")


I’m wondering if we need bazel at all if we stick to go modules and a recent version (>1.12).

@saschagrunert -- Though I grumble on occasion about bazel, we get the advantage of leveraging the verify tests from k/repo-infra.

Updating & verifying all things is now simple:

update: ./hack/update-all.sh

verify: ./hack/verify-all.sh

See here: #889, #890

It, being the verify, does not seem to work for me: #908

It might well be an issue on my side, could someone maybe check if that actually works for you?

saschagrunert · 2019-10-31T12:21:13Z

cmd/krel/cmd/ff.go

+		dryRunFlag = "--dry-run"
+	}
+
+	re := regexp.MustCompile(util.BranchRE)


I would add this as a stand alone function to the utils and avoid making BranchRE public, like: util.IsReleaseBranch(string)

saschagrunert · 2019-10-31T12:23:26Z

cmd/krel/cmd/ff.go

+		log.Fatalf("%s is not a release branch\n", branch)
+	}
+
+	branchExists, err := util.BranchExists(branch)


Can we return either a bool or an error here? Because the value of err is unhandled and it could be that we return nil below.

Still returning both, but the error is handled now.

saschagrunert · 2019-10-31T12:24:54Z

cmd/krel/cmd/ff.go

+		defer cleanupTmpDir(baseDir)
+	}
+
+	workingDir := fmt.Sprintf("%s/%s", baseDir, branch)


I would stick to filepath.Join for path manipulations.

saschagrunert · 2019-10-31T12:30:20Z

cmd/krel/cmd/ff.go

+
+	currentPath := fmt.Sprintf("%s", os.Getenv("PATH"))
+	etcdPath := fmt.Sprintf("%s/third_party/etcd:%s", gitRoot, currentPath)
+	os.Setenv("PATH", os.Getenv("PATH"))


We should set it to etcdPath here, right?

saschagrunert · 2019-10-31T12:35:16Z

cmd/krel/cmd/root.go

+)
+
+// rootCmd represents the base command when called without any subcommands
+var rootCmd = &cobra.Command{


This is more a question to which style we want to stick:

Do we want to avoid global variables and init functions by moving them into main?

@saschagrunert -- I think I like the layout for this iteration, but we can talk style more once this merges. :)

Signed-off-by: Stephen Augustus <[email protected]>

justaugustus · 2019-10-31T13:40:19Z

@hasheddan @saschagrunert -- I've addressed all of your comments, so this should be good to approve/merge now!

hasheddan

Assuming we are good with deferring tests to a future iteration (which seems fine considering we don't plan to be using this in a production setting until the tool is more robust) then LGTM!

/lgtm /approve

justaugustus · 2019-10-31T14:23:32Z

@hasheddan -- As a heads up, /lgtm and /approve need to be on separate lines as the first and only thing on the line, like so:

/lgtm
/approve

k8s-ci-robot · 2019-10-31T14:23:33Z

@justaugustus: you cannot LGTM your own PR.

In response to this:

@hasheddan -- As a heads up, /lgtm and /approve need to be on separate lines as the first and only thing on the line, like so:
/lgtm
/approve

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

justaugustus · 2019-10-31T14:24:17Z

(haha, I knew it was going to say that.)
/honk

k8s-ci-robot · 2019-10-31T14:24:18Z

@justaugustus:

In response to this:

(haha, I knew it was going to say that.)
/honk

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

hasheddan · 2019-10-31T15:42:40Z

/lgtm

k8s-ci-robot · 2019-10-31T15:42:58Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: hasheddan, justaugustus

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [justaugustus]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

hoegaarden

While I really appreciate all the work and thoughts that are going into re-writing the tools in go, I am abit concerned about the general direction:

This seems to be generally go int o a direction of a monolithic tool for all the release things. I think it would be more useful to break the release tools apart. What do I mean by that:
In short: don't implement things & "wiring" that other tools can do for us, just implement our specifics. Specifically In the case of branchff: really only implement our fast forwarding and merging business, push and pull and other stuff should be done elsewhere.

Why?

We can leverage other tools (GCB) to wire the things together
It's easier to change / adapt / ... e.g. we can just run other steps before/after the thing (branchff in this case) and the thing (branchff in this case) doesn't need to know or change

Consider this:

# "monostep" / bakcbox-ish step
- name: gcr.io/k8s-release-tools/krel
  args: [ "branchff", "--some-arg" ]

vs.

# get a thing, maybe not even git.k8s.io/kubernetes but some downstream vendor's repo
- name: gcr.io/cloud-builders/git
  dir: src/k8s
  args: [ "pull", "$_K8S_RPEO", "--branch", "$_K8S_REPO" ]
# only do the FF, nothing else 
- name: gcr.io/k8s-release-tools/branchff
  dir: src.k8s
  args: [ "$_FF_BRANCH" ]
# push it upstream, wherever that is
- name:  gcr.io/cloud-builders/git
  dir:  src/k8s
  args: [ "push", "upstream", "$_FF_BRANCH" ]

Now I want to run some tests on the branchff'ed branch (e.g. run e2e on kind or whatnot). Ideally that would be just adding a new GCB task, and not changing any code in branchff/krel/...

   [....]
# only do the FF, nothing else 
- name: gcr.io/k8s-release-tools/branchff
  dir: src.k8s
  args: [ "$_FF_BRANCH" ]
# run some tests with the branchff'ed code
- name: gcr.io/k8s-release-tools/kind-e2e
  args: [ "--k8s-src", "src/k8s" ]
# push it upstream, wherever that is -- IFF the last step succeeded
- name:  gcr.io/cloud-builders/git
  dir:  src/k8s
  args: [ "push", "upstream", "$_FF_BRANCH" ]

To summarise:

I am not saying the idea of krel is bad, I am merely questioning the direction it seems to go a bit.
I am concerned we might be on a path were we pack too much responsibility & wiring into the tool, and not leveraging enough of the other tools we are using already anyway.
I'd rather have some small sharp tools with a defined interface between each other. In the case of GCB the interface between multiple steps are directories to be consumed and produced. That's an easy enough thing to
- run/wire locally
- extract, reshuffle, extend, run single, ... steps
- move to another system somewhen (e.g. if we find we need tekton).

hoegaarden · 2019-10-31T15:25:51Z

cmd/krel/BUILD.bazel

@@ -0,0 +1,32 @@
+load("@io_bazel_rules_go//go:def.bzl", "go_binary", "go_library")


It, being the verify, does not seem to work for me: #908

It might well be an issue on my side, could someone maybe check if that actually works for you?

justaugustus · 2019-11-01T11:24:46Z

We're chatting in Slack (https://kubernetes.slack.com/archives/CJH2GBF7Y/p1572605765115500) w.r.t. @hoegaarden's review.
I'm going to drop notes later on the umbrella issue to talk more about what's in my head direction-wise.

tpepper · 2019-11-01T15:39:33Z

I think @hoegaarden 's concerns are valid. As new things are built there definitely should also be refactoring common things out of existing tools and instituting new splits in the tools and flow (as in the pull, branchff, push example). Those are more testable. It's totally conceivable one person wont see a pattern and other reviewers will, and ask for a refactor. We'll have to decide on a case by case basis if the refactor should happen before merge, be done by the original change proposer, or if it can follow and if somebody else can do it. The latter path will likely lead to more merge conflicts and rebase issues if folks aren't communicating.

k8s-ci-robot requested review from claurence and idealhack September 3, 2019 14:58

justaugustus force-pushed the bff branch 5 times, most recently from 0b14acd to ad43a85 Compare September 10, 2019 17:41

idealhack removed the request for review from claurence September 29, 2019 15:38

hasheddan mentioned this pull request Oct 18, 2019

scripts: template json safely #898

Merged

saschagrunert reviewed Oct 23, 2019

View reviewed changes

justaugustus force-pushed the bff branch from ad43a85 to 907555b Compare October 29, 2019 11:00

k8s-ci-robot removed the do-not-merge/blocked-paths Indicates that a PR should not merge because it touches files in blocked paths. label Oct 29, 2019

justaugustus force-pushed the bff branch from 907555b to 8f416a3 Compare October 31, 2019 11:08

justaugustus changed the title ~~[WIP] cmd/branchff: Initial commit of Go-based branchff tool~~ cmd/krel/cmd/ff: Initial commit of Go-based branchff tool Oct 31, 2019

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Oct 31, 2019

k8s-ci-robot assigned hasheddan and saschagrunert Oct 31, 2019

k8s-ci-robot added this to the v1.17 milestone Oct 31, 2019

justaugustus force-pushed the bff branch from 8f416a3 to 4308cdf Compare October 31, 2019 11:29

hasheddan reviewed Oct 31, 2019

View reviewed changes

saschagrunert reviewed Oct 31, 2019

View reviewed changes

cmd/krel/cmd/ff: Initial commit of Go-based branchff tool

ca87ff2

Signed-off-by: Stephen Augustus <[email protected]>

justaugustus force-pushed the bff branch from 4308cdf to ca87ff2 Compare October 31, 2019 13:31

hasheddan approved these changes Oct 31, 2019

View reviewed changes

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Oct 31, 2019

k8s-ci-robot merged commit 82366a4 into kubernetes:master Oct 31, 2019

hoegaarden reviewed Nov 1, 2019

View reviewed changes

justaugustus mentioned this pull request Nov 1, 2019

[krel] Refactor existing bash scripts into go tools #918

Closed

16 tasks

		@@ -0,0 +1,32 @@
		load("@io_bazel_rules_go//go:def.bzl", "go_binary", "go_library")

cmd/krel/cmd/ff: Initial commit of Go-based branchff tool #869

cmd/krel/cmd/ff: Initial commit of Go-based branchff tool #869

Uh oh!

Conversation

justaugustus commented Sep 3, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

justaugustus commented Sep 3, 2019

Uh oh!

saschagrunert left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

justaugustus commented Oct 28, 2019

Uh oh!

justaugustus commented Oct 29, 2019

Uh oh!

justaugustus commented Oct 31, 2019

Uh oh!

hasheddan left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

justaugustus commented Oct 31, 2019

Uh oh!

hasheddan left a comment

justaugustus commented Sep 3, 2019 •

edited

Loading