Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions .github/workflows/test-build-deploy.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ jobs:
lint:
runs-on: ubuntu-latest
container:
image: quay.io/cortexproject/build-image:2021w14_go1.16-afe06f021
image: quay.io/cortexproject/build-image:build-image-multiarch-1d2497ff6
steps:
- name: Checkout Repo
uses: actions/checkout@v2
Expand All @@ -34,7 +34,7 @@ jobs:
test:
runs-on: ubuntu-latest
container:
image: quay.io/cortexproject/build-image:2021w14_go1.16-afe06f021
image: quay.io/cortexproject/build-image:build-image-multiarch-1d2497ff6
services:
cassandra:
image: cassandra:3.11
Expand All @@ -55,7 +55,7 @@ jobs:
build:
runs-on: ubuntu-latest
container:
image: quay.io/cortexproject/build-image:2021w14_go1.16-afe06f021
image: quay.io/cortexproject/build-image:build-image-multiarch-1d2497ff6
steps:
- name: Checkout Repo
uses: actions/checkout@v2
Expand Down Expand Up @@ -173,14 +173,14 @@ jobs:
run: |
touch build-image/.uptodate
MIGRATIONS_DIR=$(pwd)/cmd/cortex/migrations
make BUILD_IMAGE=quay.io/cortexproject/build-image:2021w14_go1.16-afe06f021 TTY='' configs-integration-test
make BUILD_IMAGE=quay.io/cortexproject/build-image:build-image-multiarch-1d2497ff6 TTY='' configs-integration-test

deploy_website:
needs: [build, test]
if: (github.ref == 'refs/heads/master' || startsWith(github.ref, 'refs/tags/')) && github.repository == 'cortexproject/cortex'
runs-on: ubuntu-latest
container:
image: quay.io/cortexproject/build-image:2021w14_go1.16-afe06f021
image: quay.io/cortexproject/build-image:build-image-multiarch-1d2497ff6
steps:
- name: Checkout Repo
uses: actions/checkout@v2
Expand Down Expand Up @@ -217,7 +217,7 @@ jobs:
if: (github.ref == 'refs/heads/master' || startsWith(github.ref, 'refs/tags/')) && github.repository == 'cortexproject/cortex'
runs-on: ubuntu-latest
container:
image: quay.io/cortexproject/build-image:2021w14_go1.16-afe06f021
image: quay.io/cortexproject/build-image:build-image-multiarch-1d2497ff6
steps:
- name: Checkout Repo
uses: actions/checkout@v2
Expand Down
13 changes: 12 additions & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
# WARNING: do not commit to a repository!
-include Makefile.local

.PHONY: all test cover clean images protos exes dist doc clean-doc check-doc
.PHONY: all test cover clean images protos exes dist doc clean-doc check-doc push-multiarch-build-image
.DEFAULT_GOAL := all

# Version number
Expand Down Expand Up @@ -40,8 +40,19 @@ SED ?= $(shell which gsed 2>/dev/null || which sed)
@echo
$(SUDO) docker build --build-arg=revision=$(GIT_REVISION) --build-arg=goproxyValue=$(GOPROXY_VALUE) -t $(IMAGE_PREFIX)$(shell basename $(@D)) $(@D)/
$(SUDO) docker tag $(IMAGE_PREFIX)$(shell basename $(@D)) $(IMAGE_PREFIX)$(shell basename $(@D)):$(IMAGE_TAG)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why was this removed? I think we need it to keep CI working, am I wrong?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's back now... in single docker build command, with multiple -t flags.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reverted it back, due to upcoming change in #4211, which builds images for both platforms, but only tags native one with $(IMAGE_PREFIX)$(shell basename $(@D)).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had to revert that change in #4211, as Github Actions don't support --platform argument when running docker :(

@echo
@echo Please use push-multiarch-build-image to build and push build image for all supported architectures.
touch $@

push-multiarch-build-image:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Who is expected to run this?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This target is to be used by maintainer building a new build-image. I've updated how-to-update-the-build-image.md file to make it clear. Note that it should work regardless of platform that dev is using.

@echo
# Build image for each platform separately... it tends to generate fewer errors.
$(SUDO) docker buildx build --platform linux/amd64 --build-arg=revision=$(GIT_REVISION) --build-arg=goproxyValue=$(GOPROXY_VALUE) build-image/
$(SUDO) docker buildx build --platform linux/arm64 --build-arg=revision=$(GIT_REVISION) --build-arg=goproxyValue=$(GOPROXY_VALUE) build-image/
# This command will run the same build as above, but it will reuse existing platform-specific images,
# put them together and push to registry.
$(SUDO) docker buildx build -o type=registry --platform linux/amd64,linux/arm64 --build-arg=revision=$(GIT_REVISION) --build-arg=goproxyValue=$(GOPROXY_VALUE) -t $(IMAGE_PREFIX)build-image:$(IMAGE_TAG) build-image/

# We don't want find to scan inside a bunch of directories, to accelerate the
# 'make: Entering directory '/go/src/github.com/cortexproject/cortex' phase.
DONT_FIND := -name vendor -prune -o -name .git -prune -o -name .cache -prune -o -name .pkg -prune -o -name packaging -prune -o
Expand Down
45 changes: 23 additions & 22 deletions build-image/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -13,42 +13,43 @@ RUN npm install -g [email protected] [email protected]
ENV HUGO_VERSION=v0.72.0
RUN git clone https://github.com/gohugoio/hugo.git --branch ${HUGO_VERSION} --depth 1 && \
cd hugo && go install --tags extended && cd ../ && \
rm -rf hugo/ && rm -rf /go/pkg /go/src
ENV SHFMT_VERSION=3.1.0
RUN curl -fsSLo shfmt https://github.com/mvdan/sh/releases/download/v${SHFMT_VERSION}/shfmt_v${SHFMT_VERSION}_linux_amd64 && \
echo "cb91ea08a075a2f96b5230f09b4e211b7c108b1c97603caceb48d117d2ac5508 shfmt" | sha256sum -c && \
rm -rf hugo/ && rm -rf /go/pkg /go/src /root/.cache

ENV SHFMT_VERSION=3.2.4
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bumped version, as 3.1.0 had no arm64 binary.

RUN GOARCH=$(go env GOARCH) && \
if [ "$GOARCH" = "amd64" ]; then \
DIGEST=3f5a47f8fec27fae3e06d611559a2063f5d27e4b9501171dde9959b8c60a3538; \
elif [ "$GOARCH" = "arm64" ]; then \
DIGEST=6474d9cc08a1c9fe2ef4be7a004951998e3067d46cf55a011ddd5ff7bfab3de6; \
fi && \
URL=https://github.com/mvdan/sh/releases/download/v${SHFMT_VERSION}/shfmt_v${SHFMT_VERSION}_linux_${GOARCH}; \
curl -fsSLo shfmt "${URL}" && \
echo "$DIGEST shfmt" | sha256sum -c && \
chmod +x shfmt && \
mv shfmt /usr/bin

RUN curl -sfL https://github.com/raw/golangci/golangci-lint/master/install.sh| sh -s -- -b /usr/bin v1.27.0
RUN GO111MODULE=on go get -tags netgo \
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the impact of removing -tags netgo?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

According to https://golang.org/doc/go1.5, netgo is no longer necessary.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In practice, I haven't noticed any difference.

Copy link
Contributor Author

@pstibrany pstibrany May 21, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another nice test to do could be pushing the image and updating it in CI workflows, so we can test CI running on the new build-image.

I've updated this PR to use newly built image from this PR.


RUN GO111MODULE=on go get \
github.com/client9/misspell/cmd/[email protected] \
github.com/golang/protobuf/[email protected] \
github.com/gogo/protobuf/[email protected] \
github.com/gogo/protobuf/[email protected] \
github.com/weaveworks/tools/cover@bdd647e92546027e12cdde3ae0714bb495e43013 && \
rm -rf /go/pkg /go/src
github.com/weaveworks/tools/cover@bdd647e92546027e12cdde3ae0714bb495e43013 \
github.com/fatih/[email protected] \
github.com/campoy/[email protected] \
&& rm -rf /go/pkg /go/src /root/.cache

ENV KUBEVAL_VERSION=0.15.0
RUN curl -Ls https://github.com/instrumenta/kubeval/releases/download/${KUBEVAL_VERSION}/kubeval-linux-amd64.tar.gz -o /tmp/kubeval-linux-amd64.tar.gz && \
tar -xf /tmp/kubeval-linux-amd64.tar.gz -C /tmp && \
cp /tmp/kubeval /usr/local/bin && \
rm -f /tmp/kubeval*
# Cannot get it to run together in above go get.
RUN GO111MODULE=on go get \
github.com/instrumenta/[email protected] \
&& rm -rf /go/pkg /go/src /root/.cache

ENV NODE_PATH=/usr/lib/node_modules
COPY build.sh /
ENV GOCACHE=/go/cache
ENTRYPOINT ["/build.sh"]

# Install faillint used to lint go imports in CI.
ENV FAILLINT_VERSION=1.5.0
RUN GO111MODULE=on go get github.com/fatih/faillint@v${FAILLINT_VERSION} && \
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved faillint to shared go get build.

rm -rf /go/pkg /go/src

# Install embedmd used to embed content into markdown files.
ENV EMBEDMD_VERSION=1.0.0
RUN GO111MODULE=on go get github.com/campoy/embedmd@v${EMBEDMD_VERSION} && \
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved embedmd to shared go get build.

rm -rf /go/pkg /go/src

ARG revision
LABEL org.opencontainers.image.title="build-image" \
org.opencontainers.image.source="https://github.com/cortexproject/cortex/tree/master/build-image" \
Expand Down
9 changes: 4 additions & 5 deletions docs/contributing/how-to-update-the-build-image.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,7 @@ slug: how-to-update-the-build-image

The build image currently can only be updated by a Cortex maintainer. If you're not a maintainer you can still open a PR with the changes, asking a maintainer to assist you publishing the updated image. The procedure is:

1. Update `build-image/Docker`
2. Build the image running `make build-image/.uptodate`
3. Publish the image to the repository running `docker push quay.io/cortexproject/build-image:TAG` (this can only be done by a maintainer)
4. Replace the image tag in `.github/workflows/*` (_there may be multiple references_)
5. Open a PR and make sure the CI with new build-image passes
1. Update `build-image/Docker`.
2. Build the and publish the image by using `make push-multiarch-build-image`. This will build and push multiplatform docker image (for linux/amd64 and linux/arm64). Pushing to `quay.io/cortexproject/build-image` repository can only be done by a maintainer. Running this step successfully requires [Docker Buildx](https://docs.docker.com/buildx/working-with-buildx/), but does not require a specific platform.
3. Replace the image tag in `.github/workflows/*` (_there may be multiple references_)
4. Open a PR and make sure the CI with new build-image passes