Skip to content

Commit b4d96bf

Browse files
atc0005atc0005
committed
Refactor binary generation options
The previous release generated Windows binaries that were broken (GH-53) and Linux binaries that were unintentionally dynamic (GH-48). This commit provides an updated Makefile and GitHub workflow which provides multiple build options for both dynamic and static linking, but defaults to the previous dynamically linked behavior. The Makefile build options now explicitly enable the `CGO_ENABLED` environment variable so that all builds have the required cgo functionality enabled. A `docker` Makefile recipe is provided to generate binaries using new Docker images from the `atc0005/go-ci` project based on the official Golang Alpine Linux image. The result is statically linked binaries based on the musl C library instead of glibc. The intent is to help prevent licensing issues surround the GNU C library's LGPL licensing (which I do not fully understand). Multiple build tags are specified for static builds which enable Go-specific replacements for common glibc-provided features: - `osusergo` - `netgo` and a build tag specific to disabling SQLite extensions, which we do not use with this specific project: - `sqlite_omit_load_extension` Minor documentation updates have been included which update the build requirements and specific steps for building binaries for this project. Further updates are likely needed to add polish. A Docker Compose file has been included for kicking off multiple static binary builds in parallel, but it may end up getting tossed in a later PR if we don't make sufficient use of it. - refs GH-48 - refs golang/go 38789 - refs golang/go 26492 - refs atc0005/go-ci#85
1 parent 9642c01 commit b4d96bf

File tree

6 files changed

+326
-49
lines changed

6 files changed

+326
-49
lines changed

‎.github/workflows/lint-and-build-using-make.yml

+74-4
Original file line numberDiff line numberDiff line change
@@ -52,8 +52,10 @@ jobs:
5252
- name: Run Go linting tools using project Makefile
5353
run: make linting
5454

55-
build_code_with_makefile:
56-
name: Build codebase using Makefile
55+
# This is run from *within* a container that is itself within the GitHub
56+
# Actions environment. All of these commands are run within our container.
57+
build_dynamic_binaries_with_makefile:
58+
name: Build dynamically linked binaries using Makefile
5759
runs-on: ubuntu-latest
5860
# Default: 360 minutes
5961
timeout-minutes: 10
@@ -68,8 +70,76 @@ jobs:
6870
uses: actions/[email protected]
6971

7072
# bsdmainutils provides "column" which is used by the Makefile
71-
- name: Install Ubuntu packages
72-
run: apt-get update && apt-get install -y --no-install-recommends make gcc bsdmainutils
73+
# other packages are needed for cross-compilation
74+
- name: Install Ubuntu packages needed for cross-compilation
75+
run: |
76+
apt-get update && \
77+
apt-get install -y --no-install-recommends \
78+
make \
79+
bsdmainutils \
80+
gcc \
81+
gcc-multilib \
82+
gcc-mingw-w64
7383
7484
- name: Build using project Makefile
7585
run: make all
86+
87+
# This is run from *within* a container that is itself within the GitHub
88+
# Actions environment. All of these commands are run within our container.
89+
build_static_binaries_with_makefile:
90+
name: Build statically linked binaries using Makefile
91+
runs-on: ubuntu-latest
92+
# Default: 360 minutes
93+
timeout-minutes: 10
94+
container:
95+
image: "index.docker.io/golang:latest"
96+
97+
steps:
98+
- name: Print go version
99+
run: go version
100+
101+
- name: Check out code into the Go module directory
102+
uses: actions/[email protected]
103+
104+
# bsdmainutils provides "column" which is used by the Makefile
105+
# other packages are needed for cross-compilation
106+
- name: Install Ubuntu packages needed for cross-compilation
107+
run: |
108+
apt-get update && \
109+
apt-get install -y --no-install-recommends \
110+
make \
111+
bsdmainutils \
112+
gcc \
113+
gcc-multilib \
114+
gcc-mingw-w64
115+
116+
- name: Build using project Makefile
117+
run: make all-static
118+
119+
# This is run directly within the GitHub Actions environment and calls the
120+
# `docker` command to perform specific build tasks within Docker containers.
121+
# Prep steps are run within the GitHub Actions environment and not within
122+
# the containers.
123+
build_static_binaries_with_makefile_docker_recipe:
124+
name: Build static binaries using Docker images
125+
runs-on: ubuntu-latest
126+
# Default: 360 minutes
127+
timeout-minutes: 20
128+
129+
steps:
130+
- name: Print go version
131+
run: go version
132+
133+
- name: Check out code into the Go module directory
134+
uses: actions/[email protected]
135+
136+
# bsdmainutils provides "column" which is used by the Makefile
137+
- name: Install Ubuntu packages
138+
run: |
139+
sudo apt-get update && \
140+
sudo apt-get install -y --no-install-recommends \
141+
bsdmainutils \
142+
make
143+
144+
- name: Build using project Makefile Docker recipe
145+
run: make docker

‎Makefile

+173-25
Original file line numberDiff line numberDiff line change
@@ -19,30 +19,85 @@
1919
# https://gist.github.com/subfuzion/0bd969d08fe0d8b5cc4b23c795854a13
2020
# https://stackoverflow.com/questions/10858261/abort-makefile-if-variable-not-set
2121
# https://stackoverflow.com/questions/38801796/makefile-set-if-variable-is-empty
22+
# https://www.gnu.org/software/make/manual/make.html#Flavors
2223

23-
SHELL = /bin/bash
24+
SHELL := /bin/bash
2425

2526
# Space-separated list of cmd/BINARY_NAME directories to build
26-
WHAT = mysql2sqlite check_mysql2sqlite
27+
WHAT := mysql2sqlite check_mysql2sqlite
2728

2829
# What package holds the "version" variable used in branding/version output?
29-
# VERSION_VAR_PKG = $(shell go list .)
30-
# VERSION_VAR_PKG = main
31-
VERSION_VAR_PKG = $(shell go list .)/internal/config
30+
# VERSION_VAR_PKG := $(shell go list .)
31+
# VERSION_VAR_PKG := main
32+
VERSION_VAR_PKG := $(shell go list .)/internal/config
3233

33-
OUTPUTDIR = release_assets
34+
OUTPUTDIR := release_assets
3435

3536
# https://gist.github.com/TheHippo/7e4d9ec4b7ed4c0d7a39839e6800cc16
36-
VERSION = $(shell git describe --always --long --dirty)
37+
VERSION := $(shell git describe --always --long --dirty)
3738

3839
# The default `go build` process embeds debugging information. Building
3940
# without that debugging information reduces the binary size by around 28%.
40-
BUILDCMD = go build -mod=vendor -a -ldflags="-s -w -X $(VERSION_VAR_PKG).Version=$(VERSION)"
41-
GOCLEANCMD = go clean -mod=vendor ./...
42-
GITCLEANCMD = git clean -xfd
43-
CHECKSUMCMD = sha256sum -b
41+
#
42+
# We also include additional flags in an effort to generate static binaries
43+
# that do not have external dependencies. As of Go 1.15 this still appears to
44+
# be a mixed bag, so YMMV.
45+
#
46+
# See https://github.com/golang/go/issues/26492 for more information.
47+
#
48+
# -s
49+
# Omit the symbol table and debug information.
50+
#
51+
# -w
52+
# Omit the DWARF symbol table.
53+
#
54+
# -tags 'osusergo,netgo'
55+
# Use pure Go implementation of user and group id/name resolution.
56+
# Use pure Go implementation of DNS resolver.
57+
#
58+
# -extldflags '-static'
59+
# Pass 'static' flag to external linker.
60+
#
61+
# -linkmode=external
62+
# https://golang.org/src/cmd/cgo/doc.go
63+
#
64+
# NOTE: Using external linker requires installation of `gcc-multilib`
65+
# package when building 32-bit binaries on a Debian/Ubuntu system. It also
66+
# seems to result in an unstable build that crashes on startup. This *might*
67+
# be specific to the WSL environment used for builds, but since this is a
68+
# new issue and and I do not yet know much about this option, I am leaving
69+
# it out.
70+
#
71+
# CGO_ENABLED=1
72+
# CGO is disabled by default for cross-compilation. You need to enable it
73+
# explicitly to use CGO for multiple architectures.
74+
BUILD_LDFLAGS_COMMON := -s -w -X $(VERSION_VAR_PKG).version=$(VERSION)
75+
BUILD_LDFLAGS_STATIC := -linkmode=external -extldflags '-static'
76+
# BUILDCMD_STATIC := CGO_ENABLED=1 go build -mod=vendor -tags 'osusergo,netgo,sqlite_omit_load_extension' -a -ldflags "$(BUILD_LDFLAGS_STATIC) $(BUILD_LDFLAGS_COMMON)"
77+
# BUILDCMD_DYNAMIC := CGO_ENABLED=1 go build -mod=vendor -tags 'osusergo,netgo,sqlite_omit_load_extension' -a -ldflags "$(BUILD_LDFLAGS_COMMON)"
78+
BUILDCMD_COMMON := CGO_ENABLED=1 go build -mod=vendor -a
79+
BUILDCMD_STATIC := $(BUILDCMD_COMMON) -tags 'osusergo,netgo,sqlite_omit_load_extension' -ldflags "$(BUILD_LDFLAGS_STATIC) $(BUILD_LDFLAGS_COMMON)"
80+
BUILDCMD_DYNAMIC := $(BUILDCMD_COMMON) -ldflags "$(BUILD_LDFLAGS_COMMON)"
81+
82+
BUILD_TYPE_STATIC := static
83+
BUILD_TYPE_DYNAMIC := dynamic
84+
85+
# Default build command and type if not overridden
86+
BUILDCMD := $(BUILDCMD_DYNAMIC)
87+
BUILDTYPE := $(BUILD_TYPE_DYNAMIC)
4488

45-
.DEFAULT_GOAL := help
89+
# Use mingw as C compiler to build Windows cgo-enabled binaries.
90+
WINCOMPILERX86 := CC=i686-w64-mingw32-gcc
91+
WINCOMPILERX64 := CC=x86_64-w64-mingw32-gcc
92+
93+
DOCKER_BUILD_IMG_X86 := atc0005/go-ci:go-ci-stable-alpine-buildx86
94+
DOCKER_BUILD_IMG_X64 := atc0005/go-ci:go-ci-stable-alpine-buildx64
95+
96+
GOCLEANCMD := go clean -mod=vendor ./...
97+
GITCLEANCMD := git clean -xfd
98+
CHECKSUMCMD := sha256sum -b
99+
100+
.DEFAULT_GOAL := help
46101

47102
##########################################################################
48103
# Targets will not work properly if a file with the same name is ever
@@ -122,46 +177,139 @@ pristine: goclean gitclean
122177

123178
.PHONY: all
124179
# https://stackoverflow.com/questions/3267145/makefile-execute-another-target
125-
## all: generates assets for Linux distros and Windows
180+
## all: generates dynamically linked assets for Linux and Windows systems
126181
all: clean windows linux
127182
@echo "Completed all cross-platform builds ..."
128183

184+
.PHONE: all-static
185+
## all-static: generates statically linked x86 and x64 assets for Linux and Windows systems
186+
all-static: clean windows-static linux-static
187+
@echo "Completed all cross-platform builds ..."
188+
129189
.PHONY: windows
130-
## windows: generates assets for Windows systems
131-
windows:
132-
@echo "Building release assets for windows ..."
190+
## windows: generates dynamically linked x86 and x64 Windows assets
191+
windows: windows-x86 windows-x64
192+
@echo "Completed build tasks for windows"
193+
194+
.PHONY: windows-static
195+
## windows-static: generates dynamically linked x86 and x64 Windows assets
196+
windows-static: windows-x86-static windows-x64-static
197+
@echo "Completed build tasks for windows"
133198

199+
.PHONY: windows-x86
200+
## windows-x86: generates dynamically linked Windows x86 assets
201+
windows-x86:
202+
@echo "Building ($(BUILDTYPE)) release assets for windows x86 ..."
134203
@for target in $(WHAT); do \
135204
mkdir -p $(OUTPUTDIR)/$$target && \
136205
echo "Building $$target 386 binaries" && \
137-
env GOOS=windows GOARCH=386 $(BUILDCMD) -o $(OUTPUTDIR)/$$target/$$target-$(VERSION)-windows-386.exe ./cmd/$$target && \
206+
env GOOS=windows GOARCH=386 $(WINCOMPILERX86) $(BUILDCMD) -o $(OUTPUTDIR)/$$target/$$target-$(VERSION)-windows-386.exe ./cmd/$$target && \
207+
echo "Generating $$target x86 checksum files" && \
208+
cd $(OUTPUTDIR)/$$target && \
209+
$(CHECKSUMCMD) $$target-$(VERSION)-windows-386.exe > $$target-$(VERSION)-windows-386.exe.sha256 && \
210+
cd $$OLDPWD; \
211+
done
212+
@echo "Completed ($(BUILDTYPE)) release assets build tasks for windows x86"
213+
214+
.PHONY: windows-x86-static
215+
## windows-x86-static: generates assets statically, specifically for Windows x86 systems
216+
windows-x86-static: BUILDCMD = $(BUILDCMD_STATIC)
217+
windows-x86-static: BUILDTYPE = $(BUILD_TYPE_STATIC)
218+
windows-x86-static: windows-x86
219+
220+
.PHONY: windows-x64
221+
## windows-x64: generates assets specifically for x64 Windows systems
222+
windows-x64:
223+
@echo "Building ($(BUILDTYPE)) release assets for windows x64 ..."
224+
@for target in $(WHAT); do \
225+
mkdir -p $(OUTPUTDIR)/$$target && \
138226
echo "Building $$target amd64 binaries" && \
139-
env GOOS=windows GOARCH=amd64 $(BUILDCMD) -o $(OUTPUTDIR)/$$target/$$target-$(VERSION)-windows-amd64.exe ./cmd/$$target && \
227+
env GOOS=windows GOARCH=amd64 $(WINCOMPILERX64) $(BUILDCMD) -o $(OUTPUTDIR)/$$target/$$target-$(VERSION)-windows-amd64.exe ./cmd/$$target && \
140228
echo "Generating $$target checksum files" && \
141229
cd $(OUTPUTDIR)/$$target && \
142-
$(CHECKSUMCMD) $$target-$(VERSION)-windows-386.exe > $$target-$(VERSION)-windows-386.exe.sha256 && \
143230
$(CHECKSUMCMD) $$target-$(VERSION)-windows-amd64.exe > $$target-$(VERSION)-windows-amd64.exe.sha256 && \
144231
cd $$OLDPWD; \
145232
done
233+
@echo "Completed ($(BUILDTYPE)) release assets build tasks for windows x64"
146234

147-
@echo "Completed build tasks for windows"
235+
.PHONY: windows-x64-static
236+
## windows-x64-static: generates assets statically, specifically for Windows x64 systems
237+
windows-x64-static: BUILDCMD = $(BUILDCMD_STATIC)
238+
windows-x64-static: BUILDTYPE = $(BUILD_TYPE_STATIC)
239+
windows-x64-static: windows-x64
148240

149241
.PHONY: linux
150-
## linux: generates assets for Linux distros
151-
linux:
152-
@echo "Building release assets for linux ..."
242+
## linux: generates dynamically linked x86 and x64 assets for Linux distros
243+
linux: linux-x86 linux-x64
244+
@echo "Completed ($(BUILDTYPE)) release assets build tasks for linux"
245+
246+
.PHONE: linux-static
247+
## linux-static: generates statically linked x86 and x64 assets for Linux distros
248+
linux-static: linux-x86-static linux-x64-static
249+
@echo "Completed ($(BUILDTYPE)) release assets build tasks for linux"
250+
251+
.PHONY: linux-x86
252+
## linux-x86: generates assets specifically for Linux x86 systems
253+
linux-x86:
254+
@echo "Building ($(BUILDTYPE)) release assets for linux x86 ..."
153255

154256
@for target in $(WHAT); do \
155257
mkdir -p $(OUTPUTDIR)/$$target && \
156258
echo "Building $$target 386 binaries" && \
157259
env GOOS=linux GOARCH=386 $(BUILDCMD) -o $(OUTPUTDIR)/$$target/$$target-$(VERSION)-linux-386 ./cmd/$$target && \
260+
echo "Generating $$target checksum files" && \
261+
cd $(OUTPUTDIR)/$$target && \
262+
$(CHECKSUMCMD) $$target-$(VERSION)-linux-386 > $$target-$(VERSION)-linux-386.sha256 && \
263+
cd $$OLDPWD; \
264+
done
265+
266+
@echo "Completed ($(BUILDTYPE)) release assets build tasks for linux x86"
267+
268+
.PHONY: linux-x86-static
269+
## linux-x86-static: generates assets statically, specifically for Linux x86 systems
270+
linux-x86-static: BUILDCMD = $(BUILDCMD_STATIC)
271+
linux-x86-static: BUILDTYPE = $(BUILD_TYPE_STATIC)
272+
linux-x86-static: linux-x86
273+
274+
.PHONY: linux-x64
275+
## linux-x64: generates assets specifically for Linux x64 systems
276+
linux-x64:
277+
@echo "Building ($(BUILDTYPE)) release assets for linux x64 ..."
278+
279+
@for target in $(WHAT); do \
280+
mkdir -p $(OUTPUTDIR)/$$target && \
158281
echo "Building $$target amd64 binaries" && \
159282
env GOOS=linux GOARCH=amd64 $(BUILDCMD) -o $(OUTPUTDIR)/$$target/$$target-$(VERSION)-linux-amd64 ./cmd/$$target && \
160283
echo "Generating $$target checksum files" && \
161284
cd $(OUTPUTDIR)/$$target && \
162-
$(CHECKSUMCMD) $$target-$(VERSION)-linux-386 > $$target-$(VERSION)-linux-386.sha256 && \
163285
$(CHECKSUMCMD) $$target-$(VERSION)-linux-amd64 > $$target-$(VERSION)-linux-amd64.sha256 && \
164286
cd $$OLDPWD; \
165287
done
166288

167-
@echo "Completed build tasks for linux"
289+
@echo "Completed ($(BUILDTYPE)) release assets build tasks for linux x64"
290+
291+
292+
.PHONY: linux-x64-static
293+
## linux-x64-static: generates assets statically, specifically for Linux x64 systems
294+
linux-x64-static: BUILDCMD = $(BUILDCMD_STATIC)
295+
linux-x64-static: BUILDTYPE = $(BUILD_TYPE_STATIC)
296+
linux-x64-static: linux-x64
297+
298+
.PHONY: docker
299+
## docker: generates assets for Linux distros and Windows using Docker
300+
docker: clean
301+
@docker run \
302+
--rm \
303+
-i \
304+
-v $$PWD:$$PWD \
305+
-w $$PWD \
306+
$(DOCKER_BUILD_IMG_X86) \
307+
make windows-x86-static linux-x86-static
308+
@docker run \
309+
--rm \
310+
-i \
311+
-v $$PWD:$$PWD \
312+
-w $$PWD \
313+
$(DOCKER_BUILD_IMG_X64) \
314+
make windows-x64-static linux-x64-static
315+
@echo "Completed all cross-platform builds via Docker containers ..."

‎README.md

+7
Original file line numberDiff line numberDiff line change
@@ -95,13 +95,20 @@ been tested.
9595

9696
### Building source code
9797

98+
These requirements are specific to Ubuntu 18.04+. Packages will likely be
99+
named differently for other distributions.
100+
98101
- Go 1.14+
99102
- `CGO_ENABLED=1` environment variable (if not set by default)
100103
- requirement of SQLite database driver used
101104
- `GCC`
105+
- `GCC multilib`
106+
- `GCC for Windows` (`mingw-w64`)
102107
- `make`
103108
- if using the provided `Makefile`
104109

110+
See the [build](docs/build.md) instructions for more information.
111+
105112
### Running
106113

107114
- Windows 7, Server 2008R2 or later

‎contrib/mysql2sqlite/docker-compose.yml

+33
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
---
2+
# Copyright 2020 Adam Chalkley
3+
#
4+
# https://github.com/atc0005/mysql2sqlite
5+
#
6+
# Licensed under the MIT License. See LICENSE file in the project root for
7+
# full license information.
8+
9+
# Purpose: Batch build static binaries from this project's source code.
10+
11+
# Usage:
12+
#
13+
# Copy this file to the root of the repo and run it via `docker-compose up`.
14+
# This will kick off two parallel builds and thoroughly thrash the build
15+
# system. The upside is that this *may* better utilize multiple cores vs
16+
# spinning up one container at a time.
17+
18+
version: "2"
19+
20+
services:
21+
mysql2sqlite-x86:
22+
image: atc0005/go-ci:go-ci-stable-alpine-buildx86
23+
volumes:
24+
- "./:/src"
25+
working_dir: "/src"
26+
entrypoint: ["make", "windows-x86-static", "linux-x86-static"]
27+
28+
mysql2sqlite-x64:
29+
image: atc0005/go-ci:go-ci-stable-alpine-buildx64
30+
volumes:
31+
- "./:/src"
32+
working_dir: "/src"
33+
entrypoint: ["make", "windows-x64-static", "linux-x64-static"]

0 commit comments

Comments
 (0)