Skip to content

Commit a7adc49

Browse files
atc0005atc0005
committed
Refactor binary generation options
The previous release generated Windows binaries that were broken (GH-53) and Linux binaries that were unintentionally dynamic (GH-48). This commit provides an updated Makefile and GitHub workflow which provides multiple build options for both dynamic and static linking, but defaults to the previous dynamically linked behavior. The GitHub Workflow jobs based on the Makefile build tasks use a 20 minute timeout vs the 10 minute timeout used previously. Also, `CGO_ENABLED` has been explicitly set in order to ensure that cgo support is enabled. The Makefile build options now explicitly enable the `CGO_ENABLED` environment variable so that all builds have the required cgo functionality enabled. A `docker` Makefile recipe is provided to generate binaries using new Docker images from the `atc0005/go-ci` project based on the official Golang Alpine Linux image. The result is statically linked binaries based on the musl C library instead of glibc. The intent is to help prevent licensing issues surround the GNU C library's LGPL licensing (which I do not fully understand). Multiple build tags are specified for static builds which enable Go-specific replacements for common glibc-provided features: - `osusergo` - `netgo` and a build tag specific to disabling SQLite extensions, which we do not use with this specific project: - `sqlite_omit_load_extension` Minor documentation updates have been included which update the build requirements and specific steps for building binaries for this project. Further updates are likely needed to add polish. Several reference links have also been added. A Docker Compose file has been included for kicking off multiple static binary builds in parallel, but it may end up getting tossed in a later PR if we don't make sufficient use of it. - refs GH-48 - refs golang/go 38789 - refs golang/go 26492 - refs atc0005/go-ci#85
1 parent 9642c01 commit a7adc49

File tree

7 files changed

+344
-55
lines changed

7 files changed

+344
-55
lines changed

‎.github/workflows/lint-and-build-code.yml

+3-3
Original file line numberDiff line numberDiff line change
@@ -80,7 +80,7 @@ jobs:
8080
- name: Check out code
8181
uses: actions/[email protected]
8282

83-
- name: Build using vendored dependencies
83+
- name: Build using vendored dependencies and explicitly enabled CGO
8484
run: |
85-
go build -v -mod=vendor ./cmd/mysql2sqlite
86-
go build -v -mod=vendor ./cmd/check_mysql2sqlite
85+
CGO_ENABLED=1 go build -v -mod=vendor ./cmd/mysql2sqlite
86+
CGO_ENABLED=1 go build -v -mod=vendor ./cmd/check_mysql2sqlite

‎.github/workflows/lint-and-build-using-make.yml

+75-5
Original file line numberDiff line numberDiff line change
@@ -52,11 +52,13 @@ jobs:
5252
- name: Run Go linting tools using project Makefile
5353
run: make linting
5454

55-
build_code_with_makefile:
56-
name: Build codebase using Makefile
55+
# This is run from *within* a container that is itself within the GitHub
56+
# Actions environment. All of these commands are run within our container.
57+
build_dynamic_binaries_with_makefile:
58+
name: Build dynamically linked binaries using Makefile
5759
runs-on: ubuntu-latest
5860
# Default: 360 minutes
59-
timeout-minutes: 10
61+
timeout-minutes: 20
6062
container:
6163
image: "index.docker.io/golang:latest"
6264

@@ -68,8 +70,76 @@ jobs:
6870
uses: actions/[email protected]
6971

7072
# bsdmainutils provides "column" which is used by the Makefile
71-
- name: Install Ubuntu packages
72-
run: apt-get update && apt-get install -y --no-install-recommends make gcc bsdmainutils
73+
# other packages are needed for cross-compilation
74+
- name: Install Ubuntu packages needed for cross-compilation
75+
run: |
76+
apt-get update && \
77+
apt-get install -y --no-install-recommends \
78+
make \
79+
bsdmainutils \
80+
gcc \
81+
gcc-multilib \
82+
gcc-mingw-w64
7383
7484
- name: Build using project Makefile
7585
run: make all
86+
87+
# This is run from *within* a container that is itself within the GitHub
88+
# Actions environment. All of these commands are run within our container.
89+
build_static_binaries_with_makefile:
90+
name: Build statically linked binaries using Makefile
91+
runs-on: ubuntu-latest
92+
# Default: 360 minutes
93+
timeout-minutes: 20
94+
container:
95+
image: "index.docker.io/golang:latest"
96+
97+
steps:
98+
- name: Print go version
99+
run: go version
100+
101+
- name: Check out code into the Go module directory
102+
uses: actions/[email protected]
103+
104+
# bsdmainutils provides "column" which is used by the Makefile
105+
# other packages are needed for cross-compilation
106+
- name: Install Ubuntu packages needed for cross-compilation
107+
run: |
108+
apt-get update && \
109+
apt-get install -y --no-install-recommends \
110+
make \
111+
bsdmainutils \
112+
gcc \
113+
gcc-multilib \
114+
gcc-mingw-w64
115+
116+
- name: Build using project Makefile
117+
run: make all-static
118+
119+
# This is run directly within the GitHub Actions environment and calls the
120+
# `docker` command to perform specific build tasks within Docker containers.
121+
# Prep steps are run within the GitHub Actions environment and not within
122+
# the containers.
123+
build_static_binaries_with_makefile_docker_recipe:
124+
name: Build static binaries using Docker images
125+
runs-on: ubuntu-latest
126+
# Default: 360 minutes
127+
timeout-minutes: 20
128+
129+
steps:
130+
- name: Print go version
131+
run: go version
132+
133+
- name: Check out code into the Go module directory
134+
uses: actions/[email protected]
135+
136+
# bsdmainutils provides "column" which is used by the Makefile
137+
- name: Install Ubuntu packages
138+
run: |
139+
sudo apt-get update && \
140+
sudo apt-get install -y --no-install-recommends \
141+
bsdmainutils \
142+
make
143+
144+
- name: Build using project Makefile Docker recipe
145+
run: make docker

‎Makefile

+177-27
Original file line numberDiff line numberDiff line change
@@ -19,30 +19,91 @@
1919
# https://gist.github.com/subfuzion/0bd969d08fe0d8b5cc4b23c795854a13
2020
# https://stackoverflow.com/questions/10858261/abort-makefile-if-variable-not-set
2121
# https://stackoverflow.com/questions/38801796/makefile-set-if-variable-is-empty
22+
# https://www.gnu.org/software/make/manual/make.html#Flavors
23+
# https://stackoverflow.com/questions/6283320/vs-in-make-macros
2224

23-
SHELL = /bin/bash
25+
26+
#############################################################################
27+
# "Variables defined with := in GNU make are expanded when they are defined
28+
# rather than when they are used."
29+
#
30+
# https://stackoverflow.com/a/6283363/903870
31+
#############################################################################
32+
33+
SHELL := /bin/bash
2434

2535
# Space-separated list of cmd/BINARY_NAME directories to build
26-
WHAT = mysql2sqlite check_mysql2sqlite
36+
WHAT := mysql2sqlite check_mysql2sqlite
2737

2838
# What package holds the "version" variable used in branding/version output?
29-
# VERSION_VAR_PKG = $(shell go list .)
30-
# VERSION_VAR_PKG = main
31-
VERSION_VAR_PKG = $(shell go list .)/internal/config
39+
# VERSION_VAR_PKG := $(shell go list .)
40+
# VERSION_VAR_PKG := main
41+
VERSION_VAR_PKG := $(shell go list .)/internal/config
3242

33-
OUTPUTDIR = release_assets
43+
OUTPUTDIR := release_assets
3444

3545
# https://gist.github.com/TheHippo/7e4d9ec4b7ed4c0d7a39839e6800cc16
36-
VERSION = $(shell git describe --always --long --dirty)
46+
VERSION := $(shell git describe --always --long --dirty)
3747

3848
# The default `go build` process embeds debugging information. Building
3949
# without that debugging information reduces the binary size by around 28%.
40-
BUILDCMD = go build -mod=vendor -a -ldflags="-s -w -X $(VERSION_VAR_PKG).Version=$(VERSION)"
41-
GOCLEANCMD = go clean -mod=vendor ./...
42-
GITCLEANCMD = git clean -xfd
43-
CHECKSUMCMD = sha256sum -b
50+
#
51+
# We also include additional flags in an effort to generate static binaries
52+
# that do not have external dependencies. As of Go 1.15 this still appears to
53+
# be a mixed bag, so YMMV.
54+
#
55+
# See https://github.com/golang/go/issues/26492 for more information.
56+
#
57+
# -s
58+
# Omit the symbol table and debug information.
59+
#
60+
# -w
61+
# Omit the DWARF symbol table.
62+
#
63+
# -tags 'osusergo,netgo'
64+
# Use pure Go implementation of user and group id/name resolution.
65+
# Use pure Go implementation of DNS resolver.
66+
#
67+
# -extldflags '-static'
68+
# Pass 'static' flag to external linker.
69+
#
70+
# -linkmode=external
71+
# https://golang.org/src/cmd/cgo/doc.go
72+
#
73+
# NOTE: Using external linker requires installation of `gcc-multilib`
74+
# package when building 32-bit binaries on a Debian/Ubuntu system. It also
75+
# seems to result in an unstable build that crashes on startup. This *might*
76+
# be specific to the WSL environment used for builds. Further testing is
77+
# needed to confirm.
78+
#
79+
# CGO_ENABLED=1
80+
# CGO is disabled by default for cross-compilation. You need to enable it
81+
# explicitly to use CGO for multiple architectures.
82+
BUILD_LDFLAGS_COMMON := -s -w -X $(VERSION_VAR_PKG).Version=$(VERSION)
83+
BUILD_LDFLAGS_STATIC := -linkmode=external -extldflags '-static'
84+
BUILDCMD_COMMON := CGO_ENABLED=1 go build -mod=vendor -a
85+
BUILDCMD_STATIC := $(BUILDCMD_COMMON) -tags 'osusergo,netgo,sqlite_omit_load_extension' -ldflags "$(BUILD_LDFLAGS_STATIC) $(BUILD_LDFLAGS_COMMON)"
86+
BUILDCMD_DYNAMIC := $(BUILDCMD_COMMON) -ldflags "$(BUILD_LDFLAGS_COMMON)"
87+
88+
BUILD_TYPE_STATIC := static
89+
BUILD_TYPE_DYNAMIC := dynamic
90+
91+
# Default build command and type if not overridden
92+
BUILDCMD := $(BUILDCMD_DYNAMIC)
93+
BUILDTYPE := $(BUILD_TYPE_DYNAMIC)
94+
95+
# Use mingw as C compiler to build Windows cgo-enabled binaries.
96+
WINCOMPILERX86 := CC=i686-w64-mingw32-gcc
97+
WINCOMPILERX64 := CC=x86_64-w64-mingw32-gcc
4498

45-
.DEFAULT_GOAL := help
99+
DOCKER_BUILD_IMG_X86 := atc0005/go-ci:go-ci-stable-alpine-buildx86
100+
DOCKER_BUILD_IMG_X64 := atc0005/go-ci:go-ci-stable-alpine-buildx64
101+
102+
GOCLEANCMD := go clean -mod=vendor ./...
103+
GITCLEANCMD := git clean -xfd
104+
CHECKSUMCMD := sha256sum -b
105+
106+
.DEFAULT_GOAL := help
46107

47108
##########################################################################
48109
# Targets will not work properly if a file with the same name is ever
@@ -122,46 +183,135 @@ pristine: goclean gitclean
122183

123184
.PHONY: all
124185
# https://stackoverflow.com/questions/3267145/makefile-execute-another-target
125-
## all: generates assets for Linux distros and Windows
186+
## all: generates dynamically linked assets for Linux and Windows systems
126187
all: clean windows linux
127188
@echo "Completed all cross-platform builds ..."
128189

190+
.PHONE: all-static
191+
## all-static: generates statically linked x86 and x64 assets for Linux and Windows systems
192+
all-static: clean windows-static linux-static
193+
@echo "Completed all cross-platform builds ..."
194+
129195
.PHONY: windows
130-
## windows: generates assets for Windows systems
131-
windows:
132-
@echo "Building release assets for windows ..."
196+
## windows: generates dynamically linked x86 and x64 Windows assets
197+
windows: windows-x86 windows-x64
198+
@echo "Completed build tasks for windows"
133199

200+
.PHONY: windows-static
201+
## windows-static: generates dynamically linked x86 and x64 Windows assets
202+
windows-static: windows-x86-static windows-x64-static
203+
@echo "Completed build tasks for windows"
204+
205+
.PHONY: windows-x86
206+
## windows-x86: generates dynamically linked Windows x86 assets
207+
windows-x86:
208+
@echo "Building ($(BUILDTYPE)) release assets for windows x86 ..."
134209
@for target in $(WHAT); do \
135210
mkdir -p $(OUTPUTDIR)/$$target && \
136211
echo "Building $$target 386 binaries" && \
137-
env GOOS=windows GOARCH=386 $(BUILDCMD) -o $(OUTPUTDIR)/$$target/$$target-$(VERSION)-windows-386.exe ./cmd/$$target && \
212+
env GOOS=windows GOARCH=386 $(WINCOMPILERX86) $(BUILDCMD) -o $(OUTPUTDIR)/$$target/$$target-$(VERSION)-windows-386.exe ./cmd/$$target && \
213+
echo "Generating $$target x86 checksum files" && \
214+
cd $(OUTPUTDIR)/$$target && \
215+
$(CHECKSUMCMD) $$target-$(VERSION)-windows-386.exe > $$target-$(VERSION)-windows-386.exe.sha256 && \
216+
cd $$OLDPWD; \
217+
done
218+
@echo "Completed ($(BUILDTYPE)) release assets build tasks for windows x86"
219+
220+
.PHONY: windows-x86-static
221+
## windows-x86-static: generates assets statically, specifically for Windows x86 systems
222+
windows-x86-static: BUILDCMD = $(BUILDCMD_STATIC)
223+
windows-x86-static: BUILDTYPE = $(BUILD_TYPE_STATIC)
224+
windows-x86-static: windows-x86
225+
226+
.PHONY: windows-x64
227+
## windows-x64: generates assets specifically for x64 Windows systems
228+
windows-x64:
229+
@echo "Building ($(BUILDTYPE)) release assets for windows x64 ..."
230+
@for target in $(WHAT); do \
231+
mkdir -p $(OUTPUTDIR)/$$target && \
138232
echo "Building $$target amd64 binaries" && \
139-
env GOOS=windows GOARCH=amd64 $(BUILDCMD) -o $(OUTPUTDIR)/$$target/$$target-$(VERSION)-windows-amd64.exe ./cmd/$$target && \
233+
env GOOS=windows GOARCH=amd64 $(WINCOMPILERX64) $(BUILDCMD) -o $(OUTPUTDIR)/$$target/$$target-$(VERSION)-windows-amd64.exe ./cmd/$$target && \
140234
echo "Generating $$target checksum files" && \
141235
cd $(OUTPUTDIR)/$$target && \
142-
$(CHECKSUMCMD) $$target-$(VERSION)-windows-386.exe > $$target-$(VERSION)-windows-386.exe.sha256 && \
143236
$(CHECKSUMCMD) $$target-$(VERSION)-windows-amd64.exe > $$target-$(VERSION)-windows-amd64.exe.sha256 && \
144237
cd $$OLDPWD; \
145238
done
239+
@echo "Completed ($(BUILDTYPE)) release assets build tasks for windows x64"
146240

147-
@echo "Completed build tasks for windows"
241+
.PHONY: windows-x64-static
242+
## windows-x64-static: generates assets statically, specifically for Windows x64 systems
243+
windows-x64-static: BUILDCMD = $(BUILDCMD_STATIC)
244+
windows-x64-static: BUILDTYPE = $(BUILD_TYPE_STATIC)
245+
windows-x64-static: windows-x64
148246

149247
.PHONY: linux
150-
## linux: generates assets for Linux distros
151-
linux:
152-
@echo "Building release assets for linux ..."
153-
248+
## linux: generates dynamically linked x86 and x64 assets for Linux distros
249+
linux: linux-x86 linux-x64
250+
@echo "Completed ($(BUILDTYPE)) release assets build tasks for linux"
251+
252+
.PHONE: linux-static
253+
## linux-static: generates statically linked x86 and x64 assets for Linux distros
254+
linux-static: linux-x86-static linux-x64-static
255+
@echo "Completed ($(BUILDTYPE)) release assets build tasks for linux"
256+
257+
.PHONY: linux-x86
258+
## linux-x86: generates assets specifically for Linux x86 systems
259+
linux-x86:
260+
@echo "Building ($(BUILDTYPE)) release assets for linux x86 ..."
154261
@for target in $(WHAT); do \
155262
mkdir -p $(OUTPUTDIR)/$$target && \
156263
echo "Building $$target 386 binaries" && \
157264
env GOOS=linux GOARCH=386 $(BUILDCMD) -o $(OUTPUTDIR)/$$target/$$target-$(VERSION)-linux-386 ./cmd/$$target && \
265+
echo "Generating $$target checksum files" && \
266+
cd $(OUTPUTDIR)/$$target && \
267+
$(CHECKSUMCMD) $$target-$(VERSION)-linux-386 > $$target-$(VERSION)-linux-386.sha256 && \
268+
cd $$OLDPWD; \
269+
done
270+
@echo "Completed ($(BUILDTYPE)) release assets build tasks for linux x86"
271+
272+
.PHONY: linux-x86-static
273+
## linux-x86-static: generates assets statically, specifically for Linux x86 systems
274+
linux-x86-static: BUILDCMD = $(BUILDCMD_STATIC)
275+
linux-x86-static: BUILDTYPE = $(BUILD_TYPE_STATIC)
276+
linux-x86-static: linux-x86
277+
278+
.PHONY: linux-x64
279+
## linux-x64: generates assets specifically for Linux x64 systems
280+
linux-x64:
281+
@echo "Building ($(BUILDTYPE)) release assets for linux x64 ..."
282+
@for target in $(WHAT); do \
283+
mkdir -p $(OUTPUTDIR)/$$target && \
158284
echo "Building $$target amd64 binaries" && \
159285
env GOOS=linux GOARCH=amd64 $(BUILDCMD) -o $(OUTPUTDIR)/$$target/$$target-$(VERSION)-linux-amd64 ./cmd/$$target && \
160286
echo "Generating $$target checksum files" && \
161287
cd $(OUTPUTDIR)/$$target && \
162-
$(CHECKSUMCMD) $$target-$(VERSION)-linux-386 > $$target-$(VERSION)-linux-386.sha256 && \
163288
$(CHECKSUMCMD) $$target-$(VERSION)-linux-amd64 > $$target-$(VERSION)-linux-amd64.sha256 && \
164289
cd $$OLDPWD; \
165290
done
166-
167-
@echo "Completed build tasks for linux"
291+
@echo "Completed ($(BUILDTYPE)) release assets build tasks for linux x64"
292+
293+
294+
.PHONY: linux-x64-static
295+
## linux-x64-static: generates assets statically, specifically for Linux x64 systems
296+
linux-x64-static: BUILDCMD = $(BUILDCMD_STATIC)
297+
linux-x64-static: BUILDTYPE = $(BUILD_TYPE_STATIC)
298+
linux-x64-static: linux-x64
299+
300+
.PHONY: docker
301+
## docker: generates assets for Linux distros and Windows using Docker
302+
docker: clean
303+
@docker run \
304+
--rm \
305+
-i \
306+
-v $$PWD:$$PWD \
307+
-w $$PWD \
308+
$(DOCKER_BUILD_IMG_X86) \
309+
make windows-x86-static linux-x86-static
310+
@docker run \
311+
--rm \
312+
-i \
313+
-v $$PWD:$$PWD \
314+
-w $$PWD \
315+
$(DOCKER_BUILD_IMG_X64) \
316+
make windows-x64-static linux-x64-static
317+
@echo "Completed all cross-platform builds via Docker containers ..."

‎README.md

+7
Original file line numberDiff line numberDiff line change
@@ -95,13 +95,20 @@ been tested.
9595

9696
### Building source code
9797

98+
These requirements are specific to Debian/Ubuntu-based distros. Packages will
99+
likely be named differently for other distributions.
100+
98101
- Go 1.14+
99102
- `CGO_ENABLED=1` environment variable (if not set by default)
100103
- requirement of SQLite database driver used
101104
- `GCC`
105+
- `GCC multilib`
106+
- `GCC for Windows` (`mingw-w64`)
102107
- `make`
103108
- if using the provided `Makefile`
104109

110+
See the [build](docs/build.md) instructions for more information.
111+
105112
### Running
106113

107114
- Windows 7, Server 2008R2 or later

0 commit comments

Comments
 (0)