Skip to content

Commit bf9b80c

Browse files
atc0005atc0005
committed
Refactor binary generation options
The previous release generated Windows binaries that were broken (GH-53) and Linux binaries that were unintentionally dynamic (GH-48). This commit provides an updated Makefile and GitHub workflow which provides multiple build options for both dynamic and static linking, but defaults to the previous dynamically linked behavior. The GitHub Workflow jobs based on the Makefile build tasks use a 20 minute timeout vs the 10 minute timeout used previously. Also, `CGO_ENABLED` has been explicitly set in order to ensure that cgo support is enabled. The Makefile build options now explicitly enable the `CGO_ENABLED` environment variable so that all builds have the required cgo functionality enabled. A `docker` Makefile recipe is provided to generate binaries using new Docker images from the `atc0005/go-ci` project based on the official Golang Alpine Linux image. The result is statically linked binaries based on the musl C library instead of glibc. The intent is to help prevent licensing issues surround the GNU C library's LGPL licensing (which I do not fully understand). Multiple build tags are specified for static builds which enable Go-specific replacements for common glibc-provided features: - `osusergo` - `netgo` and a build tag specific to disabling SQLite extensions, which we do not use with this specific project: - `sqlite_omit_load_extension` Minor documentation updates have been included which update the build requirements and specific steps for building binaries for this project. Further updates are likely needed to add polish. A Docker Compose file has been included for kicking off multiple static binary builds in parallel, but it may end up getting tossed in a later PR if we don't make sufficient use of it. - refs GH-48 - refs golang/go 38789 - refs golang/go 26492 - refs atc0005/go-ci#85
1 parent 9642c01 commit bf9b80c

File tree

7 files changed

+327
-53
lines changed

7 files changed

+327
-53
lines changed

‎.github/workflows/lint-and-build-code.yml

+3-3
Original file line numberDiff line numberDiff line change
@@ -80,7 +80,7 @@ jobs:
8080
- name: Check out code
8181
uses: actions/[email protected]
8282

83-
- name: Build using vendored dependencies
83+
- name: Build using vendored dependencies and explicitly enabled CGO
8484
run: |
85-
go build -v -mod=vendor ./cmd/mysql2sqlite
86-
go build -v -mod=vendor ./cmd/check_mysql2sqlite
85+
CGO_ENABLED=1 go build -v -mod=vendor ./cmd/mysql2sqlite
86+
CGO_ENABLED=1 go build -v -mod=vendor ./cmd/check_mysql2sqlite

‎.github/workflows/lint-and-build-using-make.yml

+75-5
Original file line numberDiff line numberDiff line change
@@ -52,11 +52,13 @@ jobs:
5252
- name: Run Go linting tools using project Makefile
5353
run: make linting
5454

55-
build_code_with_makefile:
56-
name: Build codebase using Makefile
55+
# This is run from *within* a container that is itself within the GitHub
56+
# Actions environment. All of these commands are run within our container.
57+
build_dynamic_binaries_with_makefile:
58+
name: Build dynamically linked binaries using Makefile
5759
runs-on: ubuntu-latest
5860
# Default: 360 minutes
59-
timeout-minutes: 10
61+
timeout-minutes: 20
6062
container:
6163
image: "index.docker.io/golang:latest"
6264

@@ -68,8 +70,76 @@ jobs:
6870
uses: actions/[email protected]
6971

7072
# bsdmainutils provides "column" which is used by the Makefile
71-
- name: Install Ubuntu packages
72-
run: apt-get update && apt-get install -y --no-install-recommends make gcc bsdmainutils
73+
# other packages are needed for cross-compilation
74+
- name: Install Ubuntu packages needed for cross-compilation
75+
run: |
76+
apt-get update && \
77+
apt-get install -y --no-install-recommends \
78+
make \
79+
bsdmainutils \
80+
gcc \
81+
gcc-multilib \
82+
gcc-mingw-w64
7383
7484
- name: Build using project Makefile
7585
run: make all
86+
87+
# This is run from *within* a container that is itself within the GitHub
88+
# Actions environment. All of these commands are run within our container.
89+
build_static_binaries_with_makefile:
90+
name: Build statically linked binaries using Makefile
91+
runs-on: ubuntu-latest
92+
# Default: 360 minutes
93+
timeout-minutes: 20
94+
container:
95+
image: "index.docker.io/golang:latest"
96+
97+
steps:
98+
- name: Print go version
99+
run: go version
100+
101+
- name: Check out code into the Go module directory
102+
uses: actions/[email protected]
103+
104+
# bsdmainutils provides "column" which is used by the Makefile
105+
# other packages are needed for cross-compilation
106+
- name: Install Ubuntu packages needed for cross-compilation
107+
run: |
108+
apt-get update && \
109+
apt-get install -y --no-install-recommends \
110+
make \
111+
bsdmainutils \
112+
gcc \
113+
gcc-multilib \
114+
gcc-mingw-w64
115+
116+
- name: Build using project Makefile
117+
run: make all-static
118+
119+
# This is run directly within the GitHub Actions environment and calls the
120+
# `docker` command to perform specific build tasks within Docker containers.
121+
# Prep steps are run within the GitHub Actions environment and not within
122+
# the containers.
123+
build_static_binaries_with_makefile_docker_recipe:
124+
name: Build static binaries using Docker images
125+
runs-on: ubuntu-latest
126+
# Default: 360 minutes
127+
timeout-minutes: 20
128+
129+
steps:
130+
- name: Print go version
131+
run: go version
132+
133+
- name: Check out code into the Go module directory
134+
uses: actions/[email protected]
135+
136+
# bsdmainutils provides "column" which is used by the Makefile
137+
- name: Install Ubuntu packages
138+
run: |
139+
sudo apt-get update && \
140+
sudo apt-get install -y --no-install-recommends \
141+
bsdmainutils \
142+
make
143+
144+
- name: Build using project Makefile Docker recipe
145+
run: make docker

‎Makefile

+170-25
Original file line numberDiff line numberDiff line change
@@ -19,30 +19,82 @@
1919
# https://gist.github.com/subfuzion/0bd969d08fe0d8b5cc4b23c795854a13
2020
# https://stackoverflow.com/questions/10858261/abort-makefile-if-variable-not-set
2121
# https://stackoverflow.com/questions/38801796/makefile-set-if-variable-is-empty
22+
# https://www.gnu.org/software/make/manual/make.html#Flavors
2223

23-
SHELL = /bin/bash
24+
SHELL := /bin/bash
2425

2526
# Space-separated list of cmd/BINARY_NAME directories to build
26-
WHAT = mysql2sqlite check_mysql2sqlite
27+
WHAT := mysql2sqlite check_mysql2sqlite
2728

2829
# What package holds the "version" variable used in branding/version output?
29-
# VERSION_VAR_PKG = $(shell go list .)
30-
# VERSION_VAR_PKG = main
31-
VERSION_VAR_PKG = $(shell go list .)/internal/config
30+
# VERSION_VAR_PKG := $(shell go list .)
31+
# VERSION_VAR_PKG := main
32+
VERSION_VAR_PKG := $(shell go list .)/internal/config
3233

33-
OUTPUTDIR = release_assets
34+
OUTPUTDIR := release_assets
3435

3536
# https://gist.github.com/TheHippo/7e4d9ec4b7ed4c0d7a39839e6800cc16
36-
VERSION = $(shell git describe --always --long --dirty)
37+
VERSION := $(shell git describe --always --long --dirty)
3738

3839
# The default `go build` process embeds debugging information. Building
3940
# without that debugging information reduces the binary size by around 28%.
40-
BUILDCMD = go build -mod=vendor -a -ldflags="-s -w -X $(VERSION_VAR_PKG).Version=$(VERSION)"
41-
GOCLEANCMD = go clean -mod=vendor ./...
42-
GITCLEANCMD = git clean -xfd
43-
CHECKSUMCMD = sha256sum -b
41+
#
42+
# We also include additional flags in an effort to generate static binaries
43+
# that do not have external dependencies. As of Go 1.15 this still appears to
44+
# be a mixed bag, so YMMV.
45+
#
46+
# See https://github.com/golang/go/issues/26492 for more information.
47+
#
48+
# -s
49+
# Omit the symbol table and debug information.
50+
#
51+
# -w
52+
# Omit the DWARF symbol table.
53+
#
54+
# -tags 'osusergo,netgo'
55+
# Use pure Go implementation of user and group id/name resolution.
56+
# Use pure Go implementation of DNS resolver.
57+
#
58+
# -extldflags '-static'
59+
# Pass 'static' flag to external linker.
60+
#
61+
# -linkmode=external
62+
# https://golang.org/src/cmd/cgo/doc.go
63+
#
64+
# NOTE: Using external linker requires installation of `gcc-multilib`
65+
# package when building 32-bit binaries on a Debian/Ubuntu system. It also
66+
# seems to result in an unstable build that crashes on startup. This *might*
67+
# be specific to the WSL environment used for builds. Further testing is
68+
# needed to confirm.
69+
#
70+
# CGO_ENABLED=1
71+
# CGO is disabled by default for cross-compilation. You need to enable it
72+
# explicitly to use CGO for multiple architectures.
73+
BUILD_LDFLAGS_COMMON := -s -w -X $(VERSION_VAR_PKG).version=$(VERSION)
74+
BUILD_LDFLAGS_STATIC := -linkmode=external -extldflags '-static'
75+
BUILDCMD_COMMON := CGO_ENABLED=1 go build -mod=vendor -a
76+
BUILDCMD_STATIC := $(BUILDCMD_COMMON) -tags 'osusergo,netgo,sqlite_omit_load_extension' -ldflags "$(BUILD_LDFLAGS_STATIC) $(BUILD_LDFLAGS_COMMON)"
77+
BUILDCMD_DYNAMIC := $(BUILDCMD_COMMON) -ldflags "$(BUILD_LDFLAGS_COMMON)"
78+
79+
BUILD_TYPE_STATIC := static
80+
BUILD_TYPE_DYNAMIC := dynamic
81+
82+
# Default build command and type if not overridden
83+
BUILDCMD := $(BUILDCMD_DYNAMIC)
84+
BUILDTYPE := $(BUILD_TYPE_DYNAMIC)
4485

45-
.DEFAULT_GOAL := help
86+
# Use mingw as C compiler to build Windows cgo-enabled binaries.
87+
WINCOMPILERX86 := CC=i686-w64-mingw32-gcc
88+
WINCOMPILERX64 := CC=x86_64-w64-mingw32-gcc
89+
90+
DOCKER_BUILD_IMG_X86 := atc0005/go-ci:go-ci-stable-alpine-buildx86
91+
DOCKER_BUILD_IMG_X64 := atc0005/go-ci:go-ci-stable-alpine-buildx64
92+
93+
GOCLEANCMD := go clean -mod=vendor ./...
94+
GITCLEANCMD := git clean -xfd
95+
CHECKSUMCMD := sha256sum -b
96+
97+
.DEFAULT_GOAL := help
4698

4799
##########################################################################
48100
# Targets will not work properly if a file with the same name is ever
@@ -122,46 +174,139 @@ pristine: goclean gitclean
122174

123175
.PHONY: all
124176
# https://stackoverflow.com/questions/3267145/makefile-execute-another-target
125-
## all: generates assets for Linux distros and Windows
177+
## all: generates dynamically linked assets for Linux and Windows systems
126178
all: clean windows linux
127179
@echo "Completed all cross-platform builds ..."
128180

181+
.PHONE: all-static
182+
## all-static: generates statically linked x86 and x64 assets for Linux and Windows systems
183+
all-static: clean windows-static linux-static
184+
@echo "Completed all cross-platform builds ..."
185+
129186
.PHONY: windows
130-
## windows: generates assets for Windows systems
131-
windows:
132-
@echo "Building release assets for windows ..."
187+
## windows: generates dynamically linked x86 and x64 Windows assets
188+
windows: windows-x86 windows-x64
189+
@echo "Completed build tasks for windows"
190+
191+
.PHONY: windows-static
192+
## windows-static: generates dynamically linked x86 and x64 Windows assets
193+
windows-static: windows-x86-static windows-x64-static
194+
@echo "Completed build tasks for windows"
133195

196+
.PHONY: windows-x86
197+
## windows-x86: generates dynamically linked Windows x86 assets
198+
windows-x86:
199+
@echo "Building ($(BUILDTYPE)) release assets for windows x86 ..."
134200
@for target in $(WHAT); do \
135201
mkdir -p $(OUTPUTDIR)/$$target && \
136202
echo "Building $$target 386 binaries" && \
137-
env GOOS=windows GOARCH=386 $(BUILDCMD) -o $(OUTPUTDIR)/$$target/$$target-$(VERSION)-windows-386.exe ./cmd/$$target && \
203+
env GOOS=windows GOARCH=386 $(WINCOMPILERX86) $(BUILDCMD) -o $(OUTPUTDIR)/$$target/$$target-$(VERSION)-windows-386.exe ./cmd/$$target && \
204+
echo "Generating $$target x86 checksum files" && \
205+
cd $(OUTPUTDIR)/$$target && \
206+
$(CHECKSUMCMD) $$target-$(VERSION)-windows-386.exe > $$target-$(VERSION)-windows-386.exe.sha256 && \
207+
cd $$OLDPWD; \
208+
done
209+
@echo "Completed ($(BUILDTYPE)) release assets build tasks for windows x86"
210+
211+
.PHONY: windows-x86-static
212+
## windows-x86-static: generates assets statically, specifically for Windows x86 systems
213+
windows-x86-static: BUILDCMD = $(BUILDCMD_STATIC)
214+
windows-x86-static: BUILDTYPE = $(BUILD_TYPE_STATIC)
215+
windows-x86-static: windows-x86
216+
217+
.PHONY: windows-x64
218+
## windows-x64: generates assets specifically for x64 Windows systems
219+
windows-x64:
220+
@echo "Building ($(BUILDTYPE)) release assets for windows x64 ..."
221+
@for target in $(WHAT); do \
222+
mkdir -p $(OUTPUTDIR)/$$target && \
138223
echo "Building $$target amd64 binaries" && \
139-
env GOOS=windows GOARCH=amd64 $(BUILDCMD) -o $(OUTPUTDIR)/$$target/$$target-$(VERSION)-windows-amd64.exe ./cmd/$$target && \
224+
env GOOS=windows GOARCH=amd64 $(WINCOMPILERX64) $(BUILDCMD) -o $(OUTPUTDIR)/$$target/$$target-$(VERSION)-windows-amd64.exe ./cmd/$$target && \
140225
echo "Generating $$target checksum files" && \
141226
cd $(OUTPUTDIR)/$$target && \
142-
$(CHECKSUMCMD) $$target-$(VERSION)-windows-386.exe > $$target-$(VERSION)-windows-386.exe.sha256 && \
143227
$(CHECKSUMCMD) $$target-$(VERSION)-windows-amd64.exe > $$target-$(VERSION)-windows-amd64.exe.sha256 && \
144228
cd $$OLDPWD; \
145229
done
230+
@echo "Completed ($(BUILDTYPE)) release assets build tasks for windows x64"
146231

147-
@echo "Completed build tasks for windows"
232+
.PHONY: windows-x64-static
233+
## windows-x64-static: generates assets statically, specifically for Windows x64 systems
234+
windows-x64-static: BUILDCMD = $(BUILDCMD_STATIC)
235+
windows-x64-static: BUILDTYPE = $(BUILD_TYPE_STATIC)
236+
windows-x64-static: windows-x64
148237

149238
.PHONY: linux
150-
## linux: generates assets for Linux distros
151-
linux:
152-
@echo "Building release assets for linux ..."
239+
## linux: generates dynamically linked x86 and x64 assets for Linux distros
240+
linux: linux-x86 linux-x64
241+
@echo "Completed ($(BUILDTYPE)) release assets build tasks for linux"
242+
243+
.PHONE: linux-static
244+
## linux-static: generates statically linked x86 and x64 assets for Linux distros
245+
linux-static: linux-x86-static linux-x64-static
246+
@echo "Completed ($(BUILDTYPE)) release assets build tasks for linux"
247+
248+
.PHONY: linux-x86
249+
## linux-x86: generates assets specifically for Linux x86 systems
250+
linux-x86:
251+
@echo "Building ($(BUILDTYPE)) release assets for linux x86 ..."
153252

154253
@for target in $(WHAT); do \
155254
mkdir -p $(OUTPUTDIR)/$$target && \
156255
echo "Building $$target 386 binaries" && \
157256
env GOOS=linux GOARCH=386 $(BUILDCMD) -o $(OUTPUTDIR)/$$target/$$target-$(VERSION)-linux-386 ./cmd/$$target && \
257+
echo "Generating $$target checksum files" && \
258+
cd $(OUTPUTDIR)/$$target && \
259+
$(CHECKSUMCMD) $$target-$(VERSION)-linux-386 > $$target-$(VERSION)-linux-386.sha256 && \
260+
cd $$OLDPWD; \
261+
done
262+
263+
@echo "Completed ($(BUILDTYPE)) release assets build tasks for linux x86"
264+
265+
.PHONY: linux-x86-static
266+
## linux-x86-static: generates assets statically, specifically for Linux x86 systems
267+
linux-x86-static: BUILDCMD = $(BUILDCMD_STATIC)
268+
linux-x86-static: BUILDTYPE = $(BUILD_TYPE_STATIC)
269+
linux-x86-static: linux-x86
270+
271+
.PHONY: linux-x64
272+
## linux-x64: generates assets specifically for Linux x64 systems
273+
linux-x64:
274+
@echo "Building ($(BUILDTYPE)) release assets for linux x64 ..."
275+
276+
@for target in $(WHAT); do \
277+
mkdir -p $(OUTPUTDIR)/$$target && \
158278
echo "Building $$target amd64 binaries" && \
159279
env GOOS=linux GOARCH=amd64 $(BUILDCMD) -o $(OUTPUTDIR)/$$target/$$target-$(VERSION)-linux-amd64 ./cmd/$$target && \
160280
echo "Generating $$target checksum files" && \
161281
cd $(OUTPUTDIR)/$$target && \
162-
$(CHECKSUMCMD) $$target-$(VERSION)-linux-386 > $$target-$(VERSION)-linux-386.sha256 && \
163282
$(CHECKSUMCMD) $$target-$(VERSION)-linux-amd64 > $$target-$(VERSION)-linux-amd64.sha256 && \
164283
cd $$OLDPWD; \
165284
done
166285

167-
@echo "Completed build tasks for linux"
286+
@echo "Completed ($(BUILDTYPE)) release assets build tasks for linux x64"
287+
288+
289+
.PHONY: linux-x64-static
290+
## linux-x64-static: generates assets statically, specifically for Linux x64 systems
291+
linux-x64-static: BUILDCMD = $(BUILDCMD_STATIC)
292+
linux-x64-static: BUILDTYPE = $(BUILD_TYPE_STATIC)
293+
linux-x64-static: linux-x64
294+
295+
.PHONY: docker
296+
## docker: generates assets for Linux distros and Windows using Docker
297+
docker: clean
298+
@docker run \
299+
--rm \
300+
-i \
301+
-v $$PWD:$$PWD \
302+
-w $$PWD \
303+
$(DOCKER_BUILD_IMG_X86) \
304+
make windows-x86-static linux-x86-static
305+
@docker run \
306+
--rm \
307+
-i \
308+
-v $$PWD:$$PWD \
309+
-w $$PWD \
310+
$(DOCKER_BUILD_IMG_X64) \
311+
make windows-x64-static linux-x64-static
312+
@echo "Completed all cross-platform builds via Docker containers ..."

‎README.md

+7
Original file line numberDiff line numberDiff line change
@@ -95,13 +95,20 @@ been tested.
9595

9696
### Building source code
9797

98+
These requirements are specific to Ubuntu 18.04+. Packages will likely be
99+
named differently for other distributions.
100+
98101
- Go 1.14+
99102
- `CGO_ENABLED=1` environment variable (if not set by default)
100103
- requirement of SQLite database driver used
101104
- `GCC`
105+
- `GCC multilib`
106+
- `GCC for Windows` (`mingw-w64`)
102107
- `make`
103108
- if using the provided `Makefile`
104109

110+
See the [build](docs/build.md) instructions for more information.
111+
105112
### Running
106113

107114
- Windows 7, Server 2008R2 or later

0 commit comments

Comments
 (0)