Skip to content

Commit e94d3d3

Browse files
TimPansinomergify[bot]
TimPansino
and
mergify[bot]
authored
Backport main into develop-scikitlearn (#847)
* Containerized CI Pipeline (#836) * Revert "Remove Python 2.7 and pypy2 testing (#835)" This reverts commit abb6405. * Containerize CI process * Publish new docker container for CI images * Rename github actions job * Copyright tag scripts * Drop debug line * Swap to new CI image * Move pip install to just main python * Remove libcurl special case from tox * Install special case packages into main image * Remove unused packages * Remove all other triggers besides manual * Add make run command * Cleanup small bugs * Fix CI Image Tagging (#838) * Correct templated CI image name * Pin pypy2.7 in image * Fix up scripting * Temporarily Restore Old CI Pipeline (#841) * Restore old pipelines * Remove python 2 from setup-python * Rework CI Pipeline (#839) Change pypy to pypy27 in tox. Fix checkout logic Pin tox requires * Fix Tests on New CI (#843) * Remove non-root user * Test new CI image * Change pypy to pypy27 in tox. * Fix checkout logic * Fetch git tags properly * Pin tox requires * Adjust default db settings for github actions * Rename elasticsearch services * Reset to new pipelines * [Mega-Linter] Apply linters fixes * Fix timezone * Fix docker networking * Pin dev image to new sha * Standardize gearman DB settings * Fix elasticsearch settings bug * Fix gearman bug * Add missing odbc headers * Add more debug messages * Swap out dev ci image * Fix required virtualenv version * Swap out dev ci image * Swap out dev ci image * Remove aioredis v1 for EOL * Add coverage paths for docker container * Unpin ci container --------- Co-authored-by: TimPansino <[email protected]> * Fix pypy27 dependency * Add skip for OTLP on py27 --------- Co-authored-by: TimPansino <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
1 parent cccd7f4 commit e94d3d3

File tree

15 files changed

+674
-347
lines changed

15 files changed

+674
-347
lines changed

‎.github/containers/Dockerfile‎

Lines changed: 83 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,83 @@
1+
2+
# Copyright 2010 New Relic, Inc.
3+
#
4+
# Licensed under the Apache License, Version 2.0 (the "License");
5+
# you may not use this file except in compliance with the License.
6+
# You may obtain a copy of the License at
7+
#
8+
# http://www.apache.org/licenses/LICENSE-2.0
9+
#
10+
# Unless required by applicable law or agreed to in writing, software
11+
# distributed under the License is distributed on an "AS IS" BASIS,
12+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
# See the License for the specific language governing permissions and
14+
# limitations under the License.
15+
16+
FROM ubuntu:20.04
17+
18+
# Install OS packages
19+
RUN export DEBIAN_FRONTEND=noninteractive && \
20+
apt-get update && \
21+
apt-get install -y \
22+
bash \
23+
build-essential \
24+
curl \
25+
expat \
26+
gcc \
27+
git \
28+
libbz2-dev \
29+
libcurl4-openssl-dev \
30+
libffi-dev \
31+
libgmp-dev \
32+
liblzma-dev \
33+
libmpfr-dev \
34+
libncurses-dev \
35+
libpq-dev \
36+
libreadline-dev \
37+
libsqlite3-dev \
38+
libssl-dev \
39+
locales \
40+
make \
41+
odbc-postgresql \
42+
openssl \
43+
python2-dev \
44+
python3-dev \
45+
python3-pip \
46+
tzdata \
47+
unixodbc-dev \
48+
unzip \
49+
wget \
50+
zip \
51+
zlib1g \
52+
zlib1g-dev && \
53+
rm -rf /var/lib/apt/lists/*
54+
55+
# Setup ODBC config
56+
RUN sed -i 's/Driver=psqlodbca.so/Driver=\/usr\/lib\/x86_64-linux-gnu\/odbc\/psqlodbca.so/g' /etc/odbcinst.ini && \
57+
sed -i 's/Driver=psqlodbcw.so/Driver=\/usr\/lib\/x86_64-linux-gnu\/odbc\/psqlodbcw.so/g' /etc/odbcinst.ini && \
58+
sed -i 's/Setup=libodbcpsqlS.so/Setup=\/usr\/lib\/x86_64-linux-gnu\/odbc\/libodbcpsqlS.so/g' /etc/odbcinst.ini
59+
60+
# Set the locale
61+
RUN locale-gen --no-purge en_US.UTF-8
62+
ENV LANG=en_US.UTF-8 \ LANGUAGE=en_US:en \ LC_ALL=en_US.UTF-8
63+
ENV TZ="Etc/UTC"
64+
RUN ln -fs "/usr/share/zoneinfo/${TZ}" /etc/localtime && \
65+
dpkg-reconfigure -f noninteractive tzdata
66+
67+
# Use root user
68+
ENV HOME /root
69+
WORKDIR "${HOME}"
70+
71+
# Install pyenv
72+
ENV PYENV_ROOT="${HOME}/.pyenv"
73+
RUN curl https://pyenv.run/ | /bin/bash
74+
ENV PATH="$PYENV_ROOT/bin:$PYENV_ROOT/shims:${PATH}"
75+
RUN echo 'eval "$(pyenv init -)"' >>$HOME/.bashrc && \
76+
pyenv update
77+
78+
# Install Python
79+
ARG PYTHON_VERSIONS="3.10 3.9 3.8 3.7 3.11 2.7 pypy2.7-7.3.11 pypy3.7"
80+
COPY --chown=1000:1000 --chmod=+x ./install-python.sh /tmp/install-python.sh
81+
COPY ./requirements.txt /requirements.txt
82+
RUN /tmp/install-python.sh && \
83+
rm /tmp/install-python.sh

‎.github/containers/Makefile‎

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
# Copyright 2010 New Relic, Inc.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
# Repository root for mounting into container.
16+
MAKEFILE_DIR:=$(dir $(realpath $(firstword $(MAKEFILE_LIST))))
17+
REPO_ROOT:=$(realpath $(MAKEFILE_DIR)../../)
18+
19+
.PHONY: default
20+
default: test
21+
22+
.PHONY: build
23+
build:
24+
@# Perform a shortened build for testing
25+
@docker build --build-arg='PYTHON_VERSIONS=3.10 2.7' $(MAKEFILE_DIR) -t ghcr.io/newrelic/newrelic-python-agent-ci:local
26+
27+
.PHONY: test
28+
test: build
29+
@# Ensure python versions are usable
30+
@docker run --rm ghcr.io/newrelic/python-agent-ci:local /bin/bash -c '\
31+
python3.10 --version && \
32+
python2.7 --version && \
33+
touch tox.ini && tox --version && \
34+
echo "Success! Python versions installed."'
35+
36+
.PHONY: run
37+
run: build
38+
@docker run --rm -it \
39+
--mount type=bind,source="$(REPO_ROOT)",target=/home/github/python-agent \
40+
--workdir=/home/github/python-agent \
41+
-e NEW_RELIC_HOST="${NEW_RELIC_HOST}" \
42+
-e NEW_RELIC_LICENSE_KEY="${NEW_RELIC_LICENSE_KEY}" \
43+
-e NEW_RELIC_DEVELOPER_MODE="${NEW_RELIC_DEVELOPER_MODE}" \
44+
ghcr.io/newrelic/newrelic-python-agent-ci:local /bin/bash

‎.github/containers/install-python.sh‎

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,56 @@
1+
#!/bin/bash
2+
# Copyright 2010 New Relic, Inc.
3+
#
4+
# Licensed under the Apache License, Version 2.0 (the "License");
5+
# you may not use this file except in compliance with the License.
6+
# You may obtain a copy of the License at
7+
#
8+
# http://www.apache.org/licenses/LICENSE-2.0
9+
#
10+
# Unless required by applicable law or agreed to in writing, software
11+
# distributed under the License is distributed on an "AS IS" BASIS,
12+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
# See the License for the specific language governing permissions and
14+
# limitations under the License.
15+
16+
set -e
17+
18+
SCRIPT_DIR=$(dirname "$0")
19+
PIP_REQUIREMENTS=$(cat /requirements.txt)
20+
21+
main() {
22+
# Coerce space separated string to array
23+
if [[ ${#PYTHON_VERSIONS[@]} -eq 1 ]]; then
24+
PYTHON_VERSIONS=($PYTHON_VERSIONS)
25+
fi
26+
27+
if [[ -z "${PYTHON_VERSIONS[@]}" ]]; then
28+
echo "No python versions specified. Make sure PYTHON_VERSIONS is set." 1>&2
29+
exit 1
30+
fi
31+
32+
# Find all latest pyenv supported versions for requested python versions
33+
PYENV_VERSIONS=()
34+
for v in "${PYTHON_VERSIONS[@]}"; do
35+
LATEST=$(pyenv latest -k "$v" || pyenv latest -k "$v-dev")
36+
if [[ -z "$LATEST" ]]; then
37+
echo "Latest version could not be found for ${v}." 1>&2
38+
exit 1
39+
fi
40+
PYENV_VERSIONS+=($LATEST)
41+
done
42+
43+
# Install each specific version
44+
for v in "${PYENV_VERSIONS[@]}"; do
45+
pyenv install "$v" &
46+
done
47+
wait
48+
49+
# Set all installed versions as globally accessible
50+
pyenv global ${PYENV_VERSIONS[@]}
51+
52+
# Install dependencies for main python installation
53+
pyenv exec pip install --upgrade $PIP_REQUIREMENTS
54+
}
55+
56+
main

‎.github/containers/requirements.txt‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
pip
2+
setuptools
3+
wheel
4+
virtualenv<20.22.0
5+
tox

‎.github/scripts/retry.sh‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,18 @@
11
#!/bin/bash
2+
# Copyright 2010 New Relic, Inc.
3+
#
4+
# Licensed under the Apache License, Version 2.0 (the "License");
5+
# you may not use this file except in compliance with the License.
6+
# You may obtain a copy of the License at
7+
#
8+
# http://www.apache.org/licenses/LICENSE-2.0
9+
#
10+
# Unless required by applicable law or agreed to in writing, software
11+
# distributed under the License is distributed on an "AS IS" BASIS,
12+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
# See the License for the specific language governing permissions and
14+
# limitations under the License.
15+
216

317
# Time in seconds to backoff after the initial attempt.
418
INITIAL_BACKOFF=10

‎.github/workflows/build-ci-image.yml‎

Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,68 @@
1+
# Copyright 2010 New Relic, Inc.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
name: Build CI Image
16+
17+
on:
18+
workflow_dispatch: # Allow manual trigger
19+
20+
concurrency:
21+
group: ${{ github.ref || github.run_id }}
22+
cancel-in-progress: true
23+
24+
jobs:
25+
build:
26+
runs-on: ubuntu-latest
27+
28+
steps:
29+
- uses: actions/checkout@v3
30+
with:
31+
persist-credentials: false
32+
fetch-depth: 0
33+
34+
- name: Set up Docker Buildx
35+
id: buildx
36+
uses: docker/setup-buildx-action@v2
37+
38+
- name: Generate Docker Metadata (Tags and Labels)
39+
id: meta
40+
uses: docker/metadata-action@v4
41+
with:
42+
images: ghcr.io/${{ github.repository }}-ci
43+
flavor: |
44+
prefix=
45+
suffix=
46+
latest=false
47+
tags: |
48+
type=raw,value=latest,enable={{is_default_branch}}
49+
type=schedule,pattern={{date 'YYYY-MM-DD'}}
50+
type=sha,format=short,prefix=sha-
51+
type=sha,format=long,prefix=sha-
52+
53+
- name: Login to GitHub Container Registry
54+
if: github.event_name != 'pull_request'
55+
uses: docker/login-action@v2
56+
with:
57+
registry: ghcr.io
58+
username: ${{ github.repository_owner }}
59+
password: ${{ secrets.GITHUB_TOKEN }}
60+
61+
- name: Build and Publish Image
62+
uses: docker/build-push-action@v3
63+
with:
64+
push: ${{ github.event_name != 'pull_request' }}
65+
context: .github/containers
66+
platforms: linux/amd64
67+
tags: ${{ steps.meta.outputs.tags }}
68+
labels: ${{ steps.meta.outputs.labels }}

‎.github/workflows/get-envs.py‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,18 @@
11
#!/usr/bin/env python3.8
2+
# Copyright 2010 New Relic, Inc.
3+
#
4+
# Licensed under the Apache License, Version 2.0 (the "License");
5+
# you may not use this file except in compliance with the License.
6+
# You may obtain a copy of the License at
7+
#
8+
# http://www.apache.org/licenses/LICENSE-2.0
9+
#
10+
# Unless required by applicable law or agreed to in writing, software
11+
# distributed under the License is distributed on an "AS IS" BASIS,
12+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
# See the License for the specific language governing permissions and
14+
# limitations under the License.
15+
216
import fileinput
317
import os
418

0 commit comments

Comments
 (0)