Skip to content

Update pip dependencies, fix conflicting dependency version, ensure there are no conflicting dependencies #4819

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 14 commits into from
Nov 20, 2019
Merged

Update pip dependencies, fix conflicting dependency version, ensure there are no conflicting dependencies #4819

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
c99c02b
Upgrade to cryptography 2.8 to avoid conflicting dependency.
Kami Nov 19, 2019
40827e7
Use latest versions of pip, setuptools and pbr when installing
Kami Nov 19, 2019
1cf03e1
Upgrade requests dependency to the latest version.
Kami Nov 19, 2019
d49bb0e
Make sure requirements.txt and test-requirements.txt are up to date.
Kami Nov 19, 2019
271e23c
Use latest version of jinja2 dependency.
Kami Nov 19, 2019
4ff0bbe
Update requirements make target so it removes any *egg-info directories
Kami Nov 19, 2019
b5a3afa
Run pip conflict checker as par of Make requirements step to ensure
Kami Nov 19, 2019
9c2ea48
Update more dependencies to latest stable versions.
Kami Nov 19, 2019
5979ae8
Revert breaking six upgrade.
Kami Nov 19, 2019
4274326
Add changelog entry.
Kami Nov 19, 2019
d2794aa
Revert "Revert breaking six upgrade."
Kami Nov 19, 2019
80c615f
Revert pymongo upgrade due to regression.
Kami Nov 19, 2019
a654cff
Update changelog.
Kami Nov 19, 2019
febde46
Use forks from StackStorm org.
Kami Nov 19, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,8 @@ Changed
* Remove automatic rendering of workflow output when updating task state for orquesta workflows.
This caused workflow output to render incorrectly in certain use case. The render_workflow_output
function must be called separately. (improvement)
* Update various internal dependencies to latest stable versions (cryptography, jinja2, requests,
apscheduler, eventlet, amqp, kombu, semver, six) #4819 (improvement)

Fixed
~~~~~
Expand Down
14 changes: 10 additions & 4 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -150,7 +150,6 @@ check: check-requirements flake8 checklogs
# make targets. This speeds up the build
.PHONY: install-runners
install-runners:

@echo ""
@echo "================== INSTALL RUNNERS ===================="
@echo ""
Expand Down Expand Up @@ -329,7 +328,7 @@ flake8: requirements .flake8
touch $(VIRTUALENV_ST2CLIENT_DIR)/bin/activate
chmod +x $(VIRTUALENV_ST2CLIENT_DIR)/bin/activate

$(VIRTUALENV_ST2CLIENT_DIR)/bin/pip install --upgrade "pip>=9.0,<9.1"
$(VIRTUALENV_ST2CLIENT_DIR)/bin/pip install --upgrade "pip==19.3.1"
# NOTE We need to upgrade setuptools to avoid bug with dependency resolving in old versions
$(VIRTUALENV_ST2CLIENT_DIR)/bin/pip install --upgrade "setuptools==41.0.1"
$(VIRTUALENV_ST2CLIENT_DIR)/bin/activate; cd st2client ; ../$(VIRTUALENV_ST2CLIENT_DIR)/bin/python setup.py install ; cd ..
Expand Down Expand Up @@ -440,12 +439,16 @@ requirements: virtualenv .sdist-requirements install-runners
@echo
# Make sure we use latest version of pip which is 19
$(VIRTUALENV_DIR)/bin/pip --version
$(VIRTUALENV_DIR)/bin/pip install --upgrade "pip>=19.0,<20.0"
$(VIRTUALENV_DIR)/bin/pip install --upgrade "virtualenv==16.6.0" # Required for packs.install in dev envs
$(VIRTUALENV_DIR)/bin/pip install --upgrade "pip==19.3.1"
$(VIRTUALENV_DIR)/bin/pip install --upgrade "setuptools==41.0.1" # Required for packs.install in dev envs
$(VIRTUALENV_DIR)/bin/pip install --upgrade "pbr==5.4.3" # workaround for pbr issue

# Generate all requirements to support current CI pipeline.
$(VIRTUALENV_DIR)/bin/python scripts/fixate-requirements.py --skip=virtualenv,virtualenv-osx -s st2*/in-requirements.txt contrib/runners/*/in-requirements.txt -f fixed-requirements.txt -o requirements.txt

# Remove any *.egg-info files which polute PYTHONPATH
rm -rf *.egg-info*
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should have done that before.

We add repo root to PYTHONPATH in virtualenv we create which means if you run python setup.py install or similar in st2client/ directory, it will create egg-info directory for that Python package which will mask actual st2client/ directory in PYTHONPATH.


# Generate finall requirements.txt file for each component
@for component in $(COMPONENTS_WITH_RUNNERS); do\
echo "==========================================================="; \
Expand Down Expand Up @@ -491,6 +494,9 @@ requirements: virtualenv .sdist-requirements install-runners
# Some of the tests rely on submodule so we need to make sure submodules are check out
git submodule update --recursive --remote

# Verify there are no conflicting dependencies
$(VIRTUALENV_DIR)/bin/pipconflictchecker

.PHONY: virtualenv
# Note: We always want to update virtualenv/bin/activate file to make sure
# PYTHONPATH is up to date and to avoid caching issues on Travis
Expand Down
2 changes: 1 addition & 1 deletion contrib/linux/requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
# used by file watcher sensor
pyinotify>=0.9.5,<=0.10
-e git+https://github.com/Kami/logshipper.git@stackstorm_patched#egg=logshipper
-e git+https://github.com/StackStorm/logshipper.git@stackstorm_patched#egg=logshipper
18 changes: 9 additions & 9 deletions fixed-requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,19 +3,19 @@
greenlet==0.4.15
# Note: 0.20.0 removed select.poll() on which some of our code and libraries we
# depend on rely
eventlet==0.25.0
eventlet==0.25.1
gunicorn==19.9.0
kombu==4.6.4
kombu==4.6.6
# Note: amqp is used by kombu
amqp==2.5.1
amqp==2.5.2
# NOTE: Recent version substantially affect the performance and add big import time overhead
# See https://github.com/StackStorm/st2/issues/4160#issuecomment-394386433 for details
oslo.config>=1.12.1,<1.13
oslo.utils>=3.36.2,<=3.37.0
six==1.12.0
six==1.13.0
pyyaml==5.1.2
requests[security]>=2.21.0,<2.22.0
apscheduler==3.6.1
requests[security]==2.22.0
apscheduler==3.6.3
gitpython==2.1.11
jsonschema==2.6.0
pymongo==3.7.2
Expand All @@ -25,13 +25,13 @@ lockfile==0.12.2
python-gnupg==0.4.5
jsonpath-rw==1.4.0
pyinotify==0.9.6
semver==2.8.1
semver==2.9.0
pytz==2019.1
stevedore==1.30.1
paramiko==2.6.0
networkx==1.11
python-keyczar==0.716
cryptography==2.7
cryptography==2.8
retrying==1.3.3
# Note: We use latest version of virtualenv which uses pip 19
virtualenv==16.6.0
Expand All @@ -56,4 +56,4 @@ mock==2.0.0
ujson==1.35
python-dateutil==2.8.0
bcrypt==3.1.7
jinja2==2.10.1
jinja2==2.10.3
20 changes: 10 additions & 10 deletions requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,25 +6,25 @@
# in-requirements.txt for that component and then run 'make requirements' to
# update the component requirements.txt
RandomWords
amqp==2.5.1
apscheduler==3.6.1
amqp==2.5.2
apscheduler==3.6.3
argcomplete
bcrypt==3.1.7
cryptography==2.7
eventlet==0.25.0
cryptography==2.8
eventlet==0.25.1
flex==6.14.0
git+https://github.com/Kami/logshipper.git@stackstorm_patched#egg=logshipper
git+https://github.com/StackStorm/logshipper.git@stackstorm_patched#egg=logshipper
git+https://github.com/StackStorm/orquesta.git@c80b139d8bbd7d59ccdc0461564e6a754066adaa#egg=orquesta
git+https://github.com/StackStorm/python-mistralclient.git#egg=python-mistralclient
git+https://github.com/StackStorm/st2-auth-backend-flat-file.git@master#egg=st2-auth-backend-flat-file
gitpython==2.1.11
greenlet==0.4.15
gunicorn==19.9.0
ipaddr
jinja2==2.10.1
jinja2==2.10.3
jsonpath-rw==1.4.0
jsonschema==2.6.0
kombu==4.6.4
kombu==4.6.6
lockfile==0.12.2
mock==2.0.0
mongoengine==0.18.2
Expand All @@ -51,11 +51,11 @@ pytz==2019.1
pywinrm==0.3.0
pyyaml==5.1.2
rednose
requests[security]<2.22.0,>=2.21.0
requests[security]==2.22.0
retrying==1.3.3
routes==2.4.1
semver==2.8.1
six==1.12.0
semver==2.9.0
six==1.13.0
sseclient-py==1.7
stevedore==1.30.1
tooz==1.66.1
Expand Down
2 changes: 1 addition & 1 deletion st2actions/in-requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,6 @@ gitpython
lockfile
# needed by core "linux" pack - TODO: create virtualenv for linux pack on postinst
pyinotify
git+https://github.com/Kami/logshipper.git@stackstorm_patched#egg=logshipper
git+https://github.com/StackStorm/logshipper.git@stackstorm_patched#egg=logshipper
# required by pack_mgmt/setup_virtualenv.py#L135
virtualenv
14 changes: 7 additions & 7 deletions st2actions/requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,19 +5,19 @@
# If you want to update depdencies for a single component, modify the
# in-requirements.txt for that component and then run 'make requirements' to
# update the component requirements.txt
apscheduler==3.6.1
eventlet==0.25.0
git+https://github.com/Kami/logshipper.git@stackstorm_patched#egg=logshipper
apscheduler==3.6.3
eventlet==0.25.1
git+https://github.com/StackStorm/logshipper.git@stackstorm_patched#egg=logshipper
git+https://github.com/StackStorm/python-mistralclient.git#egg=python-mistralclient
gitpython==2.1.11
jinja2==2.10.1
kombu==4.6.4
jinja2==2.10.3
kombu==4.6.6
lockfile==0.12.2
oslo.config<1.13,>=1.12.1
oslo.utils<=3.37.0,>=3.36.2
pyinotify==0.9.6
python-dateutil==2.8.0
python-json-logger
pyyaml==5.1.2
requests[security]<2.22.0,>=2.21.0
six==1.12.0
requests[security]==2.22.0
six==1.13.0
6 changes: 3 additions & 3 deletions st2api/requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,13 +5,13 @@
# If you want to update depdencies for a single component, modify the
# in-requirements.txt for that component and then run 'make requirements' to
# update the component requirements.txt
eventlet==0.25.0
eventlet==0.25.1
git+https://github.com/StackStorm/python-mistralclient#egg=python-mistralclient
gunicorn==19.9.0
jsonschema==2.6.0
kombu==4.6.4
kombu==4.6.6
mongoengine==0.18.2
oslo.config<1.13,>=1.12.1
oslo.utils<=3.37.0,>=3.36.2
pymongo==3.7.2
six==1.12.0
six==1.13.0
4 changes: 2 additions & 2 deletions st2auth/requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,11 +6,11 @@
# in-requirements.txt for that component and then run 'make requirements' to
# update the component requirements.txt
bcrypt==3.1.7
eventlet==0.25.0
eventlet==0.25.1
git+https://github.com/StackStorm/st2-auth-backend-flat-file.git@master#egg=st2-auth-backend-flat-file
gunicorn==19.9.0
oslo.config<1.13,>=1.12.1
passlib==1.7.1
pymongo==3.7.2
six==1.12.0
six==1.13.0
stevedore==1.30.1
6 changes: 3 additions & 3 deletions st2client/requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
# in-requirements.txt for that component and then run 'make requirements' to
# update the component requirements.txt
argcomplete
cryptography==2.7
cryptography==2.8
jsonpath-rw==1.4.0
jsonschema==2.6.0
prettytable
Expand All @@ -15,6 +15,6 @@ python-dateutil==2.8.0
python-editor==1.0.4
pytz==2019.1
pyyaml==5.1.2
requests[security]<2.22.0,>=2.21.0
six==1.12.0
requests[security]==2.22.0
six==1.13.0
sseclient-py==1.7
18 changes: 9 additions & 9 deletions st2common/requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,19 +5,19 @@
# If you want to update depdencies for a single component, modify the
# in-requirements.txt for that component and then run 'make requirements' to
# update the component requirements.txt
amqp==2.5.1
apscheduler==3.6.1
cryptography==2.7
eventlet==0.25.0
amqp==2.5.2
apscheduler==3.6.3
cryptography==2.8
eventlet==0.25.1
flex==6.14.0
git+https://github.com/StackStorm/orquesta.git@c80b139d8bbd7d59ccdc0461564e6a754066adaa#egg=orquesta
gitpython==2.1.11
greenlet==0.4.15
ipaddr
jinja2==2.10.1
jinja2==2.10.3
jsonpath-rw==1.4.0
jsonschema==2.6.0
kombu==4.6.4
kombu==4.6.6
lockfile==0.12.2
mongoengine==0.18.2
networkx==1.11
Expand All @@ -27,11 +27,11 @@ pymongo==3.7.2
python-dateutil==2.8.0
python-statsd==2.1.0
pyyaml==5.1.2
requests[security]<2.22.0,>=2.21.0
requests[security]==2.22.0
retrying==1.3.3
routes==2.4.1
semver==2.8.1
six==1.12.0
semver==2.9.0
six==1.13.0
tooz==1.66.1
ujson==1.35
webob==1.8.5
Expand Down
6 changes: 3 additions & 3 deletions st2debug/requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,8 @@
# If you want to update depdencies for a single component, modify the
# in-requirements.txt for that component and then run 'make requirements' to
# update the component requirements.txt
eventlet==0.25.0
eventlet==0.25.1
python-gnupg==0.4.5
pyyaml==5.1.2
requests[security]<2.22.0,>=2.21.0
six==1.12.0
requests[security]==2.22.0
six==1.13.0
6 changes: 3 additions & 3 deletions st2exporter/requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
# If you want to update depdencies for a single component, modify the
# in-requirements.txt for that component and then run 'make requirements' to
# update the component requirements.txt
eventlet==0.25.0
kombu==4.6.4
eventlet==0.25.1
kombu==4.6.6
oslo.config<1.13,>=1.12.1
six==1.12.0
six==1.13.0
8 changes: 4 additions & 4 deletions st2reactor/requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,11 +5,11 @@
# If you want to update depdencies for a single component, modify the
# in-requirements.txt for that component and then run 'make requirements' to
# update the component requirements.txt
apscheduler==3.6.1
eventlet==0.25.0
apscheduler==3.6.3
eventlet==0.25.1
jsonpath-rw==1.4.0
jsonschema==2.6.0
kombu==4.6.4
kombu==4.6.6
oslo.config<1.13,>=1.12.1
python-dateutil==2.8.0
six==1.12.0
six==1.13.0
6 changes: 3 additions & 3 deletions st2stream/requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,12 +5,12 @@
# If you want to update depdencies for a single component, modify the
# in-requirements.txt for that component and then run 'make requirements' to
# update the component requirements.txt
eventlet==0.25.0
eventlet==0.25.1
gunicorn==19.9.0
jsonschema==2.6.0
kombu==4.6.4
kombu==4.6.6
mongoengine==0.18.2
oslo.config<1.13,>=1.12.1
oslo.utils<=3.37.0,>=3.36.2
pymongo==3.7.2
six==1.12.0
six==1.13.0
7 changes: 4 additions & 3 deletions test-requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,16 +19,17 @@ nose-timer==0.7.5
# splitting tests run on a separate CI machines
nose-parallel==0.3.1
# Required by st2client tests
pyyaml==5.1
pyyaml==5.1.2
RandomWords
gunicorn==19.9.0
psutil==5.6.1
psutil==5.6.3
webtest==2.0.25
rstcheck>=3.3.1,<3.4
tox==3.13.2
tox==3.14.1
pyrabbit
# Since StackStorm v2.8.0 we now use cryptography instead of keyczar, but we still have some tests
# which utilize keyczar and ensure new cryptography code is fully compatible with keyczar code
# (those tests only run under Python 2.7 since keyczar doesn't support Python 3.x).
# See https://github.com/StackStorm/st2/pull/4165
python-keyczar
git+https://github.com/StackStorm/pip-conflict-checker.git@fix_pip_issue#egg=pip-conflict-checker