Skip to content

Added option for st2packs-PersistentVolumes to allow for seamless pack updates #160

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 1 commit into from
Closed

Added option for st2packs-PersistentVolumes to allow for seamless pack updates #160

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# Changelog

## In Development

* Add option for st2packs-PersistentVolumes to allow for seamless custom pack-updates without job-impact (#160) (by @moonrail)

## v0.40.0
* Switch st2 version to `v3.4dev` as a new latest development version (#157)
Expand Down
30 changes: 29 additions & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -182,7 +182,7 @@ As any other Helm dependency, it's possible to further configure it for specific
## Install custom st2 packs in the cluster
In distributed environment of the Kubernetes cluster `st2 pack install` won’t work.
Instead, you need to bake the packs into a custom docker image, push it to a private or public docker registry and reference that image in Helm values.
Helm chart will take it from there, sharing `/opt/stackstorm/{packs,virtualenvs}` via a sidecar container in pods which require access to the packs.
Helm chart will take it from there, sharing `/opt/stackstorm/{packs,virtualenvs}` via a sidecar container or alternatively shared PersistentVolume in pods which require access to the packs.

### Building st2packs image
For your convenience, we created a new `st2-pack-install <pack1> <pack2> <pack3>` utility and included it in a container that will help to install custom packs during the Docker build process without relying on live DB and MQ connection.
Expand Down Expand Up @@ -213,6 +213,34 @@ kubectl create secret docker-registry st2packs-auth --docker-server=<your-regist
```
Once secret created, reference its name in helm value: `st2.packs.image.pullSecret`.

### Share custom st2packs via PersistentVolume
By using a PersistentVolume instead of sidecar container (the default) for custom st2packs, the actionrunner, sensor & api-containers will not be restarted on redeploying st2packs oder config maps.

For this you have to uncomment following section in values.yaml and provide your specific requirements:
```yaml
st2:
packs:
persistentVolumes:
packs:
resources:
requests:
# If your packs bring a lot of files, this value may require a raise
storage: 512Mi
# If your kubernetes cluster does not have a default storageClass set, you can provide a specific one here
# storageClassName: 'your_provider'
venvs:
resources:
requests:
# If your packs require a lot of packages in their virtual environments, this value may require a raise
storage: 1Gi
# If your kubernetes cluster does not have a default storageClass set, you can provide a specific one here
# storageClassName: 'your_provider'
```
Don't forget running Helm upgrade to apply new changes.

The only container, that will be replaced on upgrade, is `job-st2-register-content`.

This container will update Packs & VirtualEnvironments in PersistentVolumes & register them and any given pack configs in StackStorm, without any impact on running executions.

## Tips & Tricks
Grab all logs for entire StackStorm cluster with dependent services in Helm release:
Expand Down
54 changes: 54 additions & 0 deletions templates/_helpers.tpl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,3 +71,57 @@ Create the name of the stackstorm-ha service account to use
{{ $mongo_fullname }}-{{ $index0 }}.{{ $mongo_fullname }}{{ if ne $index1 $replicas }},{{ end }}
{{- end -}}
{{- end -}}

# For custom st2packs-Container reduce duplicity by defining it here once
{{- define "packs-volumes" -}}
{{- if .Values.st2.packs.image.repository }}
{{- if .Values.st2.packs.persistentVolumes }}
- name: st2-packs-vol
persistentVolumeClaim:
claimName: 'st2-packs-pvol'
- name: st2-virtualenvs-vol
persistentVolumeClaim:
claimName: 'st2-venv-pvol'
{{- else }}
- name: st2-packs-vol
emptyDir: {}
- name: st2-virtualenvs-vol
emptyDir: {}
{{- end }}
{{- end }}
{{- end -}}

# For custom st2packs-initContainers reduce duplicity by defining them here once
{{- define "packs-initContainers" -}}
# Merge packs and virtualenvs from st2api with those from the st2.packs image
# Custom packs
- name: st2-custom-packs
image: "{{ .Values.st2.packs.image.repository }}/{{ .Values.st2.packs.image.name }}:{{ .Values.st2.packs.image.tag }}"
imagePullPolicy: {{ .Values.st2.packs.image.pullPolicy | quote }}
volumeMounts:
- name: st2-packs-vol
mountPath: /opt/stackstorm/packs-shared
- name: st2-virtualenvs-vol
mountPath: /opt/stackstorm/virtualenvs-shared
command:
- 'sh'
- '-ec'
- |
/bin/cp -aR /opt/stackstorm/packs/. /opt/stackstorm/packs-shared &&
/bin/cp -aR /opt/stackstorm/virtualenvs/. /opt/stackstorm/virtualenvs-shared
# System packs
- name: st2-system-packs
image: "{{ template "imageRepository" . }}/st2actionrunner{{ template "enterpriseSuffix" . }}:{{ .Chart.AppVersion }}"
imagePullPolicy: {{ .Values.image.pullPolicy }}
volumeMounts:
- name: st2-packs-vol
mountPath: /opt/stackstorm/packs-shared
- name: st2-virtualenvs-vol
mountPath: /opt/stackstorm/virtualenvs-shared
command:
- 'sh'
- '-ec'
- |
/bin/cp -aR /opt/stackstorm/packs/. /opt/stackstorm/packs-shared &&
/bin/cp -aR /opt/stackstorm/virtualenvs/. /opt/stackstorm/virtualenvs-shared
{{- end -}}
150 changes: 12 additions & 138 deletions templates/deployments.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -158,39 +158,9 @@ spec:
{{- if .Values.image.pullSecret }}
- name: {{ .Values.image.pullSecret }}
{{- end }}
{{- if .Values.st2.packs.image.repository }}
{{- if and .Values.st2.packs.image.repository (not .Values.st2.packs.persistentVolumes) }}
initContainers:
# Merge packs and virtualenvs from st2api with those from the st2.packs image
# Custom packs
- name: st2-custom-packs
image: "{{ .Values.st2.packs.image.repository }}/{{ .Values.st2.packs.image.name }}:{{ .Values.st2.packs.image.tag }}"
imagePullPolicy: {{ .Values.st2.packs.image.pullPolicy | quote }}
volumeMounts:
- name: st2-packs-vol
mountPath: /opt/stackstorm/packs-shared
- name: st2-virtualenvs-vol
mountPath: /opt/stackstorm/virtualenvs-shared
command:
- 'sh'
- '-ec'
- |
/bin/cp -aR /opt/stackstorm/packs/. /opt/stackstorm/packs-shared &&
/bin/cp -aR /opt/stackstorm/virtualenvs/. /opt/stackstorm/virtualenvs-shared
# System packs
- name: st2-system-packs
image: "{{ template "imageRepository" . }}/st2actionrunner{{ template "enterpriseSuffix" . }}:{{ .Chart.AppVersion }}"
imagePullPolicy: {{ .Values.image.pullPolicy }}
volumeMounts:
- name: st2-packs-vol
mountPath: /opt/stackstorm/packs-shared
- name: st2-virtualenvs-vol
mountPath: /opt/stackstorm/virtualenvs-shared
command:
- 'sh'
- '-ec'
- |
/bin/cp -aR /opt/stackstorm/packs/. /opt/stackstorm/packs-shared &&
/bin/cp -aR /opt/stackstorm/virtualenvs/. /opt/stackstorm/virtualenvs-shared
{{- include "packs-initContainers" . | indent 6 }}
{{- end }}
containers:
- name: st2api{{ template "enterpriseSuffix" . }}
Expand Down Expand Up @@ -242,10 +212,7 @@ spec:
configMap:
name: {{ .Release.Name }}-st2-config
{{- if .Values.st2.packs.image.repository }}
- name: st2-packs-vol
emptyDir: {}
- name: st2-virtualenvs-vol
emptyDir: {}
{{- include "packs-volumes" . | indent 8 }}
{{- end }}
{{- with .Values.st2api.nodeSelector }}
nodeSelector:
Expand Down Expand Up @@ -923,37 +890,9 @@ spec:
{{- if $.Values.image.pullSecret }}
- name: {{ $.Values.image.pullSecret }}
{{- end }}
{{- if $.Values.st2.packs.image.repository }}
{{- if and $.Values.st2.packs.image.repository (not $.Values.st2.packs.persistentVolumes) }}
initContainers:
# Merge packs and virtualenvs from st2sensorcontainer with those from the st2.packs image
# Custom packs
- name: st2-custom-packs
image: "{{ $.Values.st2.packs.image.repository }}/{{ $.Values.st2.packs.image.name }}:{{ $.Values.st2.packs.image.tag }}"
imagePullPolicy: {{ $.Values.st2.packs.image.pullPolicy | quote }}
volumeMounts:
- name: st2-packs-vol
mountPath: /opt/stackstorm/packs-shared
- name: st2-virtualenvs-vol
mountPath: /opt/stackstorm/virtualenvs-shared
command:
- 'sh'
- '-ec'
- |
/bin/cp -aR /opt/stackstorm/packs/. /opt/stackstorm/packs-shared &&
/bin/cp -aR /opt/stackstorm/virtualenvs/. /opt/stackstorm/virtualenvs-shared
# System packs
- name: st2-system-packs
image: "{{ template "imageRepository" $ }}/st2actionrunner{{ template "enterpriseSuffix" $ }}:{{ $.Chart.AppVersion }}"
imagePullPolicy: {{ $.Values.image.pullPolicy }}
volumeMounts:
- name: st2-packs-vol
mountPath: /opt/stackstorm/packs-shared
command:
- 'sh'
- '-ec'
- |
/bin/cp -aR /opt/stackstorm/packs/. /opt/stackstorm/packs-shared &&
/bin/cp -aR /opt/stackstorm/virtualenvs/. /opt/stackstorm/virtualenvs-shared
{{- include "packs-initContainers" $ | indent 6 }}
{{- end }}
containers:
- name: st2sensorcontainer{{ template "hyphenPrefix" .name }}{{ template "enterpriseSuffix" $ }}
Expand Down Expand Up @@ -1018,10 +957,7 @@ spec:
configMap:
name: {{ $.Release.Name }}-st2-config
{{- if $.Values.st2.packs.image.repository }}
- name: st2-packs-vol
emptyDir: {}
- name: st2-virtualenvs-vol
emptyDir: {}
{{- include "packs-volumes" $ | indent 8 }}
{{- end }}
{{- with .nodeSelector }}
nodeSelector:
Expand Down Expand Up @@ -1092,37 +1028,9 @@ spec:
{{- if .Values.image.pullSecret }}
- name: {{ .Values.image.pullSecret }}
{{- end }}
{{- if .Values.st2.packs.image.repository }}
{{- if and .Values.st2.packs.image.repository (not .Values.st2.packs.persistentVolumes) }}
initContainers:
# Merge packs and virtualenvs from st2actionrunner with those from the st2.packs image
# Custom packs
- name: st2-custom-packs
image: "{{ .Values.st2.packs.image.repository }}/{{ .Values.st2.packs.image.name }}:{{ .Values.st2.packs.image.tag }}"
imagePullPolicy: {{ .Values.st2.packs.image.pullPolicy | quote }}
volumeMounts:
- name: st2-packs-vol
mountPath: /opt/stackstorm/packs-shared
- name: st2-virtualenvs-vol
mountPath: /opt/stackstorm/virtualenvs-shared
command:
- 'sh'
- '-ec'
- |
/bin/cp -aR /opt/stackstorm/packs/. /opt/stackstorm/packs-shared &&
/bin/cp -aR /opt/stackstorm/virtualenvs/. /opt/stackstorm/virtualenvs-shared
# System packs
- name: st2-system-packs
image: "{{ template "imageRepository" . }}/st2actionrunner{{ template "enterpriseSuffix" . }}:{{ .Chart.AppVersion }}"
imagePullPolicy: {{ .Values.image.pullPolicy }}
volumeMounts:
- name: st2-packs-vol
mountPath: /opt/stackstorm/packs-shared
command:
- 'sh'
- '-ec'
- |
/bin/cp -aR /opt/stackstorm/packs/. /opt/stackstorm/packs-shared &&
/bin/cp -aR /opt/stackstorm/virtualenvs/. /opt/stackstorm/virtualenvs-shared
{{- include "packs-initContainers" . | indent 6 }}
{{- end }}
containers:
- name: st2actionrunner{{ template "enterpriseSuffix" . }}
Expand Down Expand Up @@ -1183,10 +1091,7 @@ spec:
# 0400 file permission
mode: 256
{{- if .Values.st2.packs.image.repository }}
- name: st2-packs-vol
emptyDir: {}
- name: st2-virtualenvs-vol
emptyDir: {}
{{- include "packs-volumes" . | indent 8 }}
{{- end }}
{{- with .Values.st2actionrunner.nodeSelector }}
nodeSelector:
Expand Down Expand Up @@ -1333,36 +1238,8 @@ spec:
- name: {{ .Values.image.pullSecret }}
{{- end }}
initContainers:
{{- if .Values.st2.packs.image.repository }}
# Merge packs and virtualenvs from st2actionrunner with those from the st2.packs image
# Custom packs
- name: st2-custom-packs
image: "{{ .Values.st2.packs.image.repository }}/{{ .Values.st2.packs.image.name }}:{{ .Values.st2.packs.image.tag }}"
imagePullPolicy: {{ .Values.st2.packs.image.pullPolicy | quote }}
volumeMounts:
- name: st2-packs-vol
mountPath: /opt/stackstorm/packs-shared
- name: st2-virtualenvs-vol
mountPath: /opt/stackstorm/virtualenvs-shared
command:
- 'sh'
- '-ec'
- |
/bin/cp -aR /opt/stackstorm/packs/. /opt/stackstorm/packs-shared &&
/bin/cp -aR /opt/stackstorm/virtualenvs/. /opt/stackstorm/virtualenvs-shared
# System packs
- name: st2-system-packs
image: "{{ template "imageRepository" . }}/st2actionrunner{{ template "enterpriseSuffix" . }}:{{ .Chart.AppVersion }}"
imagePullPolicy: {{ .Values.image.pullPolicy }}
volumeMounts:
- name: st2-packs-vol
mountPath: /opt/stackstorm/packs-shared
command:
- 'sh'
- '-ec'
- |
/bin/cp -aR /opt/stackstorm/packs/. /opt/stackstorm/packs-shared &&
/bin/cp -aR /opt/stackstorm/virtualenvs/. /opt/stackstorm/virtualenvs-shared
{{- if and .Values.st2.packs.image.repository (not .Values.st2.packs.persistentVolumes) }}
{{- include "packs-initContainers" . | indent 6 }}
{{- end }}
# Sidecar container for generating st2client config with st2 username & password pair and sharing produced file with the main container
- name: generate-st2client-config
Expand Down Expand Up @@ -1486,10 +1363,7 @@ spec:
# 0400 file permission
mode: 256
{{- if .Values.st2.packs.image.repository }}
- name: st2-packs-vol
emptyDir: {}
- name: st2-virtualenvs-vol
emptyDir: {}
{{- include "packs-volumes" . | indent 8 }}
{{- end }}

{{ if .Values.st2chatops.enabled -}}
Expand Down
7 changes: 1 addition & 6 deletions templates/jobs.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -403,10 +403,5 @@ spec:
- name: st2-pack-configs-vol
configMap:
name: {{ .Release.Name }}-st2-pack-configs
{{- if .Values.st2.packs.image.repository }}
- name: st2-packs-vol
emptyDir: {}
- name: st2-virtualenvs-vol
emptyDir: {}
{{- end }}
{{- include "packs-volumes" $ | indent 8}}
restartPolicy: OnFailure
32 changes: 32 additions & 0 deletions templates/persistentvolumeclaims.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
# This is only used, when user wants to share packs & venvs via persistentVolumes
{{- if and .Values.st2.packs.image.repository .Values.st2.packs.persistentVolumes }}
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: st2-packs-pvol
spec:
accessModes:
- ReadWriteMany
volumeMode: Filesystem
resources:
{{ toYaml .Values.st2.packs.persistentVolumes.packs.resources | indent 4 }}
{{- if .Values.st2.packs.persistentVolumes.packs.storageClassName }}
storageClassName: "{{ .Values.st2.packs.persistentVolumes.packs.storageClassName }}"
{{- end }}

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: st2-venv-pvol
spec:
accessModes:
- ReadWriteMany
Copy link
Member

@arm4b arm4b Nov 25, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, so that looks like the main point behind the implementation.
ReadWriteMany (https://kubernetes.io/docs/concepts/storage/persistent-volumes/#access-modes) means that some kind of network-like filesystem is still required for this to work. Does it mean it's a new possible point of failure that may affect overall availability formula?

I guess it's a hybrid between the current immutable containerized pack content approach vs mutable/dynamic way demonstrated in #118 when packs are installed in-place.

Copy link
Member

@arm4b arm4b Nov 25, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Considering it's ReadWriteMany volume shared between all the Pods that need it, do you think live pack install may work in such environment as well?

I mean getting "hybrid" best from the 2 approaches when user can have all the options and satisfy all the needs:

  1. Installing the packs in-place with st2 pack install and from the UI (easiest & best experience for the majority of users - can solve a lot of community pain points as a default solution)
  2. Optionally relying on pre-built docker images for packs (best for deployment, repeatability, versioning, availability?)
  3. Can do 1 or 2 or both.

WDYT?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Exactly - this approach requires a storageClass supporting ReadWriteMany - this should be added to README.md as well.

Hmmm, you're right - why going the container-direction on packs, that hinders from using API & UI-Pack-Handling?
Well, its the k8s way, the containerized way.
You enforce IaaC, that users strictly define their packs, versions, e.g. via code.
If you allow changes on the fly, via API/UI, there would be a "split-brain" between your code and actual installed & configured packs.

Our PR here is only aimed to improve resilience and availability by not modifying and re-creating System-Important-Containers for Packs. This leads to interrupts in the Service.

Concerning your described Approaches:
1.
Would lead to inconsistencies, but would allow to use API & UI to install/update/remove Packs.

Slower approach for sure, but more resilient as long as users are hindered from using API & UI to modify/add Packs-Contents.
Scales better for (internal) Self-Service-Deployments with several Stakeholders/Departments (Our case).

Would be somewhat OK, as long as one only handles Custom Packs (those not on Exchange) via Containers, everything else via API & UI.
I do not see how StackStorm could properly distinct between "containerized" Packs and "modifyiable" Packs for API & UI.
So this would lead to inconsistencies.

Our & My opinion:
I think there is no right or wrong answer to it.

For me personally I was confused when I saw how StackStorm handles Packs (Enterprise-UI and API), in combination with Git-Packs.
The user is able to modify Actions (Workflows) and Rules via API & UI for Git-Packs, but those will be discarded the moment, the user updates the Pack from Git.
I do see problems arising from this, so I've already planned, how I could limit Users via RBAC to not handle any resources that reside in Packs.

I would stick to building containers and would like to be able to configure an option in StackStorm itself for hindering/locking API & UI-Pack-(Un-)Installations, or Modifications to Actions, Rules altogether.
This way Packs would always be consistent with external Versions and Users could be shown a proper error message. Right now the error messages are not optimal due to simply setting Filesystem to ReadOnly :)
So either Approach 1 or 2, but not both and certainly not FreeForAll via Option 3, from my point of view.

volumeMode: Filesystem
resources:
{{ toYaml .Values.st2.packs.persistentVolumes.venvs.resources | indent 4 }}
{{- if .Values.st2.packs.persistentVolumes.venvs.storageClassName }}
storageClassName: "{{ .Values.st2.packs.persistentVolumes.venvs.storageClassName }}"
{{- end }}
{{- end }}
22 changes: 20 additions & 2 deletions values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -96,13 +96,31 @@ st2:
---
# example core pack config yaml

# Custom packs are shared via two persistent volumes across all containers
# Here you can customize these volumes, to accomodate to your specific requirements
# persistentVolumes:
# packs:
# resources:
# requests:
# If your packs bring a lot of files, this value may require a raise
# storage: 512Mi
# If your kubernetes cluster does not have a default storageClass set, you can provide a specific one here
# storageClassName: 'your_provider'
# venvs:
# resources:
# requests:
# If your packs require a lot of packages in their virtual environments, this value may require a raise
# storage: 1Gi
# If your kubernetes cluster does not have a default storageClass set, you can provide a specific one here
# storageClassName: 'your_provider'

# Custom packs image settings. The repository, name, tag and pullPolicy for this image
# are specified below.
image:
# Uncomment the following block to make the custom packs image available to the necessary pods
#repository: your-remote-docker-registry.io
# repository: index.docker.io/stackstorm
name: st2packs
tag: latest
tag: example
pullPolicy: IfNotPresent
# Optional name of the imagePullSecret if your custom packs image is hosted by a private Docker registry behind the auth
#pullSecret: st2packs-auth
Expand Down