Enhance remove S/W support

[noelmcloughlin]

The current formula has undocumented dropped state. I would like better support for the REMOVE Use Case. So this PR (replaces #182) aims to improve that UC. Comments welcomed.

CHANGELOG

1. Remove states documented in README
2. New pillars added to pillar.example and README
3. dropped.sls is now meta-state.
4. postgres.server.remove state introduced to replace dropped.sls
   • Standard UC: remove version
   • Protect data as default (postgres.remove.data: False).
   • Extended UC: remove multiple versions (postgres.remove.multiple_releases: True)
   • Extended UC: remove linux alternatives.
5. postgres.client.remove
   - Standard UC (remove one version) and Extended UC (multiple releases).
6. postgres.dev.remove
   - Standard UC (remove one version) and Extended UC (multiple releases).
   - Handle python_pkg removal too.
7. Added dev/ directory.

We could consider moving dev.sls and python.sls to this dev directory.

Verification

top.sls
base:
'*':
- postgres.dropped
- postgres

pillars

  #following is used by 'remove' states.
  remove:
    data: True   #(default False)
    multiple_releases: True   #(default False)
    releases: ['9.2', '9.3', '9.4', '9.5', '9.6', '10',]    #(default shown) used when multiple_releases=True

scenarios (+ linux alternatives):

• use_upstream_repo: False
• use_upstream_repp: True and version: '9.6'

Targets

• Ubuntu 16,
• Fedora 27,
• Centos 7.

The verification looks good. Here is summary of the high.state results - with failures noted.

UBUNTU 16: use_upstream_repo=False

    Warnings: 'extensions' and 'schemas' are invalid keyword arguments for
              'postgres_database.present'. If you were trying to pass additional
              data to be used in a template context, please populate 'context'
              with 'key: value' pairs. Your approach will work until Salt
              Fluorine is out. Please update your state files.

Succeeded: 74 (changed=24)
Failed:     0
Warnings:   1


FEDORA 27 use_upstream_repo=False


[WARNING ] The function "module.run" is using its deprecated version and will expire in version "Sodium".
[ERROR   ] Exception raised when processing __virtual__ function for salt.loaded.int.utils.boto. Module will not be loaded: 'module' object has no attribute '__version__'
[WARNING ] salt.loaded.int.utils.boto.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'boto', please fix this.
[ERROR   ] Exception raised when processing __virtual__ function for salt.loaded.int.utils.boto3. Module will not be loaded: 'module' object has no attribute '__version__'
[WARNING ] salt.loaded.int.utils.boto3.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'boto3', please fix this.
[ERROR   ] The named service postgresql is not available
[ERROR   ] Command '[u'/bin/psql', u'--no-align', u'--no-readline', u'--no-psqlrc', u'--no-password', u'--dbname', u'postgres', u'-c', u'CREATE TABLESPACE "my_space" OWNER "localUser" LOCATION \'/srv/my_tablespace\' ']' failed with return code: 1
[ERROR   ] stderr: ERROR:  could not set permissions on directory "/srv/my_tablespace": Permission denied
[ERROR   ] retcode: 1
[ERROR   ] Error connecting to Postgresql server
[ERROR   ] An exception occurred in this state: Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/salt/state.py", line 1905, in call
    **cdata['kwargs'])
  File "/usr/lib/python2.7/site-packages/salt/loader.py", line 1830, in wrapper
    return f(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/salt/states/postgres_tablespace.py", line 121, in present
    if tblspaces[name]['Location'] != directory and not __opts__['test']:
KeyError: u'my_space'

[ERROR   ] Failed to install extension uuid-ossp

Succeeded: 64 (changed=16)
Failed:     3



FEDORA 27: use_upstream_repo: True,  version: 9.6

$ sudo salt-call state.highstate --local
[ERROR   ] Exception raised when processing __virtual__ function for salt.loaded.int.utils.boto. Module will not be loaded: 'module' object has no attribute '__version__'
[WARNING ] salt.loaded.int.utils.boto.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'boto', please fix this.
[ERROR   ] Exception raised when processing __virtual__ function for salt.loaded.int.utils.boto3. Module will not be loaded: 'module' object has no attribute '__version__'
[WARNING ] salt.loaded.int.utils.boto3.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'boto3', please fix this.
[ERROR   ] alternative: pg_resetxlog does not exist
[ERROR   ] alternative: postgresql96-check-db-dir does not exist
[ERROR   ] alternative: postgresql96-setup does not exist
[ERROR   ] Exception raised when processing __virtual__ function for  BOTO BOTO BOTO ETC BLA BLA

[ERROR   ] alternative: initdb does not exist
[ERROR   ] alternative: pg_controldata does not exist
[ERROR   ] alternative: pg_ctl does not exist
[ERROR   ] alternative: postgres does not exist
[ERROR   ] alternative: postmaster does not exist
[ERROR   ] alternative: createlang does not exist
[ERROR   ] alternative: droplang does not exist
[ERROR   ] alternative: pg_receivexlog does not exist
[ERROR   ] alternative: pg_xlogdump does not exist
[ERROR   ] alternative: clusterdb does not exist
[ERROR   ] alternative: createdb does not exist
[ERROR   ] alternative: createuser does not exist
[ERROR   ] alternative: dropdb does not exist
[ERROR   ] alternative: dropuser does not exist
[ERROR   ] alternative: pg_archivecleanup does not exist
[ERROR   ] alternative: pg_basebackup does not exist
[ERROR   ] alternative: pg_config does not exist
[ERROR   ] alternative: pg_dump does not exist
[ERROR   ] alternative: pg_dumpall does not exist
[ERROR   ] alternative: pg_isready does not exist
[ERROR   ] alternative: pg_restore does not exist
[ERROR   ] alternative: pg_rewind does not exist
[ERROR   ] alternative: pg_test_fsync does not exist
[ERROR   ] alternative: pg_test_timing does not exist
[ERROR   ] alternative: pg_upgrade does not exist
[ERROR   ] alternative: pgbench does not exist
[ERROR   ] alternative: psql does not exist
[ERROR   ] alternative: reindexdb does not exist
[ERROR   ] alternative: vacuumdb does not exist
[ERROR   ] alternative: ecg does not exist
[ERROR   ] Exception raised when processing __virtual__ function for BOTO BOTO BOTO ETC..

[WARNING ] The function "module.run" is using its deprecated version and will expire in version "Sodium".
[ERROR   ] The named service postgresql-9.6 is not available
[ERROR   ] Failed to install extension uuid-ossp

    Warnings: 'extensions' and 'schemas' are invalid keyword arguments for
              'postgres_database.present'. If you were trying to pass additional
              data to be used in a template context, please populate 'context'
              with 'key: value' pairs. Your approach will work until Salt
              Fluorine is out. Please update your state files.


Succeeded: 297 (changed=108)
Failed:      1
Warnings:    1



UBUNTU: use_upstream_repo: True     version: 9.6

[ERROR   ] {'removed': {}, 'installed': {}}
[WARNING ] The function "module.run" is using its deprecated version and will expire in version "Sodium".

          ID: postgresql10-server-pkgs-removed
    Function: pkg.purged
      Result: False
     Comment: The following packages failed to purge: postgresql-contrib-10.
     Started: 15:21:04.912864
    Duration: 1411.552 ms
     Changes:   
              ----------
              installed:
                  ----------
              removed:
                  ----------

Succeeded: 74 (changed=25)
Failed:     1
Warnings:   1


CENTOS7: use_upstream_rep: False

$ sudo salt-call state.highstate --local
[WARNING ] The function "module.run" is using its deprecated version and will expire in version "Sodium".
[ERROR   ] The named service postgresql is not available
[ERROR   ] Command '['/bin/psql', '--no-align', '--no-readline', '--no-psqlrc', '--no-password', '--dbname', 'postgres', '-c', 'CREATE TABLESPACE "my_space" OWNER "localUser" LOCATION \'/srv/my_tablespace\' ']' failed with return code: 1
[ERROR   ] stderr: ERROR:  could not set permissions on directory "/srv/my_tablespace": Permission denied
[ERROR   ] retcode: 1
[ERROR   ] Error connecting to Postgresql server
[ERROR   ] An exception occurred in this state: Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/salt/state.py", line 1851, in call
    **cdata['kwargs'])
  File "/usr/lib/python2.7/site-packages/salt/loader.py", line 1795, in wrapper
    return f(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/salt/states/postgres_tablespace.py", line 121, in present
    if tblspaces[name]['Location'] != directory and not __opts__['test']:
KeyError: 'my_space'

[ERROR   ] Failed to install extension uuid-ossp


Succeeded: 64 (changed=16)
Failed:     3
-------------

CENTOS7: use_upstream_repo: True   version: 9.6


$ sudo salt-call state.highstate --local
[ERROR   ] alternative: pg_resetxlog does not exist
[ERROR   ] alternative: postgresql96-check-db-dir does not exist
[ERROR   ] alternative: postgresql96-setup does not exist
[ERROR   ] alternative: initdb does not exist
[ERROR   ] alternative: pg_controldata does not exist
[ERROR   ] alternative: pg_ctl does not exist
[ERROR   ] alternative: postgres does not exist
[ERROR   ] alternative: postmaster does not exist
[ERROR   ] alternative: clusterdb does not exist
[ERROR   ] alternative: createdb does not exist
[ERROR   ] alternative: createlang does not exist
[ERROR   ] alternative: createuser does not exist
[ERROR   ] alternative: dropdb does not exist
[ERROR   ] alternative: droplang does not exist
[ERROR   ] alternative: dropuser does not exist
[ERROR   ] alternative: pg_archivecleanup does not exist
[ERROR   ] alternative: pg_basebackup does not exist
[ERROR   ] alternative: pg_config does not exist
[ERROR   ] alternative: pg_dump does not exist
[ERROR   ] alternative: pg_dumpall does not exist
[ERROR   ] alternative: pg_isready does not exist
[ERROR   ] alternative: pg_receivexlog does not exist
[ERROR   ] alternative: pg_restore does not exist
[ERROR   ] alternative: pg_rewind does not exist
[ERROR   ] alternative: pg_test_fsync does not exist
[ERROR   ] alternative: pg_test_timing does not exist
[ERROR   ] alternative: pg_upgrade does not exist
[ERROR   ] alternative: pg_xlogdump does not exist
[ERROR   ] alternative: pgbench does not exist
[ERROR   ] alternative: psql does not exist
[ERROR   ] alternative: reindexdb does not exist
[ERROR   ] alternative: vacuumdb does not exist
[ERROR   ] alternative: ecg does not exist
[WARNING ] The function "module.run" is using its deprecated version and will expire in version "Sodium".
[ERROR   ] The named service postgresql-9.6 is not available
[ERROR   ] Failed to install extension uuid-ossp

Succeeded: 297 (changed=56)
Failed:      1
Warnings:    1

[aboe76]

@noelmcloughlin thanks for this

[noelmcloughlin]

Cool, thanks @aboe76