Pluggable architecture

[dmp42]

This is not to be merged right now.

It's an overview (and a call for comments) of the new plugin architecture.

A few facts:

• right now, the plugins are expressed as git submodules in the depends subfolder, and the git repos are under my nick
• the idea is to use namespaces - unfortunately, using docker_registry as a ns turned out to be more difficult than expected - hence the (slight) delay on this PR
• docker-registry-core (https://github.com/dmp42/docker-registry-core) is to contain (few) common utilities that are of interest to plugins and registry, its purpose being to minimize the work for plugins developers and to let us express our storage API and centrally update tests
• docker-registry-driver-file (https://github.com/dmp42/docker-registry-driver-file) (and swift) are first expressions of these new plugins
• these are not used yet in docker_registry and generally its broken and unusable

What I would love to have for now is opinions:

• take a look into core: are you happy with what is there?
• take a look into a driver (say: file): are you happy with the way it's done?
• are we ok with git submodules? or do we prefer straight folders in the main repo?
• any other remarks about the architecture

I'll push more commit as soon as I'm building successfully again.

cc @samalba @shin-

[dmp42]

Here is how plugins are expected to attach:

• https://github.com/dmp42/docker-registry-driver-swift/blob/master/setup.py#L53
• here is how plugins benefits from central tests: https://github.com/dmp42/docker-registry-driver-file/blob/master/tests/test.py

[dmp42]

The changes in init here are meant for the namespace to be used - thus, this have been moved to the run file below.

[dmp42]

We are almost there.

• Elliptics and swift have been adopted externally now (the packages are squeaky clean, including travis int., running tests and pip readyness)
• we still have boto inside core, and file and s3 as bundled drivers in registry itself.
• we are now testing against python26 as well, and the tests pass (there is a number of fellows running 2.6 apparently)
• we now support coverage threshold (40%) that breaks the build
• the build is back green on travis
• build time on travis dropped from 10 minutes to 3
• the behavior on remove and dir_list have been largely harmonized (when the dir is empty, or doesn't exists)
• we are now using nose for tests and coverage
• we support a simpler call for the tests through python setup.py nosetests

Still todo for tomorrow, before we can merge:

• fix up gcs and glance so that tests pass
• plug the unicode patch and unicode breaking tests
• merge config through env vars #350
• update the documentation

[samalba]

LGTM

[dmp42]

Ok. We are trying staging right now.

[dmp42]

@shin- ok for you?

[shin-]

cleanup?

[dmp42]

done

[shin-]

I don't know if this slipped through or if it was intentionally put in. Don't think we want editor-related files in the repo?

[dmp42]

Right, let's remove it.

[shin-]

lgtm!

[dmp42]

Let's drop young, then.

[paulczar]

This breaks the nova-docker driver from easily using the registry with a glance backend.

'Traceback (most recent call last):
  File "/usr/local/bin/docker-registry", line 9, in <module>
    load_entry_point('docker-registry==0.7.0', 'console_scripts', 'docker-registry')()
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 351, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 2363, in load_entry_point
    return ep.load()
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 2088, in load
    entry = __import__(self.module_name, globals(),globals(), ['__name__'])
  File "/usr/local/lib/python2.7/dist-packages/docker_registry/run.py", line 16, in <module>
    from .tags import *  # noqa
  File "/usr/local/lib/python2.7/dist-packages/docker_registry/tags.py", line 21, in <module>
    store = storage.load()
  File "/usr/local/lib/python2.7/dist-packages/docker_registry/storage/__init__.py", line 36, in load
    _storage[kind] = engine.fetch(kind)(None, config=cfg)
  File "/usr/local/lib/python2.7/dist-packages/docker_registry/core/driver.py", line 241, in fetch
    % (name, name, available())
docker_registry.core.exceptions.NotImplementedError: You requested storage driver docker_registry.drivers.glance
which is not installed. Try `pip install docker_registry-drivers-glance`
or check your configuration. The following are currently
available on your system: ['dumb', 'file', 's3']

it would be good for the default Dockerfile in the repo to include the glance and swift storage drivers.

[dmp42]

@paulczar:

Glance is work in progress and will get fixed before the release.

Any help in setting up glance for tests is welcome in https://github.com/dmp42/docker-registry-driver-glance/blob/master/.travis.yml

About wether to (re)include swift and glance as a default, this is not my call to make, but the whole idea is to make it as simple as pip install docker-registry-driver-glance to have an additional driver.

[shin-]

Would releasing a nova-registry image with glance and swift baked in be a viable solution?

[dmp42]

@shin- the docker-registry-driver-glance repo can easily provide a Dockerfile as well extending the registry image with glance crammed in (makes sense to maintain it there?).

[paulczar]

@shin- @dmp42 that sounds like a good viable option ... would be good to get a trusted build out of in the root namespace on index.docker.io