pants: register assets (like conf files) and dependencies on them #5846
Conversation
cognifloyd
requested review from
winem,
arm4b,
nzlosh,
rush-skills,
amanda11 and
a team
pull-request-size
bot
added
the
size/XL
cognifloyd
force-pushed
the
pants-files-resources
branch
from
5b35e9c to
c365b7c
Compare
| file( | ||
| name="st2.conf.sample", | ||
| source="st2.conf.sample", | ||
| ) |
There was a problem hiding this comment.
A future PR will add pants-plugins/sample_conf which will wire this up so that it gets regenerated whenever someone runs the fmt goal with something like: ./pants fmt conf/st2.conf.sample. That will take advantage of the fine-grained caching so that this always gets updated when the files used to generate it change.
| resources( | ||
| name="openapi_spec", | ||
| sources=["openapi.yaml", "openapi.yaml.j2"], | ||
| ) |
There was a problem hiding this comment.
A future PR will add pants-plugins/api_spec which will wire this up so that it gets regenerated whenever someone runs the fmt goal with something like: ./pants fmt st2common/st2comon/openapi.yaml. That will take advantage of the fine-grained caching so that openapi.yaml always gets updated when the files used to generate it change.
cognifloyd
force-pushed
the
pants-files-resources
branch
from
7d22163 to
d9d2e60
Compare
| resources( | ||
| name="policies_meta", | ||
| sources=["*.yaml"], | ||
| ) |
There was a problem hiding this comment.
Adding this BUILD file broke a test that was checking directory contents. So, I adjusted the test to account for it.
this will not be needed once #5846 is merged
| python_sources( | ||
| dependencies=[ | ||
| ":jsonschema", | ||
| ] | ||
| ) | ||
|
|
||
| resources( | ||
| name="jsonschema", | ||
| sources=["*.json"], | ||
| ) |
There was a problem hiding this comment.
This is required before I can submit the pants plugin that wires up pants to manage building the schemas in contrib/schemas.
cognifloyd
force-pushed
the
pants-files-resources
branch
2 times, most recently
from
dbda1f2 to
12f0dde
Compare
cognifloyd
force-pushed
the
pants-files-resources
branch
4 times, most recently
from
c33a175 to
341f372
Compare
this will not be needed once #5846 is merged
rbac fixtures are not used in st2.git They are used here: https://github.com/StackStorm/st2-rbac-backend/blob/master/tests/unit/test_rbac_loader.py
Let pants know where to get these conf files. Note that these conf files are not the same as the files in the repo root: //conf/st2.tests*.conf
No technical reason. Just trying to be more consistent.
'artifacts' refers more to distributable stuff like wheels. 'assets' encompasses 'file' and 'resource' targets.
cognifloyd
force-pushed
the
pants-files-resources
branch
from
341f372 to
e0119dc
Compare
Background
This is another part of introducing pants, as discussed in various TSC meetings.
Related PRs can be found in:
Overview of this PR
So far, we have told pants about our python and our shell sources.
./pants tailor ::made that much easier. But we have more than just*.pyand*.shfiles; we have conf, ini, yaml, json, and many other kinds of files. In pants, these miscellaneous files are all "assets", andtailordoes not know what to do with them. So, we need to register the assets manually.This PR registers most of our assets with one of these targets:
file,files,resource, orresources.This PR also registers dependencies on these assets. I think I captured all the dependencies, but we may have to tweak them if I missed something.
I recommend stepping through the commits so that you can quickly see what depends on the assets added in that commit.
Relevant Pants documentation
fileresourcefilesresourcesFiles vs Resources
It can be confusing to figure out whether we need to register the assets as files or resources. In my PoC branch, I started with files and then gradually switched things over to resources when needed (like for building wheels). We will probably have to tweak
file/resourceon some of these in the future.aside: There have been some discussions in the pantsbuild community about how
fileandresourcecould be combined into anassettarget. Whether or not that happens, registering assets in pants will hopefully improve with time. For now, just think of "assets" as a helpful term to refer to both "files" and "resources".Future PRs
Follow-up PRs will introduce some plugins that add specific handling to some of the resources/files in this PR. This way, pants will be able to auto-generate or lint things like the api spec. I will make those once this PR is merged.
I will register the
LICENSEfile in conjunction with the metadata that we need to build wheels.There is also a large set of assets that I have intentionally skipped in this PR: pack metadata (including yaml files for packs, actions, workflows, rules, etc, and a few other misc files that get included in packs). I wrote a pants plugin to handle registering these assets because there are a lot of them. The plugin teaches
./pants tailor ::how to handle those files, automatically adding a custompack_metadatatarget.