-
Notifications
You must be signed in to change notification settings - Fork 200
Json package report #399
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Json package report #399
Conversation
…o json requestn for package/:package
Yes, this looks like a perfectly reasonable direction. And if you like you can do it incrementally: start with the fields that have obvious toJSON instances already, and we can merge & deploy that, and you can add more stuff in subsequent patches. |
Coming along with the json package report. Lots of Cabal's types have toJSON functions (not instances of the ToJSON/FromJSON though, those would be orphans). The endpoint http://localhost:8080/package/pipes-rt with Three questions:
|
Ok, this is ready for round 1, with basic support for JSON rendering of most fields in a PackageRender. Round 2 will cover the few remaining fields, collect the xFromJSON functions into a FromJSON instance for PackageRender, and have some round-trip tests between that and ToJSON. If you would like to wait for Round 2 before merging anything, that's ok with me, but for now Round 1 is ready and may be useful if you would like to go ahead. Thanks! |
Anything ever happen here? :-) |
@gbaz This has gone out of date, but I'm happy to pick it back up. Could I ask your opinion, about the hand-written fromJsonXYZ functions in this PR for many types defined in the |
…o json requestn for package/:package
I replaced nearly all of the hand-written aeson instances with generic derived ones. Endpoints that don't list a specific package version collect json data for all versions in an object whose keys are the version strings. When a specific version is mentioned, the top-level JSON object is specific to that version. Below is a sample output from @gbaz and @hvr, if you have use cases for the json package description in mind, do you like this format? Are there fields that you prefer to have rendered differently (dependency version ranges, for example, seem pretty hard to consume in the format here - it may be better to pretty print dependency versions out to the format they have in a cabal file?) @dcoutts, is there anything about the PR you'd like me to before you can merge? (squash all the commits?) Thanks!!
|
Neat! This output seems pretty workable. For the use case I have in mind (easing implementation of #198) just getting the version info for the latest package via xhttp requests, which this covers, should be fine. As long as the info is there in some format, then other tools seem like they can be built on top of this, and its probably better to more faithfully represent the structures (assuming they're not going to change too drastically) than to try to process them for certain use cases ahead-of-time... |
Don't want this to languish any longer. If nobody objects I'll try to find time to review and merge. |
@gbaz my main concern with this one, is that I don't want to commit to an API; i.e. I don't want to get complaints if for any reason I decide to change/refactor the JSON representation because the original representation turned out to be wrong/inadequate; in matrix.hho I use version-prefixed urlpaths for that to give me that liberty. So, I'm ok with merging this one, but attaching a big disclaimer that this API is subject to change without notice (or alternatively using a versioned API). |
@gbaz @hvr Some of the instances here are cumbersome to use - version bounds being a prime example. This example of a dependency on
Do you have a proposal for something nicer? Or is it good to stick with something completely default? Maybe
|
Any updates to this PR? |
@alexcmcdaniel as you seem interested, what's your use-case? |
@alexcmcdaniel I haven't worked on it recently. Picking it back up would be nice, but (a) it can be problematic to define an API using derived instances on types that may change in the future, since we would want those types to be more flexible than an API should be. And (b) we don't know what the API should look like - should the response JSON be a machine-generated-looking blob such as the one I posted in a comment above? Or should we be adding/removing/reformatting some fields? I'm definitely happy to hack on this more! I just never felt like I found the right answers to the above questions. So if you or others have an opinion, I'd love to hear. |
I am looking to collect basic metadata on packages i.e. license, description, author. I believe the JSON shown would work well. In terms of reformatting the JSON, as @hvr mentioned, would it be possible to append a version to the URL? Something like /v1/package/zlib. That way any changes to the JSON could be made in a new version. |
@imalsogreg fwiw, I'd strongly prefer a hand-tuned JSON schema; when it comes to record-fields I can almost never use the default generics inferred ToJSON instances as they're just awful unless you heavily namespace your data type definitions to have the field-names you want (and it breaks down for e.g. the dependency expressions). At the very least I need to add some smarter record-fieldname <-> json key dict string translation and carefully design the haskell-land record field names, and be even more careful to not change their names during refactorings in order not to cause unintended schema changes... it's a mess :-) |
@hvr thank you! That is enough convincing for us to avoid derived instances. I'll come up with a type to model the JSON package report, whose only use is in this JSON endpoint, along with a conversion function from |
@imalsogreg is there an estimated release time for this? |
@imalsogreg @alexcmcdaniel to make some progress on this: we should do this incrementally imo: let's start exposing the no-brainer properties into a JSON rep; So it was mentioned that we want the basic data, such as
let's start with those simple fields; Moreover, I'd like to have a new endpoint to query the list of versions available for a package, including their deprecation status, upload time and uploader, and possibly list of revisions; this is something I e.g. currently have to screen-scrape for There's already prior art for that, see e.g. https://hackage.haskell.org/package/HsYAML/candidates/.json Also, I'd leave off the dependency information, and instead leave that for the http://hackage.haskell.org/package/lens-4.17/dependencies endpoint |
@imalsogreg @hvr Having the dependency information would also be nice but I understand it is a bit more difficult than fields such as the name and license |
btw, have you looked at the hpack cabal-to-yaml converter? would it be possible to take a similar approach? https://github.com/yamadapc/hpack-convert |
@alexcmcdaniel I'm not sure what you mean by that; I don't see anything relevant/reusable in hpack-convert to the task at hand |
@imalsogreg @hvr Oh sorry I misunderstood their implementation. Are there any updates to this initiative? |
@alexcmcdaniel Yes, I've picked this back up, but slowly. It's taking some work for me to get things building on my machine, but after that it should be relatively quick to implement the smaller endpoint hvr is talking about. I agree, let's save the dependency data for a followup PR adding a json endpoint for |
Thanks @imalsogreg ! |
@imalsogreg Hey sorry I hate to be a nuisance but is there any update on the timeline? |
@alexcmcdaniel Not a nuisance. :) My work in progress is at https://github.com/imalsogreg/hackage-server/commits/jsonPackageReport2
|
Closing this in favor of #810 |
Support for #338
I've written a provisional
ToJSON
instance forPackageRender
. Many of the fields' types don't haveToJSON
instances, so Ishow
them for now. To complete the pull request I'd like to writeToJSON
instances for the many types withinPackageRender
, preferably by derivingGeneric
.For now though, if you hit /package/:package with
Accept: application/json
, then some JSON comes back. If no version suffix is in the URL (i.e./package/zlib
), then the JSON is a lists all packages by that name, indexed by version. If a version is mentioned ( i.e./package/zlib-0.5.4
), then only than one version is serialized, and it comes back with no version tag. Requests for packages or versions that don't exist return JSON empty array.Comments/suggestions appreciated - is this the right direction to finish up?