Skip to content

Json package report #399

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 30 commits into from
Closed

Conversation

imalsogreg
Copy link
Contributor

Support for #338

I've written a provisional ToJSON instance for PackageRender. Many of the fields' types don't have ToJSON instances, so I show them for now. To complete the pull request I'd like to write ToJSON instances for the many types within PackageRender, preferably by deriving Generic.

For now though, if you hit /package/:package with Accept: application/json, then some JSON comes back. If no version suffix is in the URL (i.e. /package/zlib), then the JSON is a lists all packages by that name, indexed by version. If a version is mentioned ( i.e. /package/zlib-0.5.4), then only than one version is serialized, and it comes back with no version tag. Requests for packages or versions that don't exist return JSON empty array.

Comments/suggestions appreciated - is this the right direction to finish up?

@dcoutts
Copy link
Contributor

dcoutts commented Aug 20, 2015

Yes, this looks like a perfectly reasonable direction. And if you like you can do it incrementally: start with the fields that have obvious toJSON instances already, and we can merge & deploy that, and you can add more stuff in subsequent patches.

@imalsogreg
Copy link
Contributor Author

Coming along with the json package report. Lots of Cabal's types have toJSON functions (not instances of the ToJSON/FromJSON though, those would be orphans). The endpoint http://localhost:8080/package/pipes-rt with Content-Type:application/json returns this: http://lpaste.net/141396

Three questions:

  1. Is master the right branch for the PR
  2. @hvr have I hit the parts of the package description you wanted to see? I ignored these fields from PackageDescription: https://github.com/haskell/cabal/blob/master/Cabal/Distribution/PackageDescription.hs#L200-L203. They are rendered with show instead of json'ed.
  3. Any preferences about moving that giant block of json code out of Render.hs? Or replacing it all with orphan ToJSON/FromJSON instances, using generics?

@imalsogreg
Copy link
Contributor Author

Ok, this is ready for round 1, with basic support for JSON rendering of most fields in a PackageRender.

Round 2 will cover the few remaining fields, collect the xFromJSON functions into a FromJSON instance for PackageRender, and have some round-trip tests between that and ToJSON.

If you would like to wait for Round 2 before merging anything, that's ok with me, but for now Round 1 is ready and may be useful if you would like to go ahead.

Thanks!

@gbaz
Copy link
Contributor

gbaz commented Oct 23, 2016

Anything ever happen here? :-)

@imalsogreg
Copy link
Contributor Author

imalsogreg commented Oct 23, 2016

@gbaz This has gone out of date, but I'm happy to pick it back up. Could I ask your opinion, about the hand-written fromJsonXYZ functions in this PR for many types defined in the cabal package? These look pretty fragile, and it may be better to make a cabal-orphans package, use standalone deriving to get Generic orphan instances and use those to make ToJSON and FromJSON instances. What do you think? hackage-server would pick up a dependency on cabal-orphans.

@imalsogreg
Copy link
Contributor Author

I replaced nearly all of the hand-written aeson instances with generic derived ones.

Endpoints that don't list a specific package version collect json data for all versions in an object whose keys are the version strings. When a specific version is mentioned, the top-level JSON object is specific to that version. Below is a sample output from http://localhost:8002/package/groundhog-0.7.0.3.json.

@gbaz and @hvr, if you have use cases for the json package description in mind, do you like this format? Are there fields that you prefer to have rendered differently (dependency version ranges, for example, seem pretty hard to consume in the format here - it may be better to pretty print dependency versions out to the format they have in a cabal file?)

@dcoutts, is there anything about the PR you'd like me to before you can merge? (squash all the commits?)

Thanks!!

{
   "licenseName":"BSD3",
   "other":{
      "homepage":"http://github.com/lykahb/groundhog",
      "setupBuildInfo":null,
      "dataDir":"",
      "library":null,
      "copyright":"",
      "testedWith":[

      ],
      "extraSrcFiles":[
         "changelog"
      ],
      "category":"Database",
      "testSuites":[

      ],
      "specVersionRaw":{
         "Right":{
            "tag":"UnionVersionRanges",
            "contents":[
               {
                  "tag":"ThisVersion",
                  "contents":"1.6"
               },
               {
                  "tag":"LaterVersion",
                  "contents":"1.6"
               }
            ]
         }
      },
      "licenseFiles":[
         "LICENSE"
      ],
      "maintainer":"Boris Lykah <[email protected]>",
      "benchmarks":[

      ],
      "pkgUrl":"",
      "extraDocFiles":[

      ],
      "synopsis":"Type-safe datatype-database mapping library.",
      "package":{
         "pkgName":{
            "unPackageName":"groundhog"
         },
         "pkgVersion":"0.7.0.3"
      },
      "stability":"Stable",
      "author":"Boris Lykah <[email protected]>",
      "bugReports":"",
      "dataFiles":[

      ],
      "license":{
         "tag":"BSD3"
      },
      "executables":[

      ],
      "buildDepends":[

      ],
      "extraTmpFiles":[

      ],
      "buildType":{
         "tag":"Simple"
      },
      "description":"This library maps your datatypes to a relational model, in a way similar to what ORM libraries do in object-oriented programming. The mapping can be configured to work with almost any schema. Groundhog supports schema migrations, composite keys, advanced expressions in queries, and much more. See tutorial <https://www.fpcomplete.com/user/lykahb/groundhog> and examples <https://github.com/lykahb/groundhog/tree/master/examples> on GitHub.",
      "sourceRepos":[
         {
            "repoBranch":null,
            "repoModule":null,
            "repoType":{
               "tag":"Git"
            },
            "repoTag":null,
            "repoLocation":"git://github.com/lykahb/groundhog.git",
            "repoKind":{
               "tag":"RepoHead"
            },
            "repoSubdir":null
         }
      ],
      "customFieldsPD":[

      ]
   },
   "flags":[

   ],
   "execNames":[

   ],
   "pkgUri":"/package/groundhog-0.7.0.3",
   "readme":null,
   "depends":[
      [
         {
            "unPackageName":"aeson"
         },
         {
            "tag":"UnionVersionRanges",
            "contents":[
               {
                  "tag":"ThisVersion",
                  "contents":"0.5"
               },
               {
                  "tag":"LaterVersion",
                  "contents":"0.5"
               }
            ]
         }
      ],
      [
         {
            "unPackageName":"attoparsec"
         },
         {
            "tag":"UnionVersionRanges",
            "contents":[
               {
                  "tag":"ThisVersion",
                  "contents":"0.11"
               },
               {
                  "tag":"LaterVersion",
                  "contents":"0.11"
               }
            ]
         }
      ],
      [
         {
            "unPackageName":"base"
         },
         {
            "tag":"IntersectVersionRanges",
            "contents":[
               {
                  "tag":"UnionVersionRanges",
                  "contents":[
                     {
                        "tag":"ThisVersion",
                        "contents":"4.5"
                     },
                     {
                        "tag":"LaterVersion",
                        "contents":"4.5"
                     }
                  ]
               },
               {
                  "tag":"EarlierVersion",
                  "contents":"5"
               }
            ]
         }
      ],
      [
         {
            "unPackageName":"base64-bytestring"
         },
         {
            "tag":"AnyVersion"
         }
      ],
      [
         {
            "unPackageName":"blaze-builder"
         },
         {
            "tag":"IntersectVersionRanges",
            "contents":[
               {
                  "tag":"UnionVersionRanges",
                  "contents":[
                     {
                        "tag":"ThisVersion",
                        "contents":"0.3"
                     },
                     {
                        "tag":"LaterVersion",
                        "contents":"0.3"
                     }
                  ]
               },
               {
                  "tag":"EarlierVersion",
                  "contents":"0.5"
               }
            ]
         }
      ],
      [
         {
            "unPackageName":"bytestring"
         },
         {
            "tag":"UnionVersionRanges",
            "contents":[
               {
                  "tag":"ThisVersion",
                  "contents":"0.10"
               },
               {
                  "tag":"LaterVersion",
                  "contents":"0.10"
               }
            ]
         }
      ],
      [
         {
            "unPackageName":"containers"
         },
         {
            "tag":"UnionVersionRanges",
            "contents":[
               {
                  "tag":"ThisVersion",
                  "contents":"0.2"
               },
               {
                  "tag":"LaterVersion",
                  "contents":"0.2"
               }
            ]
         }
      ],
      [
         {
            "unPackageName":"monad-control"
         },
         {
            "tag":"IntersectVersionRanges",
            "contents":[
               {
                  "tag":"UnionVersionRanges",
                  "contents":[
                     {
                        "tag":"ThisVersion",
                        "contents":"0.3"
                     },
                     {
                        "tag":"LaterVersion",
                        "contents":"0.3"
                     }
                  ]
               },
               {
                  "tag":"EarlierVersion",
                  "contents":"1.1"
               }
            ]
         }
      ],
      [
         {
            "unPackageName":"mtl"
         },
         {
            "tag":"UnionVersionRanges",
            "contents":[
               {
                  "tag":"ThisVersion",
                  "contents":"2.0"
               },
               {
                  "tag":"LaterVersion",
                  "contents":"2.0"
               }
            ]
         }
      ],
      [
         {
            "unPackageName":"resourcet"
         },
         {
            "tag":"UnionVersionRanges",
            "contents":[
               {
                  "tag":"ThisVersion",
                  "contents":"1.1.2"
               },
               {
                  "tag":"LaterVersion",
                  "contents":"1.1.2"
               }
            ]
         }
      ],
      [
         {
            "unPackageName":"scientific"
         },
         {
            "tag":"AnyVersion"
         }
      ],
      [
         {
            "unPackageName":"text"
         },
         {
            "tag":"UnionVersionRanges",
            "contents":[
               {
                  "tag":"ThisVersion",
                  "contents":"0.8"
               },
               {
                  "tag":"LaterVersion",
                  "contents":"0.8"
               }
            ]
         }
      ],
      [
         {
            "unPackageName":"time"
         },
         {
            "tag":"UnionVersionRanges",
            "contents":[
               {
                  "tag":"ThisVersion",
                  "contents":"1.1.4"
               },
               {
                  "tag":"LaterVersion",
                  "contents":"1.1.4"
               }
            ]
         }
      ],
      [
         {
            "unPackageName":"transformers"
         },
         {
            "tag":"IntersectVersionRanges",
            "contents":[
               {
                  "tag":"UnionVersionRanges",
                  "contents":[
                     {
                        "tag":"ThisVersion",
                        "contents":"0.2.1"
                     },
                     {
                        "tag":"LaterVersion",
                        "contents":"0.2.1"
                     }
                  ]
               },
               {
                  "tag":"EarlierVersion",
                  "contents":"0.6"
               }
            ]
         }
      ],
      [
         {
            "unPackageName":"transformers-base"
         },
         {
            "tag":"AnyVersion"
         }
      ]
   ],
   "category":[
      "Database"
   ],
   "repoHeads":[
      [
         {
            "tag":"Git"
         },
         "git://github.com/lykahb/groundhog.git",
         {
            "repoBranch":null,
            "repoModule":null,
            "repoType":{
               "tag":"Git"
            },
            "repoTag":null,
            "repoLocation":"git://github.com/lykahb/groundhog.git",
            "repoKind":{
               "tag":"RepoHead"
            },
            "repoSubdir":null
         }
      ]
   ],
   "licenseFiles":[
      "LICENSE"
   ],
   "hasTarball":true,
   "maintainer":"Boris Lykah <[email protected]>",
   "modules":[
      [
         "Database",
         false,
         false,
         [
            [
               "Groundhog",
               true,
               false,
               [
                  [
                     "Core",
                     true,
                     false,
                     [

                     ]
                  ],
                  [
                     "Expression",
                     true,
                     false,
                     [

                     ]
                  ],
                  [
                     "Generic",
                     true,
                     false,
                     [
                        [
                           "Migration",
                           true,
                           false,
                           [

                           ]
                        ],
                        [
                           "PersistBackendHelpers",
                           true,
                           false,
                           [

                           ]
                        ],
                        [
                           "Sql",
                           true,
                           false,
                           [
                              [
                                 "Functions",
                                 true,
                                 false,
                                 [

                                 ]
                              ]
                           ]
                        ]
                     ]
                  ],
                  [
                     "Instances",
                     true,
                     false,
                     [

                     ]
                  ]
               ]
            ]
         ]
      ]
   ],
   "pkgId":{
      "pkgName":{
         "unPackageName":"groundhog"
      },
      "pkgVersion":"0.7.0.3"
   },
   "updateInfo":null,
   "changeLog":null,
   "executableDeps":[

   ],
   "uploadInfo":[
      "2016-10-27T01:12:02.468533Z",
      {
         "status":"AccountEnabled",
         "name":"imalsogreg"
      }
   ],
   "libraryDeps":{
      "condTreeComponents":[

      ],
      "condTreeData":"Buildable",
      "condTreeConstraints":[
         [
            {
               "unPackageName":"aeson"
            },
            {
               "tag":"UnionVersionRanges",
               "contents":[
                  {
                     "tag":"ThisVersion",
                     "contents":"0.5"
                  },
                  {
                     "tag":"LaterVersion",
                     "contents":"0.5"
                  }
               ]
            }
         ],
         [
            {
               "unPackageName":"attoparsec"
            },
            {
               "tag":"UnionVersionRanges",
               "contents":[
                  {
                     "tag":"ThisVersion",
                     "contents":"0.11"
                  },
                  {
                     "tag":"LaterVersion",
                     "contents":"0.11"
                  }
               ]
            }
         ],
         [
            {
               "unPackageName":"base"
            },
            {
               "tag":"IntersectVersionRanges",
               "contents":[
                  {
                     "tag":"UnionVersionRanges",
                     "contents":[
                        {
                           "tag":"ThisVersion",
                           "contents":"4.5"
                        },
                        {
                           "tag":"LaterVersion",
                           "contents":"4.5"
                        }
                     ]
                  },
                  {
                     "tag":"EarlierVersion",
                     "contents":"5"
                  }
               ]
            }
         ],
         [
            {
               "unPackageName":"base64-bytestring"
            },
            {
               "tag":"AnyVersion"
            }
         ],
         [
            {
               "unPackageName":"blaze-builder"
            },
            {
               "tag":"IntersectVersionRanges",
               "contents":[
                  {
                     "tag":"UnionVersionRanges",
                     "contents":[
                        {
                           "tag":"ThisVersion",
                           "contents":"0.3"
                        },
                        {
                           "tag":"LaterVersion",
                           "contents":"0.3"
                        }
                     ]
                  },
                  {
                     "tag":"EarlierVersion",
                     "contents":"0.5"
                  }
               ]
            }
         ],
         [
            {
               "unPackageName":"bytestring"
            },
            {
               "tag":"UnionVersionRanges",
               "contents":[
                  {
                     "tag":"ThisVersion",
                     "contents":"0.10"
                  },
                  {
                     "tag":"LaterVersion",
                     "contents":"0.10"
                  }
               ]
            }
         ],
         [
            {
               "unPackageName":"containers"
            },
            {
               "tag":"UnionVersionRanges",
               "contents":[
                  {
                     "tag":"ThisVersion",
                     "contents":"0.2"
                  },
                  {
                     "tag":"LaterVersion",
                     "contents":"0.2"
                  }
               ]
            }
         ],
         [
            {
               "unPackageName":"monad-control"
            },
            {
               "tag":"IntersectVersionRanges",
               "contents":[
                  {
                     "tag":"UnionVersionRanges",
                     "contents":[
                        {
                           "tag":"ThisVersion",
                           "contents":"0.3"
                        },
                        {
                           "tag":"LaterVersion",
                           "contents":"0.3"
                        }
                     ]
                  },
                  {
                     "tag":"EarlierVersion",
                     "contents":"1.1"
                  }
               ]
            }
         ],
         [
            {
               "unPackageName":"mtl"
            },
            {
               "tag":"UnionVersionRanges",
               "contents":[
                  {
                     "tag":"ThisVersion",
                     "contents":"2.0"
                  },
                  {
                     "tag":"LaterVersion",
                     "contents":"2.0"
                  }
               ]
            }
         ],
         [
            {
               "unPackageName":"resourcet"
            },
            {
               "tag":"UnionVersionRanges",
               "contents":[
                  {
                     "tag":"ThisVersion",
                     "contents":"1.1.2"
                  },
                  {
                     "tag":"LaterVersion",
                     "contents":"1.1.2"
                  }
               ]
            }
         ],
         [
            {
               "unPackageName":"scientific"
            },
            {
               "tag":"AnyVersion"
            }
         ],
         [
            {
               "unPackageName":"text"
            },
            {
               "tag":"UnionVersionRanges",
               "contents":[
                  {
                     "tag":"ThisVersion",
                     "contents":"0.8"
                  },
                  {
                     "tag":"LaterVersion",
                     "contents":"0.8"
                  }
               ]
            }
         ],
         [
            {
               "unPackageName":"time"
            },
            {
               "tag":"UnionVersionRanges",
               "contents":[
                  {
                     "tag":"ThisVersion",
                     "contents":"1.1.4"
                  },
                  {
                     "tag":"LaterVersion",
                     "contents":"1.1.4"
                  }
               ]
            }
         ],
         [
            {
               "unPackageName":"transformers"
            },
            {
               "tag":"IntersectVersionRanges",
               "contents":[
                  {
                     "tag":"UnionVersionRanges",
                     "contents":[
                        {
                           "tag":"ThisVersion",
                           "contents":"0.2.1"
                        },
                        {
                           "tag":"LaterVersion",
                           "contents":"0.2.1"
                        }
                     ]
                  },
                  {
                     "tag":"EarlierVersion",
                     "contents":"0.6"
                  }
               ]
            }
         ],
         [
            {
               "unPackageName":"transformers-base"
            },
            {
               "tag":"AnyVersion"
            }
         ]
      ]
   }
}

@gbaz
Copy link
Contributor

gbaz commented Oct 27, 2016

Neat! This output seems pretty workable. For the use case I have in mind (easing implementation of #198) just getting the version info for the latest package via xhttp requests, which this covers, should be fine. As long as the info is there in some format, then other tools seem like they can be built on top of this, and its probably better to more faithfully represent the structures (assuming they're not going to change too drastically) than to try to process them for certain use cases ahead-of-time...

@gbaz gbaz self-assigned this Mar 25, 2018
@gbaz
Copy link
Contributor

gbaz commented Mar 25, 2018

Don't want this to languish any longer. If nobody objects I'll try to find time to review and merge.

@hvr
Copy link
Member

hvr commented Mar 25, 2018

@gbaz my main concern with this one, is that I don't want to commit to an API; i.e. I don't want to get complaints if for any reason I decide to change/refactor the JSON representation because the original representation turned out to be wrong/inadequate; in matrix.hho I use version-prefixed urlpaths for that to give me that liberty. So, I'm ok with merging this one, but attaching a big disclaimer that this API is subject to change without notice (or alternatively using a versioned API).

@imalsogreg
Copy link
Contributor Author

@gbaz @hvr Some of the instances here are cumbersome to use - version bounds being a prime example.

This example of a dependency on transformers shows how the attempt to use default json instances results in ugly output

[
            {
               "unPackageName":"transformers"
            },
            {
               "tag":"IntersectVersionRanges",
               "contents":[
                  {
                     "tag":"UnionVersionRanges",
                     "contents":[
                        {
                           "tag":"ThisVersion",
                           "contents":"0.2.1"
                        },
                        {
                           "tag":"LaterVersion",
                           "contents":"0.2.1"
                        }
                     ]
                  },
                  {
                     "tag":"EarlierVersion",
                     "contents":"0.6"
                  }
               ]
            }
         ]

Do you have a proposal for something nicer? Or is it good to stick with something completely default? Maybe { "transformers": ">= 0.2.1 && < 0.6" }, or

{ "transformers": 
  { "intersection": 
    [ {"ge": [0,2,1]}, 
      {"lt": [0,6]} 
    ]
  }
}

@alexcmcdaniel
Copy link

Any updates to this PR?

@hvr
Copy link
Member

hvr commented Jan 11, 2019

@alexcmcdaniel as you seem interested, what's your use-case?

@imalsogreg
Copy link
Contributor Author

imalsogreg commented Jan 11, 2019

@alexcmcdaniel I haven't worked on it recently. Picking it back up would be nice, but (a) it can be problematic to define an API using derived instances on types that may change in the future, since we would want those types to be more flexible than an API should be. And (b) we don't know what the API should look like - should the response JSON be a machine-generated-looking blob such as the one I posted in a comment above? Or should we be adding/removing/reformatting some fields?

I'm definitely happy to hack on this more! I just never felt like I found the right answers to the above questions. So if you or others have an opinion, I'd love to hear.

@alexcmcdaniel
Copy link

I am looking to collect basic metadata on packages i.e. license, description, author. I believe the JSON shown would work well. In terms of reformatting the JSON, as @hvr mentioned, would it be possible to append a version to the URL? Something like /v1/package/zlib. That way any changes to the JSON could be made in a new version.

@hvr
Copy link
Member

hvr commented Jan 17, 2019

@imalsogreg fwiw, I'd strongly prefer a hand-tuned JSON schema; when it comes to record-fields I can almost never use the default generics inferred ToJSON instances as they're just awful unless you heavily namespace your data type definitions to have the field-names you want (and it breaks down for e.g. the dependency expressions). At the very least I need to add some smarter record-fieldname <-> json key dict string translation and carefully design the haskell-land record field names, and be even more careful to not change their names during refactorings in order not to cause unintended schema changes... it's a mess :-)

@imalsogreg
Copy link
Contributor Author

@hvr thank you! That is enough convincing for us to avoid derived instances. I'll come up with a type to model the JSON package report, whose only use is in this JSON endpoint, along with a conversion function from Distribution.Server.Packages.Render.PackageRender.

@alexcmcdaniel
Copy link

@imalsogreg is there an estimated release time for this?

@hvr
Copy link
Member

hvr commented Jan 21, 2019

@imalsogreg @alexcmcdaniel to make some progress on this: we should do this incrementally imo: let's start exposing the no-brainer properties into a JSON rep;

So it was mentioned that we want the basic data, such as

  • license
  • description
  • author

let's start with those simple fields;

Moreover, I'd like to have a new endpoint to query the list of versions available for a package, including their deprecation status, upload time and uploader, and possibly list of revisions; this is something I e.g. currently have to screen-scrape for hackage-cli and it tends to break whenever the HTML changes.

There's already prior art for that, see e.g. https://hackage.haskell.org/package/HsYAML/candidates/.json

Also, I'd leave off the dependency information, and instead leave that for the http://hackage.haskell.org/package/lens-4.17/dependencies endpoint

@alexcmcdaniel
Copy link

@imalsogreg @hvr Having the dependency information would also be nice but I understand it is a bit more difficult than fields such as the name and license

@alexcmcdaniel
Copy link

btw, have you looked at the hpack cabal-to-yaml converter? would it be possible to take a similar approach? https://github.com/yamadapc/hpack-convert

@hvr
Copy link
Member

hvr commented Jan 29, 2019

@alexcmcdaniel I'm not sure what you mean by that; I don't see anything relevant/reusable in hpack-convert to the task at hand

@alexcmcdaniel
Copy link

@imalsogreg @hvr Oh sorry I misunderstood their implementation. Are there any updates to this initiative?

@imalsogreg
Copy link
Contributor Author

@alexcmcdaniel Yes, I've picked this back up, but slowly. It's taking some work for me to get things building on my machine, but after that it should be relatively quick to implement the smaller endpoint hvr is talking about. I agree, let's save the dependency data for a followup PR adding a json endpoint for /packagename/dependencies.

@alexcmcdaniel
Copy link

Thanks @imalsogreg !

@alexcmcdaniel
Copy link

@imalsogreg Hey sorry I hate to be a nuisance but is there any update on the timeline?

@imalsogreg
Copy link
Contributor Author

imalsogreg commented Feb 20, 2019

@alexcmcdaniel Not a nuisance. :) My work in progress is at https://github.com/imalsogreg/hackage-server/commits/jsonPackageReport2
When it's ready for a new PR and comments, I'll close this PR and point to the new one.
TODO:

  • Implement basic JSON listing for $package-$version and $package-$version/revision/$n
  • Hook up the new JSON feature in the global feature list
  • Caching
  • Try it out, bugfix
    I think I'll get through this list and have a fresh PR in a couple weeks?

@imalsogreg
Copy link
Contributor Author

Closing this in favor of #810

@imalsogreg imalsogreg closed this Feb 23, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants