cabal-install should report build results to hackage server

[bos]

(Imported from Trac #184, reported by @dcoutts on 2007-11-22)

One way to get a lot more testing data on hackage packages is if cabal-install could report back to the hackage server about build successes and failures. This information should be kept by hackage and used to distill information about which platforms configurations a package builds successfully and which it does not. This should provide useful information to developers to enable them to discover problems more quickly and useful information to users.

An important consideration is privacy. Users should always have the option to not report anything and any information they report that is kept should not contain identifying information. This is particularly important when it comes to build logs which may contain paths etc. It should be clear to users what information they are reporting to the can decide for themselves if it meets their privacy needs. Since cabal can be used to build private code it is vital that it reports only on packages that were obtained from the hackage server. It is also vital that the information is sent to the correct hackage server. It's possible to set up private hackage server instances and it'd may be useful to collect build information "in house" too.

So what information would be helpful?

• build success or failure (qualified by what failed, build, docs, tests)
• package name and version
• hash of .cabal file just to make sure it's the same one we're all talking about and to detect local modifications.
• precise versions of dependent packages that the package was built with.
• os and arch strings
• compiler flavour and version
• versions of important build tools
• In the case of a build failure, some part of the build log would be helpful. This is the most problematic part from a privacy point of view.

This is quite a bit of raw data and we can expect to have many hundreds of such reports for popular packages. How can we distill useful information from this kind of data?

I expect we would want to do some statistical analysis where we look for common traits in the results. The data forms a multi-dimensional space of boolean values. By looking down the rows/columns of this space we should be able to identify trends. For example if it always fails with ghc-6.8.x, or always fails on Windows. Excluding those obvious failures we then may be able to say that it does work on mac osx (except ghc-6.8.1) or that it does work with regex-posix-0.92 (except on windows), etc. This should allow us to give a summary saying yes/no to various properties of the environment.

For failure cases developers should be able to get access to the more detailed data including build logs.

[bos]

(Imported comment by @dcoutts on 2007-11-22)

There are various sub-tasks:

• Make cabal install not fail overall just because one package fails. There should be a mode to carry on and build all remaining packages that did not depend on the failed package. This requires keeping a full dep graph while installing, not just a linear list of packages.
• Define a build report data type
• Generate a build report for each
• Collect build reports in a local file
• Upload build reports to hackage at appropriate points. Perhaps new reports should be sent at the same that the package index or tarballs are downloaded.
• Implement server side script to allow reports to be uploaded. Have to decide where and how reports are stored. Obvious choice is a log file per package-version. Or some other kind of data base and index. The considerations here are in keeping the reports anonymous and what queries we will want to run on the reports.
• Implement the statistical analysis necessary to distill the reports into useful information per-package. Present this info on each package page.

[bos]

(Imported comment by @dcoutts on 2008-01-20)

Fixed:

Sat Apr 12 22:49:04 CEST 2008  Andres Loeh <[email protected]>
  * fix for #187 -- directory of Paths_packagename is included when looking for source files

[bos]

(Imported comment by @dcoutts on 2008-04-12)

Oops, closed the wrong bug....

[bos]

(Imported comment by @dcoutts on 2008-04-12)

Replying to @dcoutts:

There are various sub-tasks: * Make cabal install not fail overall just because one package fails. There should be a mode to carry on and build all remaining packages that did not depend on the failed package. This requires keeping a full dep graph while installing, not just a linear list of packages. * Define a build report data type * Generate a build report for each * Collect build reports in a local file

These are all now done. Build reports are logged into ~/.cabal/packages/$server/build-reports.log.

* Upload build reports to hackage at appropriate points. Perhaps new reports should be sent at the same that the package index or tarballs are downloaded. * Implement server side script to allow reports to be uploaded. Have to decide where and how reports are stored. Obvious choice is a log file per package-version. Or some other kind of data base and index. The considerations here are in keeping the reports anonymous and what queries we will want to run on the reports. * Implement the statistical analysis necessary to distill the reports into useful information per-package. Present this info on each package page.

These are the remaining tasks.

[bos]

(Imported comment by @dcoutts on 2008-06-10)

Current cabal-install can now generate anonymous or detailed build reports and upload them to a compatible server. The remaining work is mostly server side.

[tibbe]

Closing as there's been no activity in years.

We're cleaning up the bug tracker to make it useful again and are thus closing bugs that haven't seen any activity in a long time. Please re-open (or file a new bug) if the problem reappears.