git2rdata, a companion package for git2r

[ThierryO]

Summary

• What does this package do? (explain in 50 words or less):
  Store dataframes as plain text files along with metadata to ensure that attributes like factor levels are maintained. The dataframes are optimized in order to minimize both file size and diffs, making it useful in combination with version control.

• Paste the full DESCRIPTION file inside a code block below:

Package: git2rdata
Title: Store and Retrieve Data.frames in a Git Repository
Version: 0.0.1.9000
Date: 2018-11-12
Authors@R: c(
  person(
    "Thierry", "Onkelinx", role = c("aut", "cre"), 
    email = "thierry.onkelinx@inbo.be", 
    comment = c(ORCID = "0000-0001-8804-4216")),
  person(
    "Research Institute for Nature and Forest",
    role = c("cph", "fnd"), email = "info@inbo.be"))
Description: Make versioning of data.frame easy and efficient using git repositories.
Depends: R (>= 3.4.0)
Imports:
  assertthat,
  git2r (>= 0.23.0),
  methods,
  readr
Suggests:
  knitr,
  rmarkdown,
  testthat
License: GPL-3
Encoding: UTF-8
LazyData: true
Roxygen: list(markdown = TRUE)
RoxygenNote: 6.1.1
URL: https://github.com/inbo/git2rdata
BugReports: https://github.com/inbo/git2rdata/issues
Collate:
    'write_vc.R'
    'auto_commit.R'
    'clean_data_path.R'
    'git2rdata-package.R'
    'meta.R'
    'read_vc.R'
    'recent_commit.R'
    'reexport.R'
    'rm_data.R'
VignetteBuilder: knitr

• URL for the package (the development repository, not a stylized html page):

https://github.com/inbo/git2rdata

• Please indicate which category or categories from our package fit policies this package falls under *and why(? (e.g., data retrieval, reproducibility. If you are unsure, we suggest you make a pre-submission inquiry.):
  [e.g., "data extraction, because the package parses a scientific data file format"]

reproducibility, because it help to store dataframes as plain text files without loss of information, while minimize file sizes and diff.

•   Who is the target audience and what are scientific applications of this package?  

Anyone who wants to work with medium sized dataframes and have them under version control. This is useful in case of recurrent analysis on growing datasets. E.g. each year new data is added to the data and a new report is created. A snapshot of the raw data can be stored as plain text files under version control.

• Are there other R packages that accomplish the same thing? If so, how does
  yours differ or meet our criteria for best-in-category?

The initial idea was to add this functionality into git2r. After a discussion with the maintainer, we decide to create a separate package. We use functions from read_r and improve them by storing the metadata.

•   If you made a pre-submission enquiry, please paste the link to the corresponding issue, forum post, or other discussion, or @tag the editor you contacted.

Requirements

Confirm each of the following by checking the box. This package:

• does not violate the Terms of Service of any service it interacts with.
• has a CRAN and OSI accepted license.
• contains a README with instructions for installing the development version.
• includes documentation with examples for all functions. examples are available in the vignette
• contains a vignette with examples of its essential functions and uses.
• has a test suite.
• has continuous integration, including reporting of test coverage, using services such as Travis CI, Coveralls and/or CodeCov.
• I agree to abide by ROpenSci's Code of Conduct during the review process and in maintaining my package should it be accepted.

Publication options

• Do you intend for this package to go on CRAN?
• Do you wish to automatically submit to the Journal of Open Source Software? If so:
  • The package has an obvious research application according to JOSS's definition.
  • The package contains a paper.md matching JOSS's requirements with a high-level description in the package root or in inst/.
  • The package is deposited in a long-term repository with the DOI:
  • (Do not submit your package separately to JOSS)
• Do you wish to submit an Applications Article about your package to Methods in Ecology and Evolution? If so:
  • The package is novel and will be of interest to the broad readership of the journal.
  • The manuscript describing the package is no longer than 3000 words.
  • You intend to archive the code for the package in a long-term repository which meets the requirements of the journal (see MEE's Policy on Publishing Code)
  • (Scope: Do consider MEE's Aims and Scope for your manuscript. We make no guarantee that your manuscript will be within MEE scope.)
  • (Although not required, we strongly recommend having a full manuscript prepared when you submit here.)
  • (Please do not submit your package separately to Methods in Ecology and Evolution)

Detail

• Does R CMD check (or devtools::check()) succeed? Paste and describe any errors or warnings:

• Does the package conform to rOpenSci packaging guidelines? Please describe any exceptions:

We use read_vc() rather than vc_read() because it matches read.table(), read_csv(), ...

• If this is a resubmission following rejection, please explain the change in circumstances:

• If possible, please provide recommendations of reviewers - those with experience with similar packages and/or likely users of your package - and their GitHub user names:

@jennybc @stewid

[sckott]

thanks for your submission @ThierryO - editors are discussing now

[noamross]

Editor checks:

• Fit: The package meets criteria for fit and overlap
• Automated tests: Package has a testing suite and is tested via Travis-CI or another CI service.
• License: The package has a CRAN or OSI accepted license
• Repository: The repository link resolves correctly
• Archive (JOSS only, may be post-review): The repository DOI resolves correctly
• Version (JOSS only, may be post-review): Does the release version given match the GitHub release (v1.0.0)?

---

Editor comments

Thanks for your submission @ThierryO! This is an interesting package and it passes checks with nothing but this from goodpractice::gp():

  ✖ omit "Date" in DESCRIPTION. It is not required and it
    gets invalid quite often. A build date will be added to the package
    when you perform `R CMD build` on it.

I'm seeking reviewers for it now. FYI, because of the thorny nature of the problem your tackling - data versioning workflows - I anticipate a lot of discussion around high-level concepts and architecture. I'm going to look for reviewers with a lot of experience on these workflow issues, and encourage them to start that discussion before checking off all the low-level boxes so as to have an efficient and productive review.

---

Reviewers: @jennybc @brodieG
Due date: 04-01-2019

[noamross]

Reviewers assigned: @jennybc and @brodieG. At reviewers' request, final review date is extended to deal with travel and holidays. However, I request that reviewers provide some comment on high-level workflow/architecture issues that I think are likely to be the main points of sooner, before before a more detailed look at the code. Roughly:

• Is the expected workflow well communicated and implemented in the documentation and vignette?
• How does the workflow and model of the package work with other frameworks for versioning and reproducibility? Are concepts compatible and separable?
• Is the package API (relatively) intuitive and work well with the workflow?

[ThierryO]

Thanks for assigning the reviewers. I'm looking forward for their comments. There were some minor changes to the package. The latest version is available at ropensci/git2rdata#5

[brodieG]

Hi Thierry. This looks like an interesting package to review; looking forward to it.

One small request: do you mind creating tags/releases that we can work off of so there is no issue of the code changing under our feet as we are reviewing it? Ideally this would also be accompanied by a unique package version in DESCRIPTION. This will ensure we're all exactly on the same page about what version of the code is being reviewed. Thank you.

[ThierryO]

Good point. There is currently a single release tag v0.0.1.9000 which is the current HEAD of the master branch. I suggest that you use this for the review. I won't push to the master branch during the review process.

The current changes in the develop branch are mainly cosmetic, except one hotfix.

[brodieG]

Sounds good, I will do all of my work of of v0.0.1.9000.

[brodieG]

Overview

I haven't reviewed the internals of the package very closely yet, but overall I
like the concept and the initial cut at the implementation. There is good
attention to detail (e.g. helpful error messages), parameter checking, generally
good documentation, and good test coverage of the key functions. These all
demonstrate a commitment to robust programming practices.

At the request of the editor I'm going to focus on the top level questions first
rather than go through the step by step review form. I did take the liberty to
add some comments that fall outside of the scope of the questions asked, and
would appreciate some feedback from @noamross on which points should fall under
the scope of this review vs not. I'm finding this topic pretty interesting, but
I don't want to over-burden the author by extending the review into areas that
while interesting and related do not need to be addressed for the purposes of
the review.

Editor Questions

Q1

Is the expected workflow well communicated and implemented in the
documentation and vignette?

Vignette

The vignette provides end to end "toy" examples for both the versioned and
unversioned workflows. These examples give a good sense of how the functions
operate and how the user is expected to interact with them.

There are a few areas of a versioned workflow that would benefit from additional
exploration:

• An example where the versioned data is indeed used. All the examples here are
  commit-and-forget, whereas the true value of versioning the data is to be able
  to go back and compare.
• Related to the first point, for a fully reproducible workflow we need to be
  able to tie the data with the specific version of the code that used it as an
  input or produced it as an output. I would expect some discussion on how to
  address this problem (e.g. commit code, get commit hash, run analysis, store
  data with code commit hash in yml).

Documentation

The man documentation for most of the functions I looked at do not have usage
examples. They should to satisfy Ropensci's "multiple points of entry
principle". Also, while I don't think this is a strict CRAN requirement, I have
gotten feedback from CRAN maintainers that examples should be included.

Q2

How does the workflow and model of the package work with other frameworks for
versioning and reproducibility? Are concepts compatible and separable?

I do not have extensive experience with "reproducible workflows" beyond package
testing, so what follows is based on first principles rather than first hand
knowledge.

It strikes me that the package is more intertwined with a git workflow than it
needs to be. While write_vc and read_vc produce outputs that work well with
version control, there is no real need for an explicit link. For example, if
write_vc returned file paths instead of file hashes, we could just do:

git2r::add(path=write_vc(df, 'df'))

instead of:

write_vc(df, 'df', repo, stage=TRUE)

To me the semantics are much clearer in the first example, or at worst, just as
good as the second. The big difference is that you can strip away a large
amount of the complexity in your code. You could potentially even ditch S4 in
favor of S3 as the version control would be explicitly selected by the user,
e.g. in the case of an imaginary svnr package:

svnr::commit(write_cv(df, 'df'))

It might even be reasonable to completely remove the git2r dependency, with no
real loss in functionality. I'll freely admit it is entirely possible I'm
missing use cases where the tight integration with git2r is beneficial, but I
don't think I've seen clear examples of that. Please feel free to point them
out to me if they already exist and I missed them, or make some up for review.

One possible example is that we have to resolve the question of how to link the
data to the code as I suggested in Q1, which would be a little easier to do with
the repository object directly available, but I'm not sure that tilts the
balance back to the more complex implementation. Also, detection of unversioned
files would also be easier with the repository object. That said, neither of
this functionality currently exists, and the complexity to support the explicit
git integration does.

In terms of integrating with other vcs, the expectation seems to be implicitly
that someone will write an S4 method that supports it. That's a reasonably tall
ask when the alternative above is possible.

Aside: is there a particular value of the file hashes that I'm missing? I would
imagine in most cases people wont care what they are, and for when they do, they
can just call git2r::hashfile on the names anyway.

Q3

Is the package API (relatively) intuitive and work well with the workflow?

Yes, although there are some exceptions and possible improvements:

rm_data

The semantics of rm_data(..., type='tsv'), and rm_data(..., type='yml') are
different. For the former the tsv data files are removed, whereas for the
latter the yml files are removed, but only if there is not a corresponding
tsv file. This is only indirectly documented via the vignette examples.

We would recommend that you split up the behavior across functions, possibly
called:

• rm_data or rm_tsv to remove the data.
• prune_meta or prune_yml to remove the meta data that has no matching data
  file (I gather this is what rm_data(..., type='yml') does.

You could document the difference in behavior in the function itself as a
partial remedy, but the behavior is different enough that it feels as these
should be different functions so the semantics are clear.

Another issue to consider is whether you should allow the removal of unversioned
files without some kind of warning. I realize that if you unlink or
file.remove in R you'd better be sure you know what you are doing, but
particularly for data that is in a version controlled directory, but not itself
under version control, aggressive file deletion might be undesirable behavior.

Versioned vs. Unversioned Modes

As mentioned in Q2, I think it is detrimental to try to intertwine these more
than strictly necessary.

Possible Improvements

I would encourage you to create an intermediary git2rdata data type that
inherits from data frame, possibly with the meta data attached as an attribute.
Let's call the new data type git2rdat. We can then define (all in S3 here for
convenience):

as.git2rdat.data.frame
write_vc.git2rdat
write_vc.data.frame      # calls as.git2rdat.data.frame and forward
all.equal.git2rdat

Which then simplify the workflow in the vignette:

all.equal(read_vc(file = "git/test", root = repo), x)

Is now equivalent to:

y <- read_vc(file = "git/test", root = repo)
sorted.x <- x[order(x$x), colnames(y)]
rownames(sorted.x) <- NULL
all.equal(sorted.x, y, check.attributes = FALSE)

Additional Comments

More Workflow Comments

I think it would be helpful to have more aggressive defaults to facilitate the
workflow. For example, why not have the following:

write_vc <-
  function(x, file=deparse(substitute(x)), root=getOption('proj.root'))

Then things like the following just work:

git2r::add(write_vc(iris))
git2r::commit()

This would produce 'iris.tsv' and 'iris.yml' at getOption('proj.root').

Obviously you will need something more sophisticated than what I use above (e.g.
to handle bad names or automatically detecting a project root, or setting a
default data folder under the auto-detected project root), but I find the
simplicity of the above compelling.

Implications of Versioning Data in Git

@jennybc probably has a good view on this already, so I'll defer to her input here,
but while we wait for it here is my view on this:

Git is not really meant to store data frames, so I think it would be really
useful to have a more systematic assessment of whether this is a Bad Idea or
not, and what the limits are.

You mention "medium sized data". I suspect different people will have different
ideas of what that means. I recommend that you do some analysis of what the
implications are of repeatedly committing a data set of the maximal size you
intend your package to be used with. You should examine the effects on the size
and performance of the repository over enough commits you can provide guidelines
to your users about what it is okay to commit, and what is probably too much.
You should probably also compare the effects of commits with small changes, vs
large changes.

It may be that there is actually no problem with this at all, but it is unusual
enough that it seems worth covering. At a minimum, I would imagine prospective
users to be curious about it. There seem to exist several
frameworks that separate the data from the
repository so there probably is some reason for concern.

Rationale

I think the vignette would benefit from a more thorough analysis of the
rationale for the package. In the review issue you give the following as a
potential application:

E.g. each year new data is added to the data and a new report is created. A snapshot of the raw data can be stored as plain text files under version control.

Based on my experience this is not a great reason to use version control. The
data is effectively self versioned via the date variable so the version control
is redundant (certainly, the old data could change too, but even that can be
tracked explicitly in the data via a report ID for explicit analysis).

I feel that the version control is most useful for data that is not supposed to
change. The reason for this is that when data is supposed to change, the
natural thing to do is to analyze the changes to understand the meaning of the
changes. Diffs are terrible tools for this. They will tell you that number A
changed to number B, but will do so in a way that is very difficult to
meaningfully contextualize. This applies to tabular data in general. Unless
you are looking at a tiny data set, the table view is almost always useless.
Even iris with its puny 150 rows is almost always viewed as plots or otherwise
subjected to statistical computation.

On the other hand, for things that are not supposed to change, at least without
explicit review, knowing what changed is typically the biggest part of the
battle. There a diff based workflow can be very helpful.

I see two obvious applications:

• Combining data and code under VCS for a fully reproducible analytical
  pipeline.
• Regression tests involving data frames or other tabular data.

Clearly you have these things in mind, but it might be worth being more explicit
about the benefit of versioned data in the context of reproducible analysis, and
then explicitly highlight workflows that do that.

Research Of Comparable Solutions

I found:

• Data Versioning by Thomas J Leeper.
• Data Versioning Control by various.

More generally, it might be worth doing some level of literature review to see
what solutions exist already and whether they offer any insights on best
practices to apply to your package.

Thomas's paper covers some of this, and also referencing daff which I was
going to suggest separately. Edwin De Jonge wrote an R wrapper for it. I
have not personally used it, but I've done a fair bit of diffing of data frames
with line based diffs and honestly they do not work that well. Your idea of
sorting rows/columns will help to some extent, but you should assess whether it
makes sense to bring in a tabular diff.

Another idea raised in the same paper that I found interesting is to store data
in long format. It could obviate the need for daff by making all changes row
base, but will make the data larger. I don't know how much compression would
help. This obviously introduces additional complexity that in the end may not
justify itself, but I did find the idea of reducing the data to a form where row
based diffs would work perfectly very appealing.

Also interesting:

• git-media (possibly stale).
• git-annex (note, http URL).

I have not used any of these solutions, so I cannot vouch for their efficacy or
safety.

Storage format

• Should you be creating folders that contain the two files rather than
  stand-alone files? This gives you a lot more latitude in the future if you
  decide you want to add features. It's also a little cleaner.
• E.g. let's say you decided you also want to preserve row order as a newline
  separated data file.

Dependencies

Do you really need readr? You are dealing with a special case where you know
the origin of the data, so you are less likely to need the special features
offered by readr. Also, if you're explicitly targeting medium sized data, do
you need high performance on this? Unlike asserthat, readr carries along
many of its own dependencies. I would strongly encourage you to consider
whether write.table could do the job well enough, even if it requires a little extra work.

[noamross]

Thank you @brodieG for a pre-review as detailed as many of our full reviews! All of this is in-scope for the review. I think the discussion of other frameworks is quite helpful. One does not have to adopt a totally different model as a result, but I think using that context to guide the API and documentation . I find it increasingly common and helpful to put a "Related Work" section in README/vignettes to compare a package's approach to other options.

@ThierryO, I'd suggest waiting for @jennybc's input before making decisions and starting work on any major changes, but do respond to @brodieG's comments as you see fit.

[ThierryO]

Thanks you @brodieG for the elaborate review. Here are my idea's on the topics you raised.

Editors questions

Q1

Vignette

Adding an example where the data is actually used is a good suggestion. I have presented our workflow as a poster at the International Statistical Ecology Conference ISEC 2018. Please note that this poster predates the conception of git2rdata. A precursor of the functionality was then available in n2khelper.

Tracking analysis code is outside the scope of git2rdata. However, I do that in our n2kanalysis package. This package uses the output of git2rdata::recent_commit() as an entry point for the data. The code version is documented as a list of used packages and their version (include commit hash for github package). n2kanalysis has even a file and status fingerprints in each object which makes is possible to check the integrity of the object.

If we assume that the code to create and store the data.frames is all stored in a package foo, then gitr2data::auto_commit() can be used to document this. The commit message will read "Automated commit from 'foo'" and adds the full list of loaded packages and their versions at the moment of the commit. This is of course not tamper proof, but that is not the goal of git2rdata.

Documentation

If required, I can add some toy examples to the man documentation. How extensive should these examples be?

Q2

The initial idea of the package was to read and write data.frames into a git repository. Hence the strong link with Git and git2r. Then I had the idea that simply writing data.frames without version control (or to a different version control system than Git) might be relevant as well for other users.

Currently write_vc() returns hashes because I use these hashes in our workflow (they are used to calculate the file fingerprints). The file names are returned as the names of the hashes. So the examples with git2r::add() and svnr::commit() are currently possible when you write them as below. I agree that it is slightly more cumbersome. Returning a names vector with file paths and adding the hashes as names is an option.

git2r::add(path = names(write_vc(df, 'df')))
svnr::commit(names(write_cv(df, 'df')))

The current version has only methods depending on the root argument, making S3 a relevant option. Although it would require that root is the first and mandatory object. S4 allows to write methods for different arguments. E.g. we could think of methods for x objects that are something different than a data.frame (e.g. a SpatialPointsDataFrame).

Q3

rm_data

• A check on unversioned files when applying rm_data() on a git repository is a good idea. Maybe something similar like the override argument of write_vc()?
• Splitting rm_data() over two functions is a great suggestion.

Possible improvement

I'm not sure if there is much benefit of creating a new class. We often just want to write the object to a file / vcs and don't bother about any intermediate formats.

The all.equal() example is rather verbose because I want to stress the fact that write_vc() alters the order of the rows and the columns. I'm not sure that the user would need an all.equal() method for git2rdata. This equality should be warranted by the package itself (and is through unit tests).

Additional comments

More Workflow Comments

There already is a PR which implements root = "." as default. Another option would be to use rprojroot::find_root() to find a suitable root. A sensible order of defaults would be is_git_root, is_svn_root, is_r_package, is_rstudio_project.

file = deparse(substitute(x)) is problematic as it becomes file = "." when piping is used, as demonstrated in the example below. Furthermore, it assumes that people use informative names for their object (which they seldom do). I prefer to force the user to think about naming the file.

library(dplyr)
test <- function(x, file = deparse(substitute(x))) {
  message(file)
}
test(iris)
iris %>%
  test()

Implications of Versioning Data in Git

A separate vignette with guidelines and caveat on using git2rdata on a Git repository would be useful indeed. I'd like to postpone this. This allows me to try it on larger data sets and see were it breaks.

Rationale

In my line of work, I don't have any control over the raw data source. And hence cannot guarantee that I can reconstruct the raw data. I've been repeating the same analyses every year on several data sources. At least two of them had a major overhaul of the database structure during the past five years. And there is no version control within the database itself.

In the past we published results which were questioned by one of our stakeholders. They disagreed with the results and feared that the results would lead to a detrimental change in policy. The simple fact that we were able state that both the data and the code was placed under version control instantly increased our trustworthiness. After a walk through of the data and the code, they now agree on the methodology and hence the results.

So the focus is not on "the analysis of what changed in the data", but rather on "what data was used in this version of the analysis". Therefore git2rdata is just a tool in a fully reproducible analytically pipeline, not the full pipeline.

Research Of Comparable Solutions

I'll have a look at these suggestions over the next weeks.

Storage format

Storing each object in a dedicated folder is an interesting suggestion. Another benefit of this, is that we could allow the user to specify "file" extensions. E.g. write_vc(x, file = "test.txt") would store the data in test.txt/data.tsv and test.txt/metadata.yml.

Dependencies

An earlier version of git2rdata relied on write.table() and read.table(). The downside of that is, that storing characters was somewhat limited: NA, \t and \n were not allowed when storing them unquoted.

Personally, I like tibble over data.frame because of the superior print() method. Adding only the tibble dependency would also add at least fansi, fillar and utf8. In case you omit the dependencies in 'Suggested:' it will also required cli, crayon and rlang.

While thinking on adding tibble, readr can into view. I agree that it again adds several dependencies: BH, hms and pkgconfig (plus R6 and Rcpp which are only dependencies of Suggested dependencies of git2rdata).

However, I see several gains in using readr

1. read_vc() returns a tibble (agreed, rather a matter of taste than a critical improvement)
2. read_tsv() is faster than read.table() as claimed by the readr README, Efficient R programming, this blog, ...
3. write_tsv() handles NA and special characters efficiently by adding quotes only when really needed. And hence code that I don't have to write.
4. read_tsv() allows to specify the column type when reading the data.

IMHO, these gains outweigh the extra dependencies.