fragment insertion

[sjanssen2]

I hope this is the right place to discuss how we want to integrate fragment-insertion AKA SEPP into Qiita.

I think we agree on the fact that we somehow want to cache placements of sequences into one given reference. Current only reference is GreenGenes 13.8 as provided in Siavash's repo
Jose's suggestion was to use one hd5f file to store placements. A placement looks like this:

placement = {
      "p": [[6454, -96408.34, 0.8250844, 0.0000070187216, 0.04252309],
        [6564, -96410.87, 0.066086836, 0.000006061417, 0.04802516],
        [6448, -96411.98, 0.02176576, 5.000002E-7, 0.048468385],
        [6449, -96411.98, 0.02176576, 5.000002E-7, 0.048468385],
        [6450, -96411.98, 0.02176576, 5.000002E-7, 0.048468385],
        [6447, -96411.98, 0.02176576, 5.000002E-7, 0.0484684],
        [6440, -96411.98, 0.021765757, 5.000002E-7, 0.048468385]],
      "nm": [["CGACGTGTGTACGTGTAGTGTCGGGATCGTAGTCGTAGTCGTAGTCGTAGTCGTGTGTACGTGTAGTCGTAGTC",
          1]]
    }

where placement["nm"][0][0] is the fragment sequence, and placement["p"] holds alternative placements. One of those placements is chosen by SEPP to be inserted into the reference tree later on. The first component, e.g. 6454 points to the node of the reference tree, i.e. the Greengenes tree.

Thus, I'd like to suggest the following data organization:

h5py.File('test1.hdf5', 'w').create_dataset(
     'gg13.8/%s' % placement["nm"][0][0], data=placement["p"])

I am a hd5f newbie, so please double check if this is a good design!

My suggestion would be to make SEPP a part of the deblur-qiita plugin. We could first check which sequences are already placed (a lookup to the above storage) and then run SEPP on the reduced set per biom table. Newly found placement must be reported back to the central storage, of course.

Let us defer the details of how to expose the actual insertion tree (i.e. the result of adding fragments according to placements into the reference, via program guppy tog) for now. There are alternative routes, @tanaes might want to comment on that.
For now, I think we are good to create one tree per deblur table and/or one tree per meta-analyses out of the placements from central storage.

[sjanssen2]

@josenavas could you point me to the right sources of Qiita where I could start looking into implementing such a central storage unit?

[josenavas]

Hi @sjanssen2

I'm unsure on how everything will look like - what I do know is that there is nothing like this in the system hehehe.

There are a lot of moving parts in here, and I think the 1st step is to decide which is the best way of storing this information. I suggested HDF5 because I'm a bit nervous on storing sequences in the DB if we don't have an estimate on how such database is going to be growing. Thus, I think the first step would be to decide if HDF5 is the best way to go or we can actually safely store it in a nice DB structure. Information about size (i.e. how much sequence data is going to be stored) and which type of queries need to be supported are going to be important to make a decision. For example, one query would be "given a list of sequences, give me all the placements for each element of the list". Another query, which I don't know if it may be useful, would be "given internal Greengenes node X, give me all the sequences that could be placed here". If the second query is something desirable, then the entire design (either of a DB or an HDF5) is different.

Also, in terms of describing an HDF5 file format, I would recommend describing it following the demux file DDL, which @wasade's put together and it is really useful to understand how the file works.

Also, I would like to hear the input from @antgonza, @wasade and others to see which is the best storage system that we can use for this.

[sjanssen2]

Thanks @josenavas for your thoughts. Maybe I can go ahead and just assume that there fill be a function A that returns a "dict" of placements (key=sequence, value=placement) for a "list" of sequences and another function B to which I pass a "dict" of placements to store found placements in whatever system?

[antgonza]

This was solved during the last release.