Skip to content
This repository was archived by the owner on Aug 12, 2020. It is now read-only.

fix(exporter): add some parallel fetching of blocks where possible #73

Merged
merged 2 commits into from
Sep 26, 2016
Merged

fix(exporter): add some parallel fetching of blocks where possible #73

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
43503d4
fix(exporter): add some parallel fetching of blocks where possible
dignifiedquire Sep 16, 2016
a9d303c
docs(exporter): describe directory exporter
dignifiedquire Sep 25, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,7 @@
"is-ipfs": "^0.2.0",
"multihashes": "^0.2.2",
"pull-block": "^1.0.2",
"pull-paramap": "^1.1.6",
"pull-pushable": "^2.0.1",
"pull-stream": "^3.4.5",
"pull-traverse": "^1.0.3",
Expand All @@ -70,4 +71,4 @@
"jbenet <[email protected]>",
"nginnever <[email protected]>"
]
}
}
15 changes: 9 additions & 6 deletions src/exporters/dir.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@

const path = require('path')
const pull = require('pull-stream')
const paramap = require('pull-paramap')

const fileExporter = require('./file')
const switchType = require('../util').switchType
Expand All @@ -14,15 +15,17 @@ module.exports = (node, name, dagService) => {
path: path.join(name, link.name),
hash: link.hash
})),
pull.map((item) => pull(
dagService.getStream(item.hash),
pull.map((n) => switchType(
paramap((item, cb) => dagService.get(item.hash, (err, n) => {
if (err) {
return cb(err)
}

cb(null, switchType(
n,
() => pull.values([item]),
() => fileExporter(n, item.path, dagService)
)),
pull.flatten()
)),
))
})),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dignifiedquire could you please add a bit of documentation to this function, each time I come to read it, I scratch my head to get back to understand what is supposed to be going on

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is /* this is where the magic happens */ enough?

Copy link
Contributor

@daviddias daviddias Sep 23, 2016 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no. If is helpful to have diagrams, add a file to explain the internals with those graphs :) The unixfs-engine is a big piece of the files API and it needs to be easy to understand and pluggable so that we can extend it easily with other layouts, customs chunkers and so.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@diasdavid added a description. Can we get this merged soon plese?

pull.flatten()
)
}
4 changes: 2 additions & 2 deletions src/exporters/file.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
const traverse = require('pull-traverse')
const UnixFS = require('ipfs-unixfs')
const pull = require('pull-stream')
const paramap = require('pull-paramap')

// Logic to export a single (possibly chunked) unixfs file.
module.exports = (node, name, ds) => {
Expand All @@ -18,8 +19,7 @@ module.exports = (node, name, ds) => {
function visitor (node) {
return pull(
pull.values(node.links),
pull.map((link) => ds.getStream(link.hash)),
pull.flatten()
paramap((link, cb) => ds.get(link.hash, cb))
)
}

Expand Down