Make server component separate and this module just install npm deps #70

achingbrain · 2018-10-17T06:18:25Z

I've moved the replication/mirroring functionality out of this module into ipfs-registry-mirror because it was getting very unwieldy and was starting to lose focus - this is what's currently deployed at https://registry.js.ipfs.io

This PR fixes #50 by making the purpose of this module just to install your npm deps.

When invoked on the command line, this module now:

Makes a request to https://replication.registry.js.ipfs.io to find out what the current CID of the npm index is
Dials the replication master to ensure we can ipfs.files.cp the updated index
Updates the local copy of the registry index
Starts a web server to serve dependencies
Spawns npm in the background configured to connect to us
For every tarball/manifest request, if it exists in our copy of the index, uses the CID stored alongside it to fetch it using IPFS, if not proxies for the public npm registry but updates the local copy of the index with the CID of the newly downloaded module

Some notes:

1 & 2 will go away when js-ipfs can resolve remote IPNS names
There seems to be a bug with ipfs.dag.get, in that it stops responding when invoked loads of times concurrently, but I need to dig in to this further. Until then this module is best used with the go daemon. E.g. in your project directory and with go-ipfs on your $PATH:

ipfs-npm --ipfs-type=go

Requesting modules from localhost results in having http://localhost... in your package-lock.json file, might need to rewrite these post-install and replace them with https://registry.js.ipfs.io urls..

daviddias · 2018-10-17T11:26:06Z

README.md

+# npm on IPFS

-![](/img/ip-npm-small.png)
+![Alt text](./img/npm-on-ipfs.svg)


It is a bit big. Did you have a preferred size in mind?

see the js-ipfs repo for a good size

daviddias · 2018-10-17T11:26:24Z

README.md

-===========
+# npm on IPFS

-![](/img/ip-npm-small.png)


This image was great btw

Do you prefer it to the new image? Since #55 was opened I thought you wanted it updating?

see the js-ipfs repo for a good size

Yeah, but the network of multiple npm registries is still super cool. Can we meet in the middle?

Sure, I'll put it back in :)

daviddias · 2018-10-17T11:26:59Z

README.md

-🔧 Please either update your npm config with 'npm config set registry http://localhost:50321'
-🔧 or use the '--registry' flag, eg: 'npm install --registry=http://localhost:50321'
+$ ipfs-npm
+👿 Spawning a go-IPFS node


no default to JS anymore?

It defaults to an in-process js-ipfs node. The output here is from a testing run, I missed that it was with a go-ipfs node, will update.

daviddias · 2018-10-17T11:27:08Z

README.md

-🔧 or use the '--registry' flag, eg: 'npm install --registry=http://localhost:50321'
+$ ipfs-npm
+👿 Spawning a go-IPFS node
+🗑️  Removing old registry if it exists


Removing or updating?

It removes the old one and replaces it with the latest from the replication master. Once it's finished it's first import run (might take a week or two), it should have everything you'd ever need. N.b. it's just the json package descriptions you're pulling down, not the tarballs.

Perhaps say something as "replacing old registry index if it exists"

daviddias · 2018-10-17T11:28:36Z

README.md

+🗑️  Removing old registry if it exists
+☎️  Dialing replication master /ip4/127.0.0.1/tcp/40020/ipfs/QmeXyYCLSivUn5Ju31jjPBYNKdncbzzEf6zdN2DyrdLAbe
+📠 Copying registry index /ipfs/QmQmVsNFw3stJky7agrETeB9kZqkcvLSLRnFFMrhiR8zG1 to /commons-registry
+👩‍🚀 Starting local webserver


perhaps say "local proxy"

daviddias · 2018-10-17T11:28:52Z

README.md

+👿 Spawning a go-IPFS node
+🗑️  Removing old registry if it exists
+☎️  Dialing replication master /ip4/127.0.0.1/tcp/40020/ipfs/QmeXyYCLSivUn5Ju31jjPBYNKdncbzzEf6zdN2DyrdLAbe
+📠 Copying registry index /ipfs/QmQmVsNFw3stJky7agrETeB9kZqkcvLSLRnFFMrhiR8zG1 to /commons-registry


Why not do npm-registry?

Throwback to #56. Will update.

this is just a folder name

daviddias · 2018-10-17T11:29:21Z

README.md

+🎁 Installing dependencies with /Users/alex/.nvm/versions/node/v10.8.0/bin/npm
+🆕 New version of express detected undefined vs 3864-53d75cca2a8eae32408fce2fd43f4c87
+📄 GET /express 200 993ms
+🐨 npm http fetch GET 200 http://localhost:57314/express 1029ms


Why does it try to install express on setup?

It doesn't, this was output using ipfs-npm in a project that had express as a dependency.

daviddias · 2018-10-23T13:43:50Z

README.md

-📠 Copying registry index /ipfs/QmQmVsNFw3stJky7agrETeB9kZqkcvLSLRnFFMrhiR8zG1 to /commons-registry
-👩‍🚀 Starting local webserver
+👿 Spawning an in-process IPFS node
+🗑️ Replacing old registry index if it exist


daviddias · 2018-10-23T15:16:40Z

README.md

 # npm on IPFS

-![Alt text](./img/npm-on-ipfs.svg)
+![npm on IPFS logo](./img/npm-on-ipfs.svg)


nicpick, can you make it centered. Just like libp2p https://github.com/libp2p/js-libp2p?

codecov-io · 2018-10-25T07:37:02Z

Codecov Report

❗ No coverage uploaded for pull request base (master@6ea40aa). Click here to learn what that means.
The diff coverage is 81.06%.

@@            Coverage Diff            @@
##             master      #70   +/-   ##
=========================================
  Coverage          ?   75.13%           
=========================================
  Files             ?        8           
  Lines             ?      185           
  Branches          ?        0           
=========================================
  Hits              ?      139           
  Misses            ?       46           
  Partials          ?        0

Impacted Files	Coverage Δ
src/core/config.js	`100% <100%> (ø)`
src/cli/bin.js	`100% <100%> (ø)`
src/core/handlers/root.js	`42.85% <33.33%> (ø)`
src/core/start-ipfs.js	`56.52% <56.52%> (ø)`
src/core/index.js	`82.75% <81.81%> (ø)`
src/core/handlers/tarball.js	`51.72% <83.33%> (ø)`
src/core/handlers/manifest.js	`66.66% <85.71%> (ø)`
src/core/start-server.js	`97.14% <97.14%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6ea40aa...1fe471e. Read the comment docs.

achingbrain added 23 commits October 7, 2018 13:17

feat: add registry cloner

2af14ba

chore: split into separate modules

ca17768

fix: dial all available addresses

ddd5999

feat: copy and paste like an animal

2bf789a

feat: show root cid on registry mirrors

318e68e

fix: change module path

05c578a

fix: update config

085e875

fix: correct deploy script

c7eae4b

fix: make env prod

b72aaf0

fix: configure external ip

05d0b06

fix: correct url

cc64500

test: adds lots of tests

8a75368

test: test all the things

8e644b8

chore: fix linting issues

bb10fb9

test: run tests in parallel

b37393a

chore: update readme

38efffe

Merge remote-tracking branch 'origin/master' into split-into-modules

1cd21ad

chore: update image

da29363

chore: move services out of project

6c75058

fix: update paths and only delete local reg when we have dialled master

8cb5bfa

fix: use cached index if master cannot be dialled

3684624

chore: change description and remove old binary

45acd2d

chore: restore files

8472d70

achingbrain mentioned this pull request Oct 17, 2018

fix: not eager download by default (mea culpa) #69

Closed

daviddias reviewed Oct 17, 2018

View reviewed changes

achingbrain added 3 commits October 23, 2018 11:22

chore: addres PR comments

606d3b9

chore: fix whitespace

afe7a3d

chore: update readme

ebe84dd

daviddias reviewed Oct 23, 2018

View reviewed changes

feat: pass extra commands to package manager

3861272

daviddias approved these changes Oct 23, 2018

View reviewed changes

achingbrain added 7 commits October 23, 2018 16:24

fix: use correct arg

5bbe609

chore: centre images

a0c41f6

chore: centre images

8cd5d72

chore: fix up images

e27a54a

chore: move headings around

f59f851

fix: add flush and fix tests

877053d

fix: dial mirror instead of replication master

ce56654

feat: rewrite lockfiles with repo url

1fe471e

achingbrain merged commit 1100ff6 into master Oct 25, 2018

achingbrain deleted the split-into-modules branch October 25, 2018 15:24

Make server component separate and this module just install npm deps #70

Make server component separate and this module just install npm deps #70

Uh oh!

Conversation

achingbrain commented Oct 17, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codecov-io commented Oct 25, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

codecov-io commented Oct 25, 2018 •

edited

Loading