Eligiblity criteria for meeting bug bounties

[lirantal]

As a follow-up to #475 (comment) we'd like to establish a transparent and collaboratively set criteria of how bug bounties should be carried out by individuals of the Security WG towards the community (bug researchers or package owners).

Criteria levels

Criteria levels will allows us to prioritize specific reports for full, partial or no eligibility of bounties.

1. Yes/No - whether or not a security report is eligible for a monetary recognition
2. Amount of bounty - a percent of the monetary recognition

Criteria (WIP)

1. Module download count threshold
2. Module dependents count
3. Vulnerability type - as in, DoS or Injection could yield high criteria fit, while lesser impactful vulnerabilities would yield lower fit. In contrast, we can just base this on the CVSS score as thresholds.
4. Maintainer opt-in - maintainers are positive in working on a fix and joining the conversation on the security issue and how to mitigate it.

[lirantal]

@MarcinHoppe @shivasurya @reedloden @sam-github @mhdawson lets work out further ideas and criteria to put this place. This is an important effort regardless of the warm coinbase support as it will set the basis and foundation for hopefully more programs to support us.

[shivasurya]

Sure.

[MarcinHoppe]

@lirantal It is awesome that we can start fleshing this out.

I'd add maintainer opt-in to eligibility criteria, i.e. we don't want to pay when maintainers are not in a position to remediate vulnerabilities in a timely manner.

[lirantal]

@MarcinHoppe it's times like this that I feel GitHub could use the help of threaded discussions :-)

I'd add maintainer opt-in to eligibility criteria, i.e. we don't want to pay when maintainers are not in a position to remediate vulnerabilities in a timely manner.

Explain rational? Why would a security researcher's job be impacted by a maintainer's ability to respond in a timely manner? Let's take event-stream as a use-case. Theoretically speaking and a bit of what happened there:

• the maintainer had stopped active maintenance
• a security researcher would find an issue there
• we would try to contact the maintainer of event-stream, but he isn't actively working on it so high chances they'd miss our github issues or emails, and can assume they may not respond in a timely manner
  Would we not want to recognize a researcher who found a vulnerability in a module affecting a relatively large part of the ecosystem?

Also, maintainer opt-in is hard enough to get and we'd be handling communication all day long instead of triaging and doing other parts of our scope.

I think it's great if we actually recognize maintainers themselves who are able to respond in a timely manner and work with us smoothly on this, and in this way incentives other maintainers to follow the same route, as well as increase awareness.

[shivasurya]

Recognizing the valid and legit efforts of the researcher ( either by bounty/points/swag ) should be always encouraged, if not this will affect other security researchers to stop searching or analyzing npm/nodejs bugs.

//offtopic

1. can we bring down the search rank of npm module if the maintainer doesn't respond/ doesn't fix the bug even if it's a valid bug. ( this will certainly stop or rethink users from using it )

[MarcinHoppe]

@lirantal We discussed this a while back in #475 (comment). I was under the impression we reached consensus there, but I am happy to have this discussion one more time :).

As for threaded discussions, I think PRs are a decent place to have those :).

[mhdawson]

I agree with @MarcinHoppe. I had thought we agreed to focus on modules who opt in as a starting point. It's a matter of getting the best return on the investment. In my mind what we learn working with modules that opt-in will then let us figure out if it makes sense to extend to modules who do not....

[lirantal]

I'm not saying I disagree but I would ask what are we trying to optimize for with the budget? if we'd want to encourage researchers to be more active on disclosing to the Security WG than other alternatives (i.:e Snyk/npm), or something else?

Gonna add it to the criteria which we will anyway visit again in the future but we need to flesh out what does it mean to "opt-in" exactly.

[mhdawson]

I think we want to optimize for getting reports on Modules that will fix the reported issues. On "opt-in" we'll need a process by which module maintainers can confirm they would like to be included. The simplest example would be by opening a PR to add to a list of modules which are covered by the bounties.

[lirantal]

Not disagreeing with this but I think it can be quite complex to manage to - it is more lists that needs to be kept up to date. Plus, we're talking about open source projects and maintainers which means we can add package X today but in 6 months there's a report coming in and the maintainer is gone and moved onto other things and isn't responsive.

I'm ok with trying this out. We're at an early exploration phase and it's good to experiment and adjust as we learn how this work :)

[lirantal]

Update: funds are in! thank you so much Coinbase ❤️🙏

Currently not having a criteria is blocking us from being able to reward researchers for their efforts.
What I suggest we do is:

1. Instead of trying to get a comprehensive list of all available criteria and agreement upon them all let's favour an iterative process so that we are able to iterate small increments of criteria and keep the ball rolling
2. Let's start with the list of criteria we have now and get an agreement on. Following agreement on that I'll create a new process document and we can iterate on it.

@nodejs/security-wg

[sam-github]

I think there has been enough discussion to PR criteria text. Its easier to thumbs up or down (or comment on) a specific piece of text.

Let's start with the list of criteria we have now and get an agreement on.

What we have now is the ones in the issue description? It looks OK in principle to me, but do actual numbers have to be added to 1. and 2.? I'm not really clear on what the process is, and what we need to produce as an input to it.

Is it worth just PRing a list of modules we care about? Or are we only going to publish guidelines, and then after a vuln is reported evaluate whether the vuln is bounty worthy? People might want to know up front if the module they are looking at will get bounties.

[vdeturckheim]

@sam-github for simplicity, in a first round I'd rather have a list of module for which we know maintainers

[sam-github]

OK, that sounds like unanimous agreement for starting with a short list.

[lirantal]

WDYT about the following?

1. Module download count - x >= 1000 downloads a month which accounts for 7% of npm packages (courtesy of @ChALkeR here Optimize for impact (including downloads/month) #151 (comment))
2. Module dependents count - vote to ignore for now as we don't have enough experience to gauge what this means
3. Vulnerability type - Either ignore it for now, or instead have a criteria based on vulnerability severity rather than type, so to match anything >= 4.0 which means Medium and higher.
4. Modules - @vdeturckheim it's not just us choosing them, we also need to get opt-in from module maintainers so that list at this point is more of a wishlist, not a defined one to start with.

[sam-github]

LGTM.

What is "dependents" for module A? It is count of modules which depend on A? Or count of module A depends on? I suspect the former, and I'm not sure we need to consider it, because if 500 modules depend on A, but it still doesn't get more than 1000 downloads a month, we don't care. I think the download count probably gives enough sense of its importance.

[lirantal]

I was thinking about the former indeed, and yeah we can probably leave it out for now.

@sam-github can you +1 on #503 (comment) so we can easily track it?
@nodejs/security-wg happy to get +1 from you on the comment here #503 (comment) as first criteria to get started with the bounty program

[mhdawson]

I'm +1 on #503. I think what we need is a PR capturing the requirements, along with the list of modules to which the bounties apply (which will be empty to start). Once that initial text lands, we can do a public call for module owners to "opt-in" provided that they meet the requirements. We can also reach out specifically to modules we think are important to encourage them to opt-in.

[lirantal]

👍 will PR a new doc for that in the next couple of days

[sam-github]

This was added to the agenda 2 months ago, but was already discussed. Perhaps should move off agenda?

[lirantal]

Oh indeed, I already merged the first proposal for that.