Add support for multiple sitemaps

[jsha]

Per https://support.google.com/webmasters/answer/183668?hl=en&ref_topic=4581190,

All formats limit a single sitemap to 50MB (uncompressed) and 50,000 URLs.

The current sitemap at https://docs.rs/sitemap.xml is 5.3MB (fine) and 41,000 URLs. Soon the size of the Rust ecosystem will grow to the point that additional sitemaps are needed.

[jsha]

I saw an announcement today that crates.io had reached 50,000 crates: https://twitter.com/rustlang/status/1331020962628767748. That made me think of this issue again. The sitemap is still only at 41,617 URLs, though. Purely out of curiosity, any idea why the discrepancy?

[jyn514]

Hmm, that sounds like the sitemap is buggy:

cratesfyi=> select count(*) from crates;
 count 
-------
 50938

[njasm]

Out of those 50k, how many of them have their latest version yanked?

(I expect not have crates in the site map where the latest published version is yanked)

[jyn514]

Rather than guessing, you can just read the source ;)

docs.rs/src/web/sitemap.rs

Lines 27 to 32 in 31c864e

	"SELECT DISTINCT ON (crates.name)
	crates.name,
	releases.release_time
	FROM crates
	INNER JOIN releases ON releases.crate_id = crates.id
	WHERE rustdoc_status = true",

The difference is that docs.rs doesn't include crates that failed to build in the sitemap.

[njasm]

sweet! :)

[syphar]

I created #1222 as a potential solution, up to discussion of course.

[camelid]

Should this be closed since #1222 has been merged?

[jyn514]

Let's wait until #1222 is deployed.

[Nemo157]

Is there some way to verify google's view of the sitemaps after deployment?

[syphar]

in google webmaster tools, yes.

Also you can ping them to directly load the sitemap and see the result, if I remember correctly.