Span issues with FromStr in unstable proc_macro

[mystor]

While writing improved tests for proc_macro2, I noticed some surprising behaviour of the FromStr impl for proc_macro::TokenStream.

When parsing a string with FromStr, a token parsed from the very beginning of the string can have a different source_file() than a token parsed elsewhere. For example:

let tokens = "aaa\nbbb".parse::<proc_macro::TokenStream>().unwrap().into_iter().collect::<Vec<_>>();
println!("span: {:?}", tokens[0].span);
println!("source_file: {:?}", tokens[0].span.source_file());
println!("");
println!("span: {:?}", tokens[1].span);
println!("source_file: {:?}", tokens[1].span.source_file());

Will produce the output:

span: Span(Span { lo: BytePos(6734829), hi: BytePos(6734832), ctxt: #0 })
source_file: SourceFile { path: ProcMacroSourceCode, is_real: false }

span: Span(Span { lo: BytePos(50), hi: BytePos(58), ctxt: #25 })
source_file: SourceFile { path: Real("pm_test.rs"), is_real: true }

I would've expected both tokens to have the same context, and similar BytePos-es. Specifically I would have expected both of these tokens to have a span located in a fake ProcMacroSourceCode file, but set to resolve in the context of the enclosing scope (as if span.resolved_at(Span::call_site()) was called).

[mystor]

I should note that adding a ' ' (space) character to the beginning of the string causes all tokens to have spans matching the second one.

[mystor]

I think the cause of the first-span-being-different issue is

rust/src/libsyntax/parse/lexer/mod.rs

Lines 202 to 209 in 178becd

	pub fn new(sess: &'a ParseSess, filemap: Lrc<syntax_pos::FileMap>) -> Self {
	let mut sr = StringReader::new_raw(sess, filemap);
	if sr.advance_token().is_err() {
	sr.emit_fatal_errors();
	FatalError.raise();
	}
	sr
	}

, which causes the first token to be advanced-to and peeked at before the override_span is set here:

rust/src/libsyntax/parse/mod.rs

Line 230 in 604d4ce

srdr.override_span = override_span;

The cause of the issue where all parsed tokens have the line and column of the input file instead of lines and columns into a fake proc-macro object seems to be caused by the way that override_span is used - namely how it overrides the entire span instead of just the expansion context.

[oskgo]

@rustbot label +E-needs-test

Does not reproduce anymore. A bisect reveals that it was fixed by #50924.

The fix didn't add any tests.