Skip to content

Extend lexer to cover non-ASCII unicode cases #242

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
graydon opened this issue Feb 25, 2011 · 0 comments
Closed

Extend lexer to cover non-ASCII unicode cases #242

graydon opened this issue Feb 25, 2011 · 0 comments

Comments

@graydon
Copy link
Contributor

graydon commented Feb 25, 2011

There are a few parts to this:

  • Implement the PEP 3131 rule for identifier tokenization: notice when you leave ASCII range, switch to XID_Start/XID_Continue, NFKC-normalize. This is expensive but it'll be a cold path in any ASCII-range input.
  • Copy of the rules in rustboot's handling of character and string literals (handle unicode escapes in those contexts, as well as general non-normalizing UTF-8 input).
@graydon graydon closed this as completed Feb 14, 2012
oli-obk added a commit to oli-obk/rust that referenced this issue Jul 19, 2017
@oli-obk
Merge pull request rust-lang#242 from solson/various
e362c30 
Add tests for rust-lang#113
kazcw pushed a commit to kazcw/rust that referenced this issue Oct 23, 2018
@gnzlbg @alexcrichton
More run-time detection improvements (rust-lang#242)
488c28a 
* [core/runtime] use getauxval on non-x86 platforms

* test coresimd::auxv against auxv crate

* add test files from auxv crate

* [arm] use simd_test macro

* formatting

* missing docs

* improve docs

* reading /proc/self/auxv succeeds only if reading all fields succeeds

* remove cc-crate build dependency

* getauxval succeeds only if hwcap/hwcap2 are non-zero

* fix formatting

* move getauxval to stdsimd

* delete getauxval-wrapper.c

* remove auxv crate dev-dependency from coresimd
dlrobertson pushed a commit to dlrobertson/rust that referenced this issue Nov 29, 2018
@Havvy
Merge pull request rust-lang#242 from matthewjasper/assoc-item-format…
708d8cd 
…ting

Hide more lines in code example
rchaser53 pushed a commit to rchaser53/rust that referenced this issue Jan 19, 2019
@mark-i-m
fix relative link (rust-lang#242)
3940e94
Aaron1011 pushed a commit to Aaron1011/rust that referenced this issue Oct 26, 2020
@dependabot
Merge pull request rust-lang#242 from bjorn3/dependabot/cargo/backtra…
a12fa76 
…ce-sys-0.1.28
ZuseZ4 added a commit to EnzymeAD/rust that referenced this issue Mar 7, 2023
@wsmoses @ZuseZ4
Handle ostream (rust-lang#242)
9e76494 
* Handle ostream

* add more printFnc to utils.h

* small refactor / formating

* addressing feedback

Co-authored-by: Manuel Drehwald <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@graydon