Approximate match (edit distance) support for UTF8 character set

[johnwyz88]

Hi,

Can someone explain why approximate match currently doesn't support UTF8 characters? Is it possible to provide such support easily?

[hongyang7]

The approximate match is supported by expanding the number of NFA graph vertices according to the edit distance, which already increases the space pressure of NFA/DFA generation (Hyperscan have state limitation for them).

As for UTF8 mode, things are even worse. E.g, a simple "." in UTF8 mode no longer interpreted as characters in ASCII table but all the code points ranging from 1 byte to 4 bytes which are super large. Even a bounded repeat pattern ( /(*UTF8)x.{73}/ ) with not large repeats can generate terrible state quantity. (refrer to #132)

That explains the difficulty to provide complex syntax structure support for UTF8 patterns.