Code

Major overhaul of mbstring (part 1)#6052

Code

Closed

alexdowad wants to merge 123 commits intophp:masterphp/php-src:masterfrom

alexdowad:cleanup-mbstring-1alexdowad/php-src:cleanup-mbstring-1Copy head branch name to clipboard

Commits

Commits on Nov 3, 2020

• Fix mbstring support for Shift-JIS

  Show description for af8d6bf
  alexdowad

  committed
  af8d6bf

  Copy full SHA for af8d6bf

Commits on Nov 4, 2020

• Add test suite for Shift-JIS encoding

  alexdowad

  committed
  ffb01c9

  Copy full SHA for ffb01c9

• Add identify filter for Shift-JIS-2004

  Show description for b96b1f4
  alexdowad

  committed
  b96b1f4

  Copy full SHA for b96b1f4

• SJIS-2004 encoding conversion: handle invalid (or truncated) 2nd byte for Kanji correctly

  Show description for 714a7d1
  alexdowad

  committed
  714a7d1

  Copy full SHA for 714a7d1

• Don't mangle non-Japanese chars which appear after a 'combining' kana in SJIS-2004

  Show description for 9ece2b3
  alexdowad

  committed
  9ece2b3

  Copy full SHA for 9ece2b3

• Add test suite for SJIS-2004 encoding

  alexdowad

  committed
  4bba852

  Copy full SHA for 4bba852

• Add identify filter for MacJapanese (variant of Shift-JIS)

  alexdowad

  committed
  0c0a4b8

  Copy full SHA for 0c0a4b8

• SJIS-mac encoding conversion: handle invalid (or truncated) 2nd byte for Kanji correctly

  Show description for 55dcf81
  alexdowad

  committed
  55dcf81

  Copy full SHA for 55dcf81

• Convert Unicode halfwidth Yen sign to MacJapanese halfwidth Yen sign

  Show description for 4feb701
  alexdowad

  committed
  4feb701

  Copy full SHA for 4feb701

• SJIS-mac encoding conversion: Stop the carnage of innocent Unicode codepoints

  Show description for 6cd472a
  alexdowad

  committed
  6cd472a

  Copy full SHA for 6cd472a

• Add test suite for SJIS-mac encoding

  alexdowad

  committed
  0a6e568

  Copy full SHA for 0a6e568

• Enhance mbstring support for UCS-4 text

  Show description for 86e927a
  alexdowad

  committed
  86e927a

  Copy full SHA for 86e927a

• Leading BOM is stripped for UTF-32

  Show description for 431c4b9
  alexdowad

  committed
  431c4b9

  Copy full SHA for 431c4b9

• Enhance mbstring support for UCS-2 text

  Show description for a38a4a0
  alexdowad

  committed
  a38a4a0

  Copy full SHA for a38a4a0

• Consolidate all single-byte encodings in one source file

  Show description for 675a311
  alexdowad

  committed
  675a311

  Copy full SHA for 675a311

• Fix identify filter for UTF-7

  Show description for f6e834f
  alexdowad

  committed
  f6e834f

  Copy full SHA for f6e834f

• Minor code cleanup in mbfilter_utf7.c

  alexdowad

  committed
  02fc58e

  Copy full SHA for 02fc58e

• Catch and handle errors in UTF-7 text conversion

  alexdowad

  committed
  0879484

  Copy full SHA for 0879484

• Add test suite for UTF-{7,8,16,32}

  alexdowad

  committed
  a825801

  Copy full SHA for a825801

• Add identify filter for modified UTF-7 (for IMAP protocol)

  Show description for 2c0098d
  alexdowad

  committed
  2c0098d

  Copy full SHA for 2c0098d

• Add test suite for mUTF-7 (UTF7-IMAP) encoding

  alexdowad

  committed
  7cbe62b

  Copy full SHA for 7cbe62b

• Add identify filter for CP50220-raw

  Show description for cbfc02f
  alexdowad

  committed
  cbfc02f

  Copy full SHA for cbfc02f

• Add identify filter for 'HTML entities' encoding

  Show description for 5d6ecd3
  alexdowad

  committed
  5d6ecd3

  Copy full SHA for 5d6ecd3

• Add identify filter for 'byte2be' and 'byte2le' encodings

  alexdowad

  committed
  8e9d740

  Copy full SHA for 8e9d740

• Add identify filter for 'byte4be' and 'byte4le' encodings

  alexdowad

  committed
  f3e730c

  Copy full SHA for f3e730c

• Add identify filter for uuencode 'character encoding'

  Show description for fb9a1d9
  alexdowad

  committed
  fb9a1d9

  Copy full SHA for fb9a1d9

• Add identify filter for QPrint 'character encoding'

  alexdowad

  committed
  08a064b

  Copy full SHA for 08a064b

• Add identify filter for Base64 'character encoding'

  alexdowad

  committed
  5550c71

  Copy full SHA for 5550c71

• Stricter identification of valid strings in ISO-2022-JP-2004 encoding

  alexdowad

  committed
  32bb947

  Copy full SHA for 32bb947

• ISO-2022-JP-2004 conversion: handle invalid characters correctly

  alexdowad

  committed
  b374321

  Copy full SHA for b374321

• Add test suite for ISO-2022-JP-2004 encoding

  alexdowad

  committed
  3dfc88f

  Copy full SHA for 3dfc88f

• Add comment explaining why ISO-2022-JP-2004, etc strings end with ESC ( B

  Show description for 2e06352
  alexdowad

  committed
  2e06352

  Copy full SHA for 2e06352

• Stricter identification of valid strings in JIS7/JIS8 and ISO-2022-JP encodings

  alexdowad

  committed
  f3e3ee5

  Copy full SHA for f3e3ee5

• JIS7/8 encoding: handle invalid 2nd byte for Kanji correctly

  alexdowad

  committed
  1d10932

  Copy full SHA for 1d10932

• Add test suite for JIS7/JIS8 and ISO-2022-JP encodings

  alexdowad

  committed
  5321788

  Copy full SHA for 5321788

• Update 'East Asian Width' table to comply with Unicode 13.0

  Show description for 92dd38f
  alexdowad

  committed
  92dd38f

  Copy full SHA for 92dd38f

• Correct wrong flags for 'byte2be', 'byte2le', 'byte4be', and 'byte4le' encodings

  Show description for 360187e
  alexdowad

  committed
  360187e

  Copy full SHA for 360187e

• Enhance handling of EUC-JP text encoding

  Show description for 938b517
  alexdowad

  committed
  938b517

  Copy full SHA for 938b517

• Add test suite for EUC-JP encoding

  alexdowad

  committed
  eaca866

  Copy full SHA for eaca866

• Add identify filter for text encoding 'wchar'

  alexdowad

  committed
  ae0f229

  Copy full SHA for ae0f229

• All mbstring encodings have identify filter now

  alexdowad

  committed
  215c146

  Copy full SHA for 215c146

• Tell compiler which way jumps will usually go in mbfl_memory_device.c

  Show description for 934a2d8
  alexdowad

  committed
  934a2d8

  Copy full SHA for 934a2d8

• Simplify code for handling mbstring language aliases

  Show description for 2c6db2c
  alexdowad

  committed
  2c6db2c

  Copy full SHA for 2c6db2c

• Major refactoring and optimization of mbfilter.c

  Show description for 7874786
  alexdowad

  committed
  7874786

  Copy full SHA for 7874786

• Remove useless struct: mbfl_filt_tl_jisx0201_jisx0208_param

  Show description for c0eeafd
  alexdowad

  committed
  c0eeafd

  Copy full SHA for c0eeafd

• Minor code cleanup in php_unicode.h

  Show description for 54e189f
  alexdowad

  committed
  54e189f

  Copy full SHA for 54e189f

• Remove unneeded function mbfl_convert_filter_copy from mbstring

  Show description for 35a2cec
  alexdowad

  committed
  35a2cec

  Copy full SHA for 35a2cec

• Optimize mbstring upper/lowercasing: use fast path in more cases

  Show description for 163815a
  alexdowad

  committed
  163815a

  Copy full SHA for 163815a

• Fix buggy support for negative 'start' argument to mb_strimwidth

  Show description for 9930b0c
  alexdowad

  committed
  9930b0c

  Copy full SHA for 9930b0c

• Remove broken test for validity of 'start' argument to mb_strimwidth

  Show description for 809351c
  alexdowad

  committed
  809351c

  Copy full SHA for 809351c

• Remove unneeded function mbfl_filt_tl_jisx0201_jisx0208_init

  alexdowad

  committed
  05ad087

  Copy full SHA for 05ad087

• Remove unneeded struct: mbfl_filt_conv_wchar_cp50220_ctx

  Show description for fab2151
  alexdowad

  committed
  fab2151

  Copy full SHA for fab2151

• mbfl_wchar_device_init accepts initial buffer size argument

  Show description for 20de301
  alexdowad

  committed
  20de301

  Copy full SHA for 20de301

• Remove unused COMPAT2 constants from mbstring

  Show description for a5e5430
  alexdowad

  committed
  a5e5430

  Copy full SHA for a5e5430

• Rename 'COMPAT1' constants for mbstring to 'SPECIAL' (more meaningful name)

  Show description for db69a19
  alexdowad

  committed
  db69a19

  Copy full SHA for db69a19

• Remove unused MBFL_FILT_TL_{HAN2ZEN,ZEN2HAN}_MASK constants

  alexdowad

  committed
  48e390f

  Copy full SHA for 48e390f

• Rename MBFL_FILT_TL_ZEN2HAN_{HIRA2KANA,KANA2HIRA} to accurately reflect what they do

  Show description for fd03267
  alexdowad

  committed
  fd03267

  Copy full SHA for fd03267

• Disallow nonsensical combinations of conversion flags to mb_convert_kana

  Show description for 6a74bb4
  alexdowad

  committed
  6a74bb4

  Copy full SHA for 6a74bb4

• Return values of 'filter flush' functions are never used

  Show description for e612f38
  alexdowad

  committed
  e612f38

  Copy full SHA for e612f38

• Canonicalize flags received by mb_convert_kana

  Show description for 7cf60e0
  alexdowad

  committed
  7cf60e0

  Copy full SHA for 7cf60e0

• Don't redundantly flush filter chain twice in mbfl_convert_filter_flush

  Show description for d2d0c1c
  alexdowad

  committed
  d2d0c1c

  Copy full SHA for d2d0c1c

• Remove unneeded 'filter_ctor' field from filter structs

  Show description for 1ab4f68
  alexdowad

  committed
  1ab4f68

  Copy full SHA for 1ab4f68

• Initialization of encoding identification filters is more concise

  alexdowad

  committed
  d88df51

  Copy full SHA for d88df51

• Remove unused 'score' field from mbfl_identify_filter struct

  alexdowad

  committed
  a67cb8f

  Copy full SHA for a67cb8f

• Rewrite UTF-8 -> wchar conversion to use function pointer as state variable

  Show description for e48660d
  alexdowad

  committed
  e48660d

  Copy full SHA for e48660d

• Optimize wchar -> JIS conversion (mbfl_filt_conv_wchar_jis)

  Show description for 3c510fa
  alexdowad

  committed
  3c510fa

  Copy full SHA for 3c510fa

• Optimize encoding of HTML decimal numeric entities

  Show description for 4b53585
  alexdowad

  committed
  4b53585

  Copy full SHA for 4b53585

• Offset and mask values in mb_encode_numericentity convmap do not affect which ranges apply

  Show description for 951001a
  alexdowad

  committed
  951001a

  Copy full SHA for 951001a

• Test strimwidth with UTF-16LE text

  alexdowad

  committed
  8836e3c

  Copy full SHA for 8836e3c

• Test mb_strcut more thoroughly on UTF-16LE text

  alexdowad

  committed
  3a5c5e7

  Copy full SHA for 3a5c5e7

• mb_convert_kana throws an error on unrecognized flags

  Show description for 4269462
  alexdowad

  committed
  4269462

  Copy full SHA for 4269462

• Use mbfl_memory_device_output for emitting newlines in mb_send_mail

  Show description for 4c19d61
  alexdowad

  committed
  4c19d61

  Copy full SHA for 4c19d61

• Code cleanup in mbfl_language.c

  alexdowad

  committed
  60483da

  Copy full SHA for 60483da

• Code cleanup in mb_str_split

  Show description for f6447ea
  alexdowad

  committed
  f6447ea

  Copy full SHA for f6447ea

• [WIP] Move meat of mb_str_split to mbfl_str_split (in mbfilter.c)

  Show description for 4a74877
  alexdowad

  committed
  4a74877

  Copy full SHA for 4a74877

• Minor cleanup in mbfilter_byte2.c

  alexdowad

  committed
  e19f9b2

  Copy full SHA for e19f9b2

• Minor cleanup in mbfilter_byte4.c

  alexdowad

  committed
  6223925

  Copy full SHA for 6223925

• Minor cleanup in mbfilter_base64.c

  alexdowad

  committed
  93158db

  Copy full SHA for 93158db

• Minor cleanup in mbfilter_uuencode.c

  alexdowad

  committed
  0e01d8a

  Copy full SHA for 0e01d8a

• Code cleanup in mbfilter_htmlent.c

  Show description for e9d4694
  alexdowad

  committed
  e9d4694

  Copy full SHA for e9d4694

• Code cleanup in mbfilter_qprint.c

  alexdowad

  committed
  bb5fb71

  Copy full SHA for bb5fb71

• Combine MBFL_ENCTYPE_WCS{2,4}{BE,LE} constants

  Show description for e39473d
  alexdowad

  committed
  e39473d

  Copy full SHA for e39473d

• Nothing uses return values of identity filter functions

  Show description for 03b57af
  alexdowad

  committed
  03b57af

  Copy full SHA for 03b57af

• Identify filter functions should take unsigned char, not int

  Show description for b472672
  alexdowad

  committed
  b472672

  Copy full SHA for b472672

• Encoding conversion functions don't need to return anything

  Show description for aeca9e6
  alexdowad

  committed
  aeca9e6

  Copy full SHA for aeca9e6

• Optimize calls to mb_substr which return entire input string

  Show description for fe8ad10
  alexdowad

  committed
  fe8ad10

  Copy full SHA for fe8ad10

• Minor code tweaks in mbfl_convert.c

  alexdowad

  committed
  1ce3045

  Copy full SHA for 1ce3045

• Add comment on assertion in mbstring.c which can fail

  alexdowad

  committed
  cef8d9e

  Copy full SHA for cef8d9e

• Add 'Windows-932' alias for CP932 text encoding

  alexdowad

  committed
  9274e2a

  Copy full SHA for 9274e2a

• Enhance handling of CP932 text encoding

  Show description for b2a7ad4
  alexdowad

  committed
  b2a7ad4

  Copy full SHA for b2a7ad4

• Add test suite for CP932 text encoding

  Show description for 2cee875
  alexdowad

  committed
  2cee875

  Copy full SHA for 2cee875

• Don't pass invalid JIS X 0208 characters through

  Show description for 490baf6
  alexdowad

  committed
  490baf6

  Copy full SHA for 490baf6

• Don't pass invalid JIS X 0212 and Windows-CP932 characters through

  Show description for 7100498
  alexdowad

  committed
  7100498

  Copy full SHA for 7100498

• Remove useless mbstring encoding 'CP50220-raw'

  Show description for 5f0226a
  alexdowad

  committed
  5f0226a

  Copy full SHA for 5f0226a

• Fix identify filter for CP50220 text encoding

  alexdowad

  committed
  24267b4

  Copy full SHA for 24267b4

• Stricter handling of erroneous input when converting CP5022{0,1,2} text encoding

  alexdowad

  committed
  99320df

  Copy full SHA for 99320df

• CP5022{0,1,2}: convert characters in ku 0x2D (13th row) correctly

  Show description for 246bba7
  alexdowad

  committed
  246bba7

  Copy full SHA for 246bba7

• CP5022{0,1,2}: convert Unicode codepoints in 'user' area (0xE000-E757) correctly

  Show description for 2393da2
  alexdowad

  committed
  2393da2

  Copy full SHA for 2393da2

• Add test suite for CP5022{0,1,2}

  alexdowad

  committed
  cfd3b56

  Copy full SHA for cfd3b56

• Combine MBFL_ENCTYPE_MWC2{BE,LE} constants

  Show description for 73cac6a
  alexdowad

  committed
  73cac6a

  Copy full SHA for 73cac6a

• Remove useless mbstring encoding 'JIS-ms'

  alexdowad

  committed
  9f5fee9

  Copy full SHA for 9f5fee9

• Remove unused macros from mbfilter_cp51932.c, mbfilter_iso2022jp_mobile.c

  alexdowad

  committed
  61981ad

  Copy full SHA for 61981ad

• Enhance handling of CP51932 encoding

  Show description for 1bda08d
  alexdowad

  committed
  1bda08d

  Copy full SHA for 1bda08d

• Add test suite for CP51932 encoding

  alexdowad

  committed
  b222b4d

  Copy full SHA for b222b4d

• Remove unused function php_mb_safe_strrchr

  Show description for 2540809
  alexdowad

  committed
  2540809

  Copy full SHA for 2540809

• Remove useless constant MBFL_ENCTYPE_MBCS

  Show description for 8ef5543
  alexdowad

  committed
  8ef5543

  Copy full SHA for 8ef5543

• Simplify php_mb_zend_encoding_converter

  alexdowad

  committed
  7e18f2f

  Copy full SHA for 7e18f2f

• Don't check for impossible error condition in mb_strlen

  alexdowad

  committed
  c48d07c

  Copy full SHA for c48d07c

• Remove useless constant MBFL_ERROR_ENCODING

  alexdowad

  committed
  c4cbaa7

  Copy full SHA for c4cbaa7

• Don't check for impossible error condition in mb_substr_count

  alexdowad

  committed
  a8d947c

  Copy full SHA for a8d947c

• Don't check for impossible error condition in mb_strwidth

  alexdowad

  committed
  951bc0d

  Copy full SHA for 951bc0d

• Use stack-allocated buffer in php_mb_chr

  alexdowad

  committed
  0488bbb

  Copy full SHA for 0488bbb

• Remove unused function php_mb_mbchar_bytes

  Show description for 2c53090
  alexdowad

  committed
  2c53090

  Copy full SHA for 2c53090

• No need to null-terminate buffer in php_mb_chr

  Show description for 5234d92
  alexdowad

  committed
  5234d92

  Copy full SHA for 5234d92

• Fix mbstring support for SJIS-Mobile (DoCoMo, KDDI, and Softbank variants of Shift-JIS)

  Show description for befbb46
  alexdowad

  committed
  befbb46

  Copy full SHA for befbb46

• Add test suite for SJIS-Mobile

  alexdowad

  committed
  347e14c

  Copy full SHA for 347e14c

• CP932: treat truncated multi-byte characters as an error

  alexdowad

  committed
  8da76e5

  Copy full SHA for 8da76e5

• Remove duplicate implementation of CP932 from mbstring

  Show description for 0534a1d
  alexdowad

  committed
  0534a1d

  Copy full SHA for 0534a1d

• Fix mbstring support for ISO-2022-JP-KDDI encoding

  Show description for 83653b2
  alexdowad

  committed
  83653b2

  Copy full SHA for 83653b2

• Add test suite for ISO-2022-JP-KDDI encoding

  Show description for 6cef434
  alexdowad

  committed
  6cef434

  Copy full SHA for 6cef434

• Fix mbstring support for ISO-2022-JP-MS encoding

  Show description for 3534564
  alexdowad

  committed
  3534564

  Copy full SHA for 3534564

• Add test suite for ISO-2022-JP-MS encoding

  Show description for dadfc70
  alexdowad

  committed
  dadfc70

  Copy full SHA for dadfc70

• Refactoring of UTF-8 with mobile vendor extensions (DoCoMo, KDDI, SoftBank)

  alexdowad

  committed
  25ac652

  Copy full SHA for 25ac652