Refactor/cleanup of `String`/`ByteString` usage #4666

hvr · 2017-08-05T16:37:46Z

This refactoring is intended to clean-up various instances where [Char] is used to convey
something other than Unicode text. This picks up unfinished yak-shaving that was put aside when working
on #3913 ... ;-)

Note: this PR has been carefully split into self-contained commits to ease reviewing

mention-bot · 2017-08-05T16:37:49Z

@hvr, thanks for your PR! By analyzing the history of the files in this pull request, we identified @dcoutts, @ezyang and @23Skidoo to be potential reviewers.

This is the proper type, as 'getFileContents' is supposed to return contents read in binary file mode, and prior to this patch, `[Char]` was abused to return binary data.

The API was assymetric, as there was `fromUTF8(L)BS` but not the dual operations. The plan is refactor all occurences of - `fromUTF8 :: String -> String` - `toUTF8` :: String -> String` until `fromUTF8`/`toUTF8` is unused, at which point we can officially deprecate or remove it.

This was introduced in 1821d80 but it would imply that stdout was set to binary mode, which it isn't.

This new type will be used to disentangle conflated uses of `String` and clearly distinguish between binary and textual data.

This removes the remaining occurences of the weakly typed `{to,from}UTF8` conversion.

Since we don't use those functions anymore, we can finally `DEPRECATE` them.

23Skidoo

LGTM modulo minor comments.

23Skidoo · 2017-08-09T11:18:08Z

Cabal/Distribution/PackageDescription/Check.hs

@@ -1777,12 +1776,16 @@ checkCabalFileBOM ops = do
    -- --cabal-file is specified.  So if you can't find the file,
    -- just don't bother with this check.
    Left _       -> return $ Nothing
-    Right pdfile -> (flip check pc . startsWithBOM . fromUTF8)


Can startsWithBOM be removed now?

23Skidoo · 2017-08-09T11:20:40Z

Cabal/Distribution/Utils/Generic.hs

@@ -246,8 +246,7 @@ startsWithBOM _            = False

 -- | Check whether a file has Unicode byte order mark (BOM).
 fileHasBOM :: FilePath -> NoCallStackIO Bool
-fileHasBOM f = fmap (startsWithBOM . fromUTF8)
-             . hGetContents =<< openBinaryFile f ReadMode
+fileHasBOM f = (startsWithBOM . fromUTF8LBS) <$> BS.readFile f


Can't you use BS.isPrefixOf here?

yes, but we can also just remove fileHasBOM as nobody uses it anymore iirc

23Skidoo · 2017-08-09T11:25:27Z

Cabal/Distribution/Simple/Utils.hs

+-- closed.
+--
+-- @since 2.2.0
+ioDataHGetContents :: Handle -> IODataMode -> IO IOData


Maybe move IOData & friends to a separate module and import that qualified instead of using a prefix? Like the existing BS./LBS. precedent.

@23Skidoo what do you suggest? Something like

module Distribution.Simple.IOData ( IOData(..) , hGetContents , hPutContents -- ... ) where -- ...

?

Right, that's what I had in mind.

I'll go for D.Utils.IOData, as that's where e.g. ShortText lives too

23Skidoo · 2017-08-09T11:27:21Z

Cabal/Distribution/Utils/Generic.hs

-  | c <= '\xFB' = moreBytes 5 0x200000  cs (ord c .&. 0x3)
-  | c <= '\xFD' = moreBytes 6 0x4000000 cs (ord c .&. 0x1)
-  | otherwise   = replacementChar : fromUTF8 cs
+fromUTF8 = decodeStringUtf8 . map c2w


fwiw, we could also remove to/fromUTF8 w/o any deprecation. Nothing in the code-base needs it anymore

Yeah, since this will be in Cabal 2.2, setup scripts shouldn't be affected. May be worth adding a note to the changelog about the stuff that is gone.

for changelog see b1bbf2e

hvr · 2017-08-09T18:17:32Z

@23Skidoo I've implemented your suggestion; can you look over it?

I'll write the changelog as soon as you confirm this is what you intended :-)

23Skidoo · 2017-08-09T19:51:24Z

@hvr Looks good, though you forgot to git add the new module.

This reduces the surface area of lib:Cabal by removing entry points that Setup.hs scripts are very unlikely to use.

hvr · 2017-08-09T19:56:44Z

@23Skidoo Doh! ... fixed...

[skip ci]

hvr added 3 commits August 5, 2017 19:16

Make CheckPackageContentOps return a lazy ByteString

96bcbf1

This is the proper type, as 'getFileContents' is supposed to return contents read in binary file mode, and prior to this patch, `[Char]` was abused to return binary data.

Refactor some use-sites of {to,from}UTF8 to use {to,From}UTF8LBS

881ba6b

hvr force-pushed the pr/utf8-refactor branch from a0408be to 881ba6b Compare August 5, 2017 17:16

Remove spurious UTF8 conversion for stdout

98c6dbc

This was introduced in 1821d80 but it would imply that stdout was set to binary mode, which it isn't.

hvr changed the title ~~WIP: Refactor/cleanup of String/ByteString usage~~ Refactor/cleanup of String/ByteString usage Aug 5, 2017

hvr force-pushed the pr/utf8-refactor branch 2 times, most recently from 8912f68 to 3856f76 Compare August 5, 2017 18:52

hvr added 3 commits August 6, 2017 00:03

Add new 'IOData' abstraction

424bb5f

This new type will be used to disentangle conflated uses of `String` and clearly distinguish between binary and textual data.

Refactor rawSystemStdInOut to use IOData abstraction

5027632

This removes the remaining occurences of the weakly typed `{to,from}UTF8` conversion.

Reimplement {from,to}UTF8 in terms of {encode,decode}StringUtf8

9f7f733

Since we don't use those functions anymore, we can finally `DEPRECATE` them.

hvr force-pushed the pr/utf8-refactor branch from 3856f76 to 9f7f733 Compare August 5, 2017 22:03

hvr requested a review from 23Skidoo August 6, 2017 07:40

23Skidoo approved these changes Aug 9, 2017

View reviewed changes

hvr added 2 commits August 9, 2017 21:56

Move IOData into new Distribution.Utils.IOData module

d158887

Remove obsolete functions

2430a29

This reduces the surface area of lib:Cabal by removing entry points that Setup.hs scripts are very unlikely to use.

hvr force-pushed the pr/utf8-refactor branch from 353e758 to 2430a29 Compare August 9, 2017 19:56

changelog entry for haskell#4666

b1bbf2e

[skip ci]

hvr added this to the 2.2 milestone Aug 10, 2017

hvr added the type: enhancement label Aug 10, 2017

hvr merged commit 6a08850 into haskell:master Aug 10, 2017

hvr deleted the pr/utf8-refactor branch August 10, 2017 06:33

Refactor/cleanup of String/ByteString usage #4666

Refactor/cleanup of String/ByteString usage #4666

Uh oh!

Conversation

hvr commented Aug 5, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mention-bot commented Aug 5, 2017

Uh oh!

23Skidoo left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hvr Aug 9, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hvr Aug 9, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hvr commented Aug 9, 2017

Uh oh!

23Skidoo commented Aug 9, 2017

Uh oh!

hvr commented Aug 9, 2017

Uh oh!

Uh oh!

Refactor/cleanup of `String`/`ByteString` usage #4666

Refactor/cleanup of `String`/`ByteString` usage #4666

hvr commented Aug 5, 2017 •

edited

Loading

hvr Aug 9, 2017 •

edited

Loading

hvr Aug 9, 2017 •

edited

Loading