Handing of non-utf-8 posix filenames

[kevina]

Based on what I gather from #1 of the requirements is to be able to backup an Unix filesystem onto ipfs. In order to do this it is important to be able to handle any valid Unix/Posix filename. Unix filenames may contain any character except null (0x00) and and /, they is no guarantee that they are valid UTF-8 strings, even if this is now the best practice.

So the question is how to handle them in the CBOR encoding. The simplest option would be just to (1) make them CBOR byte strings and be done with it. This could present a problem when converting it to JSON.

A slightly more complicated option is to (2) make use of the text/byte distinction in CBOR. If a string is a valid UTF-8 string then it MUST be encoded as a text string if not then it is encoded as a byte string. I use the word MUST (in the RFC sense) so that there is only one way to encoded a given filename. Given this option a UTF-8 string can be encoded in JSON as is, but a byte string still needs special treatment.

So the question is how to handle a non-valid utf-8 bytes in JSON. As I see it there are several options (j1) Encode the bytes using the non-standard JSON escape sequence \0x## where ## is the hex value of the byte for example 0x77. (j2) Somehow encode the bytes in Unicode itself, one option would be for the / to act as a marker that the next charter is a literal byte and not a utf-8 character, for example the byte 0x77 could be encoded as /\u0077. (j3) If we elect to make use of the CBOR byte/string distinction then any filenames that are bytes can just be encoded using as a BASE64-string. (j4) A slightly more compact version of 4 is to assume the non-utf-8 string is ISO-8859-1 and convert to Unicode, when decoding it will be converted back from UTF-8 to ISO-8859-1 with no lose of information.

For (j3) or (j4) there will need to be way to signal that the string should not be interpreted as UTF-8 in JSON. One idea I have is to start the string with a '/' as that can be included in any filename.

Finally, there is always the option to just force UTF-8 (and maybe even a more restive set as I think @mib-kd743naq wants), but then there will be filenames that can not be repressed in a IPLD Unixfs and some other format will likely be needed for backing up the filesystem.

@whyrusleeping @Stebalien others, thoughts?

[Edited to include option (4) for JSON encoding.]

[Stebalien]

See ipfs/kubo#1710 for a lot of discussion on this. What we should support in paths is an open question. Note: I don't endorse every suggestion in this thread.

Also, to be a bit pedantic, names in maps (and link names) are unicode, not necessarily utf-8 (cbor just encodes them using utf-8 but that's not a hard requirement). Our DagJSON format doesn't currently support binary at all so I wouldn't try optimizing for that. However...

There are a few ways to go:

1. Don't support non-unicode paths.
2. Use a format like {name: "fname" /* possibly binary */}. However, this doesn't fix paths (which we do generally want to be unicode.
3. Have rules for importing/exporting.
4. Have rules for importing only (don't convert back when exporting) and only turn this on with a flag.

Personally, I like 4 (and I'd like to be even more restrictive and forbid, e.g., newlines and control codes). While there could technically be files that have such characters in their names, they're unlikely to exist outside of either buggy or evil programs.

---

Note: I don't intend to lose any sleep over breaking existing paths if users have decided to, e.g., put newlines in them.

[Stebalien]

Just to make it clear, DagJSON doesn't currently support binary (although the go daemon will take a best-effort approach and spit out a base64 encoded string). We've discussed adding a special {'/': 'multibaseValue', 'type': 'binary'} syntax but haven't agreed on doing so.

[kevina]

Note I thought one of the eventual goals is to remove the name attribute from links as they are not always meaningful, I apologize if I was mistaken. Map keys in CBOR can be anything including byte strings (even though the RFC recommends using text strings).

(1) I do not support, but see the next paragraph (2) could be a problem if filenames are keys in a map. I do not fully understand what you are trying to say with (3) or (4).

Above all if we intend for ipld-unixfs to be used a replacement of tar (or to a lesser extended actually be used as a unixfs as a filesystem that insists on utf-8 paths could create compatibility problems) then we have to somehow support any filename that is valid in Unix. If we do not then we need to be honest and up-front about that (for example say that ipld-unixfs can be used to faithfully represented an traditional unixfs, provided that all filenames are valid utf-8). If someone wants to use ipfs in this way (replacement for tar or as a unixfs where for whatever reason filenames are not utf-8 or contain any of the forbidden characters) then a different format needs to be used---even if that format is identical to ipld-unixfs but with the name component interpreted as a byte string rather than a utf-8 string.

[mib-kd743naq]

and maybe even a more restive set as I think @mib-kd743naq wants

@kevina actually I do not want a restrictive set. My use case / interest in IPFS is strictly and literally "store random .tar files faithfully within IPFS". The reason I mentioned "maybe we just want to limit things" was specifically because of how... unprepared the tooling currently is. Consider ~$ ipfs ls QmZcYTLQNo1hJtGxQD8d9sHtmnBBamhDHydtqJ2zxzLSx7 ( from ipfs/kubo#4292 (comment) )

In other words: I would love to be able to store everything sans \x00 and \x2F in IPFS, but I am realistic what is (not) achievable given the realities and focuses of the project...

[Kubuxu]

Consolidating my comments on this topic:

---

Copied from

In response to @mib-kd743naq:

It seems clearer from that standpoint that IPFS can't be both "fully POSIX compliant" and "an upgraded protocol with less cruft".

I disagree with this statement, what is stored (and can be accessed through the API) versus what is presented to the user are two things that can be different. As most of shells are not about to handle binary strings (as variables) it makes sense to have encoding in presenting them.

In response to @kevina proposal:

The key type can either be a byte or text string

I think it being either String or bytestring (as CBOR calls it) will create more problems than we expect. It is still problematic with JSON (as we can't well represent bytearrays in JSON, we are thinking about something that would workaround this problem by allowing you to ask for many formats with order of preference and it would return first format that is about to fully represent the data).

It would mean that there is no minimal schema for this format. Depending on what we decide to do with filenames (UTF-8 only, arbitrary, safe characters only) we should make it either a string or bytestring.

[Kubuxu]

Also we need to think what IPLD resolver will be able to handle as keys. If it doesn't handle / in the key (as it is currently, I think) we have no POSIX.

[kevina]

Another possibility:

(3) Use UTF-8 or a restrictive set, but provide a field for storing the original name. A basic implementation is allowed to reject invalid filenames and required to preserve this field when copying files, but otherwise is not required to make use of it.

When the field is used the translation from an invalid name to something that is representable for the for the original filename is outside the scope of the spec and the best translation may in fact defend on the filenames themself. For example if the filenames are in a non-utf-8 encoding the best behavior may be to translate then to UTF-8 but preserver the original name. A very basic implementation could replace the invalid character with something else and then append numbers if two filenames are the same in a directory.

[Stebalien]

@Kubuxu

as we can't well represent bytearrays in JSON, we are thinking about something that would workaround this problem by allowing you to ask for many formats with order of preference and it would return first format that is about to fully represent the data

unixfs can't use DagJSON anyways (at the moment) because files are raw binary. Honestly, we only use DagJSON in the HTTP API so we should just avoid it as much as possible.

Also we need to think what IPLD resolver will be able to handle as keys. If it doesn't handle / in the key (as it is currently, I think) we have no POSIX.

I'm not sure I understand. POSIX doesn't allow for / in filenames either. However, the current thinking is to allow / in IPLD keys but say that the current pathing scheme can't path over them. Basically, the current path scheme is a language for traversing IPLD objects, not necessarily the only one (although we could specify an escaping scheme up-front).

Note: IPDL paths != unixfs paths.

@kevina

(5) Use UTF-8 or a restrictive set, but provide a field for storing the original name. A basic implementation is allowed to reject invalid filenames and required to preserve this field when copying files, but otherwise is not required to make use of it.

That's probably the best option (that's what I was getting at with option 3 but simply keeping the original filename in a separate, binary field is probably better). However, I'd add two constraints:

1. By default, we should reject such files (enabling import with a flag). Otherwise, imported files will have different names in IPFS.
2. By default, we definitely shouldn't use such names when exporting from unixfs to a local filesystem (enabled with a flag) for the same reason.

That is, provide some form of unixfs --preserve-weird-names=true flag.

---

Again, just to be pedantic, UTF-8 != Unicode (I don't want to tie ourselves to an encoding in the spec).

[Kubuxu]

I'm not sure I understand. POSIX doesn't allow for / in filenames either

I forgot about that for some reason.

[kevina]

I don't want to tie ourselves to an encoding in the spec

So you are saying we should say filenames should be Unicode but allow then be encoded in UTF-8 or UCS-2, UTF-16 and UTF-32?

I quite literal mean UTF-8 here.

[Stebalien]

So you are saying we should say filenames should be Unicode but allow then be encoded in UTF-8 or UCS-2, UTF-16 and UTF-32?

Unicode is a standard for text. It's a character set along with a set of rules governing valid strings and how to interpret them; it, in itself, doesn't dictate how these characters map to bytes. UTF-8 a (one) mapping between these conceptual character strings and bytes.

The precise encoding used in any given situation will be up to the IPLD format (e.g., CBOR), the language (e.g., go), and the OS/Filesystem.

For example, Windows uses UTF-16 for filenames. When importing files on Windows, we'd treat filenames as Unicode strings (decoding the UTF-16 filenames into whatever Unicode encoding our language uses) and then re-encode them in the IPLD format's unicode encoding (UTF-8 for CBOR).

On, e.g., Linux, we'd have to assume that paths (on import) are UTF-8 encoded as Linux doesn't actually specify how to interpret paths as Unicode (although we could probably use the $LANG or $LC_* variables but that's probably not worth it).

[Kubuxu]

Is JSON not supporting bytestrings the main (or the only) reason behind limiting to Unicode?

[kevina]

Is JSON not supporting bytestrings the main (or the only) reason behind limiting to Unicode?

Yes, the primary one (at least for me).

But there is also the issue on how to handle non-UTF-8 client side.

[kevina]

although we could probably use the $LANG or $LC_* variables but that's probably not worth it

Using LANG is not a good idea. It is not an intrinsic part of the filesystem (like say codepages on FAT) and can be set incorrectly. It could be used as a hint to translate the encoding into Unicode and store the original untranslated name in a field as I suggested in (3).