Avoid null-characters in junitxml output

[pytestbot]

Originally reported by: Floris Bruynooghe (BitBucket: flub, GitHub: flub)

---

It appears that at least Jenkins' XML parser does not like null-characters to appear in the JUnit XML file. To avoid this I've created a patch which simply removes any null characters from the XML file written by junitxml.

https://bitbucket.org/flub/pytest-queue/src/tip/junitxml_nullchar

If there is a consensus that this is indeed a bug in pytest and the correct way to fix I can commit this.

---

• Bitbucket: https://bitbucket.org/pytest-dev/pytest/issue/37

[pytestbot]

Original comment by Ronny Pfannschmidt (BitBucket: RonnyPfannschmidt, GitHub: RonnyPfannschmidt):

---

it might make to escape all nonprintable bytes to entities or escapes,
maybe just wrapping in a cdata section also makes sense

[pytestbot]

Original comment by holger krekel (BitBucket: hpk42, GitHub: hpk42):

---

I guess Ronny's more general solution would make sense - maybe he can contribute a patch? But i am also fine with Flori's patch for now - feel free to apply it and add a CHANGELOG entry (fix issue37 etc.). I plan to release 2.0.3 these days. best, holger

[pytestbot]

Original comment by Floris Bruynooghe (BitBucket: flub, GitHub: flub):

---

I'm actually midway experimenting with Ronny's suggestion, so far I know the official XML legal characters are defined as
{{{
Char ::= #x9 | #xA | #xD | [#x20-#xD7FF] | [#xE000-#xFFFD] | [#x10000-#x10FFFF]
}}}
Still need to figure out what real parsers tend to do with those characters, what to do when they're used (remove, escape by \x1234, ...) and stuff like that.

[pytestbot]

Original comment by Floris Bruynooghe (BitBucket: flub, GitHub: flub):

---

Right, my conclusion from some experimenting seems to be that all the above mentioned characters should not be allowed.

First question however is what to do with them? Should they just be stripped or should they be replaced? If replaced, what with: python-esque "\xNN" or XML-spec like "#xNN"? I think I'd prefer them to be escaped to #xNN.

Next question is where to do this. py.xml.escape would seem like an obvious choice though it would change the semantics of function significantly (it no longer just escapes your data, it actively modifies it). So at most py.xml.escape should be made to complain for invalid chars (but even that might be out of it's scope).

So my preference goes to doing this in the junitxml plugin, like my first simplistic patch did. This also has the benefit of not having to deal with XML 1.0 and 1.1 since that always outputs 1.0 (1.1 has a slightly different character range).

Does that sound reasonable?

[pytestbot]

Original comment by Floris Bruynooghe (BitBucket: flub, GitHub: flub):

---

I've pushed the fix in <<changeset 2a5219be3d3c>>, tests could still be improved some more.