BLD: Fixed pointer ambiguity in 'tokenizer.c'

[ivannz]

• closes build compatibility issue COMPAT: windows builds breaking after #13788 #13863
• tests added / passed (see notes)
• passes git diff upstream/master | flake8 --diff
• whatsnew entry (No whatsnew entry was necessary)

notes

The branch was tested on win7 64bit virtual machine with anaconda 4.1.1 both with python 2.7 (VS2008) and with python 3.5 (VS2015). Running test.bat returned this on each platform:

python 2.7

======================================================================
FAIL: test_constructor_bad_file (pandas.io.tests.test_common.TestMMapWrapper)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\vmuser\pandas\pandas\io\tests\test_common.py", line 110, in tes
t_constructor_bad_file
    tm.assertRaisesRegexp(err, msg, common.MMapWrapper, non_file)
  File "C:\Users\vmuser\pandas\pandas\util\testing.py", line 2373, in assertRais
esRegexp
    _callable(*args, **kwargs)
  File "C:\Users\vmuser\pandas\pandas\util\testing.py", line 2398, in __exit__
    return self.handle_success(exc_type, exc_value, traceback)
  File "C:\Users\vmuser\pandas\pandas\util\testing.py", line 2411, in handle_suc
cess
    raise_with_traceback(e, traceback)
  File "C:\Users\vmuser\pandas\pandas\util\testing.py", line 2373, in assertRais
esRegexp
    _callable(*args, **kwargs)
  File "C:\Users\vmuser\pandas\pandas\io\common.py", line 355, in __init__
    self.mmap = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ)
AssertionError: "The parameter is incorrect" does not match "[Error 87] "

----------------------------------------------------------------------
Ran 10634 tests in 494.796s

FAILED (SKIP=148, failures=1)

pd.show_versions()

INSTALLED VERSIONS
------------------
commit: 3b0e28e255c0319bc19397e07493bb92b4190d53
python: 2.7.12.final.0
python-bits: 64
OS: Windows
OS-release: 7
machine: AMD64
processor: Intel64 Family 6 Model 42 Stepping 7, GenuineIntel
byteorder: little
LC_ALL: None
LANG: None
LOCALE: None.None

pandas: 0.18.1+291.g3b0e28e
nose: 1.3.7
pip: 8.1.2
setuptools: 23.0.0
Cython: 0.24
numpy: 1.11.1
scipy: 0.17.1
statsmodels: 0.6.1
xarray: None
IPython: 4.2.0
sphinx: 1.4.1
patsy: 0.4.1
dateutil: 2.5.3
pytz: 2016.4
blosc: None
bottleneck: 1.1.0
tables: 3.2.2
numexpr: 2.6.0
matplotlib: 1.5.1
openpyxl: 2.3.2
xlrd: 1.0.0
xlwt: 1.1.2
xlsxwriter: 0.9.2
lxml: 3.6.0
bs4: 4.4.1
html5lib: None
httplib2: None
apiclient: None
sqlalchemy: 1.0.13
pymysql: None
psycopg2: None
jinja2: 2.8
boto: 2.40.0
pandas_datareader: None

python 3.5

======================================================================
ERROR: test_to_latex_filename (pandas.tests.formats.test_format.TestDataFrameFor
matting)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\vmuser\pandas\pandas\tests\formats\test_format.py", line 2800,
in test_to_latex_filename
    df.to_latex(path)
  File "C:\Users\vmuser\pandas\pandas\core\frame.py", line 1666, in to_latex
    encoding=encoding)
  File "C:\Users\vmuser\pandas\pandas\formats\format.py", line 658, in to_latex
    latex_renderer.write_result(f)
  File "C:\Users\vmuser\pandas\pandas\formats\format.py", line 894, in write_res
ult
    buf.write(' & '.join(crow))
  File "C:\Anaconda2\envs\py35\lib\encodings\cp1251.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '\xdf' in position 7:
 character maps to <undefined>

======================================================================
FAIL: test_constructor_bad_file (pandas.io.tests.test_common.TestMMapWrapper)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\vmuser\pandas\pandas\util\testing.py", line 2373, in assertRais
esRegexp
    _callable(*args, **kwargs)
  File "C:\Users\vmuser\pandas\pandas\io\common.py", line 355, in __init__
    self.mmap = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ)
OSError: [WinError 87] Параметр задан неверно

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\vmuser\pandas\pandas\io\tests\test_common.py", line 110, in tes
t_constructor_bad_file
    tm.assertRaisesRegexp(err, msg, common.MMapWrapper, non_file)
  File "C:\Users\vmuser\pandas\pandas\util\testing.py", line 2373, in assertRais
esRegexp
    _callable(*args, **kwargs)
  File "C:\Users\vmuser\pandas\pandas\util\testing.py", line 2398, in __exit__
    return self.handle_success(exc_type, exc_value, traceback)
  File "C:\Users\vmuser\pandas\pandas\util\testing.py", line 2411, in handle_suc
cess
    raise_with_traceback(e, traceback)
  File "C:\Users\vmuser\pandas\pandas\compat\__init__.py", line 333, in raise_wi
th_traceback
    raise exc.with_traceback(traceback)
  File "C:\Users\vmuser\pandas\pandas\util\testing.py", line 2373, in assertRais
esRegexp
    _callable(*args, **kwargs)
  File "C:\Users\vmuser\pandas\pandas\io\common.py", line 355, in __init__
    self.mmap = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ)
AssertionError: "The parameter is incorrect" does not match "[WinError 87] Параметр задан неверно"

----------------------------------------------------------------------
Ran 10634 tests in 437.328s

FAILED (SKIP=146, errors=1, failures=1)

pd.show_versions()

INSTALLED VERSIONS
------------------
commit: 3b0e28e255c0319bc19397e07493bb92b4190d53
python: 3.5.2.final.0
python-bits: 64
OS: Windows
OS-release: 7
machine: AMD64
processor: Intel64 Family 6 Model 42 Stepping 7, GenuineIntel
byteorder: little
LC_ALL: None
LANG: None
LOCALE: None.None

pandas: 0.18.1+291.g3b0e28e
nose: 1.3.7
pip: 8.1.2
setuptools: 23.0.0
Cython: 0.24
numpy: 1.11.1
scipy: 0.17.1
statsmodels: 0.6.1
xarray: None
IPython: 4.2.0
sphinx: 1.3.1
patsy: 0.4.1
dateutil: 2.5.3
pytz: 2016.4
blosc: None
bottleneck: 1.1.0
tables: 3.2.2
numexpr: 2.6.0
matplotlib: 1.5.1
openpyxl: 2.3.2
xlrd: 1.0.0
xlwt: 1.1.2
xlsxwriter: 0.9.2
lxml: 3.6.0
bs4: 4.4.1
html5lib: None
httplib2: None
apiclient: None
sqlalchemy: 1.0.13
pymysql: None
psycopg2: None
jinja2: 2.8
boto: 2.40.0
pandas_datareader: None

[jreback]

hmm, you shouldn't be getting any failing tests, these looks like locale issues. (note these are windows specific and not tested on Travis I think)

@gfyoung ?

[jreback]

@ivannz if you want to create an appveyor account then it will run the tests there as well (using pandas config). its free :>

[gfyoung]

@jreback : Agreed. These appear to be locale issues. Let Travis run! :)

[ivannz]

Here is a link to Travis CI build results in my repo.

[codecov-io]

Current coverage is 85.25% (diff: 100%)

Merging #13865 into master will not change coverage

@@             master     #13865   diff @@
==========================================
  Files           140        140          
  Lines         50455      50455          
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
  Hits          43014      43014          
  Misses         7441       7441          
  Partials          0          0          

Powered by Codecov. Last update 59f2557...3b0e28e

[jreback]

thanks @ivannz this seems to have done the trick.

for those other failing tests you are getting, if you'd create a repro and file an issue would be great.