Skip to content

MAINT Configure Windows CI using AppVeyor #258

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 21 commits into from
Jun 14, 2018
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
e5b4ac1
MAINT Configure Windows CI using AppVeyor
jacksonlee-civis May 29, 2018
6fd020a
MAINT Update the changelog
jacksonlee-civis Jun 11, 2018
e35f0fd
TST NamedTemporaryFile -> TemporaryDirectory etc for Windows compatib…
jacksonlee-civis Jun 11, 2018
444da8b
TST Use TemporaryDirectory from compat module
jacksonlee-civis Jun 11, 2018
d28d6aa
TST Avoid accessing 'name' attr of temp file objects
jacksonlee-civis Jun 11, 2018
353ef3d
STY Remove unused imports and vars to appease flake8
jacksonlee-civis Jun 11, 2018
c02f8c0
TST Close the temp files explicitly
jacksonlee-civis Jun 11, 2018
35a92b0
REF Switch to context managers
jacksonlee-civis Jun 11, 2018
a211a3f
TST More time for polling call count test
jacksonlee-civis Jun 11, 2018
c142274
TST Relax the polling call count test
jacksonlee-civis Jun 11, 2018
c93b8f2
TST Adjust sleep time in poller call count test
jacksonlee-civis Jun 11, 2018
4be1616
DEP No feather-format on Windows when python version < 3.5
jacksonlee-civis Jun 11, 2018
b7d8fb7
TST pytest skipif needs a reason like a human
jacksonlee-civis Jun 11, 2018
298e176
TST Use a distinct archive name in zipfile testing
jacksonlee-civis Jun 12, 2018
8474aac
TST Remove skipif for Windows
jacksonlee-civis Jun 12, 2018
9fbb4be
STY Remove unused import #flake8
jacksonlee-civis Jun 12, 2018
2f2413d
TST uuid is not needed
jacksonlee-civis Jun 13, 2018
e294c36
TST flushing not needed when existing file context manager
jacksonlee-civis Jun 13, 2018
80fb27d
Merge branch 'master' into set-up-windows-continuous-integration
jacksonllee Jun 13, 2018
6bed89f
MAINT Update the changelog
jacksonlee-civis Jun 13, 2018
46c5fca
MAINT Fix PR number in changelog
jacksonlee-civis Jun 14, 2018
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ This project adheres to [Semantic Versioning](http://semver.org/).
### Added
- Added instructions in the README for adding an API key to a Windows 10
environment
- Configured Windows CI using AppVeyor. (#258)

### Changed
- Coalesced `README.rst` and `index.rst`. (#254)
Expand Down
39 changes: 39 additions & 0 deletions appveyor.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
build: false # Not a .NET project


environment:
# For python environments: https://www.appveyor.com/docs/build-environment/#python
matrix:
- PYTHON: "C:\\Python27-x64"
PYTHON_VERSION: "2.7.15"
PYTHON_ARCH: "64"

- PYTHON: "C:\\Python34-x64"
PYTHON_VERSION: "3.4.4"
PYTHON_ARCH: "64"

- PYTHON: "C:\\Python35-x64"
PYTHON_VERSION: "3.5.3"
PYTHON_ARCH: "64"

- PYTHON: "C:\\Python36-x64"
PYTHON_VERSION: "3.6.5"
PYTHON_ARCH: "64"

# Environmental variables
CIVIS_API_KEY: "FOOBAR"


init:
- "ECHO %PYTHON% %PYTHON_VERSION% %PYTHON_ARCH%"


install:
- "%PYTHON%\\python.exe -m pip install --upgrade pip setuptools"
- "%PYTHON%\\python.exe -m pip install -r dev-requirements.txt"
- "%PYTHON%\\python.exe -m pip install -e ."


test_script:
- "%PYTHON%\\Scripts\\flake8.exe civis"
- "%PYTHON%\\Scripts\\pytest.exe -rxXs --cov civis"
7 changes: 3 additions & 4 deletions civis/ml/tests/test_model.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,6 @@
import json
import os
import pickle
import tempfile

import joblib
try:
Expand All @@ -32,7 +31,7 @@
from civis import APIClient
from civis._utils import camel_to_snake
from civis.base import CivisAPIError, CivisJobFailure
from civis.compat import mock, FileNotFoundError
from civis.compat import mock, FileNotFoundError, TemporaryDirectory
from civis.response import Response
import pytest

Expand Down Expand Up @@ -143,8 +142,8 @@ def foo(arg):

@mock.patch.object(_model.cio, 'file_to_civis', return_value=-11)
def test_stash_local_data_from_file(mock_file):
with tempfile.NamedTemporaryFile() as tempfname:
fname = tempfname.name
with TemporaryDirectory() as temp_dir:
fname = os.path.join(temp_dir, 'tempfile')
with open(fname, 'wt') as _fout:
_fout.write("a,b,c\n1,2,3\n")

Expand Down
41 changes: 23 additions & 18 deletions civis/tests/test_io.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,6 @@
import json
import os
from six import StringIO, BytesIO
import tempfile
import zipfile

import pytest
Expand Down Expand Up @@ -85,10 +84,11 @@ def setup_class(cls, *mocks):
res.result() # block

# create an export to check get_url. also tests export_csv
with tempfile.NamedTemporaryFile() as tmp:
with TemporaryDirectory() as temp_dir:
fname = os.path.join(temp_dir, 'tempfile')
sql = "SELECT * FROM scratch.api_client_test_fixture"
database = 'redshift-general'
result = civis.io.civis_to_csv(tmp.name, sql, database,
result = civis.io.civis_to_csv(fname, sql, database,
polling_interval=POLL_INTERVAL)
result = result.result()
cls.export_url = result['output'][0]['path']
Expand All @@ -98,9 +98,11 @@ def setup_class(cls, *mocks):

@mock.patch(api_import_str, return_value=civis_api_spec)
def test_zip_member_to_civis(self, *mocks):
with tempfile.NamedTemporaryFile() as tmp:
with zipfile.ZipFile(tmp, 'w', zipfile.ZIP_DEFLATED) as zip_file:
zip_file.writestr(tmp.name, 'a,b,c\n1,2,3')
with TemporaryDirectory() as temp_dir:
fname = os.path.join(temp_dir, 'tempfile')
with zipfile.ZipFile(fname, 'w', zipfile.ZIP_DEFLATED) as zip_file:
archive_name = 'archive_name'
zip_file.writestr(archive_name, 'a,b,c\n1,2,3')
zip_member = zip_file.namelist()[0]
with zip_file.open(zip_member) as zip_member_buf:
result = civis.io.file_to_civis(zip_member_buf, zip_member)
Expand Down Expand Up @@ -129,11 +131,12 @@ def test_bytes_file_to_civis(self, *mocks):
def test_large_file_to_civis(self, *mocks):
curr_size = civis.io._files.MIN_MULTIPART_SIZE
civis.io._files.MIN_MULTIPART_SIZE = 1
with tempfile.NamedTemporaryFile() as tmp:
tmp.write(b'a,b,c\n1,2,3')
tmp.flush()
tmp.seek(0)
result = civis.io.file_to_civis(tmp, tmp.name)
with TemporaryDirectory() as temp_dir:
fname = os.path.join(temp_dir, 'tempfile')
with open(fname, 'w+b') as tmp:
tmp.write(b'a,b,c\n1,2,3')
with open(fname, 'r+b') as tmp:
result = civis.io.file_to_civis(tmp, fname)

civis.io._files.MIN_MULTIPART_SIZE = curr_size

Expand All @@ -148,13 +151,14 @@ def test_civis_to_file(self, *mocks):

@mock.patch(api_import_str, return_value=civis_api_spec)
def test_csv_to_civis(self, *mocks):
with tempfile.NamedTemporaryFile() as tmp:
tmp.write(b'a,b,c\n1,2,3')
tmp.flush()
with TemporaryDirectory() as temp_dir:
fname = os.path.join(temp_dir, 'tempfile')
with open(fname, 'w+b') as tmp:
tmp.write(b'a,b,c\n1,2,3')

table = "scratch.api_client_test_fixture"
database = 'redshift-general'
result = civis.io.csv_to_civis(tmp.name, database, table,
result = civis.io.csv_to_civis(fname, database, table,
existing_table_rows='truncate',
polling_interval=POLL_INTERVAL)
result = result.result() # block until done
Expand Down Expand Up @@ -265,10 +269,11 @@ def test_get_sql_select(self, *mocks):

def test_download_file(self, *mocks):
expected = '"1","2","3"\n'
with tempfile.NamedTemporaryFile() as tmp:
civis.io._tables._download_file(self.export_url, tmp.name,
with TemporaryDirectory() as temp_dir:
fname = os.path.join(temp_dir, 'tempfile')
civis.io._tables._download_file(self.export_url, fname,
b'', 'none')
with open(tmp.name, "r") as f:
with open(fname, "r") as f:
data = f.read()
assert data == expected

Expand Down
4 changes: 2 additions & 2 deletions civis/tests/test_polling.py
Original file line number Diff line number Diff line change
Expand Up @@ -71,8 +71,8 @@ def test_poll_on_creation(self):
poll_on_creation=False)
pollable.done() # Check status once to start the polling thread
assert poller.call_count == 0
time.sleep(0.015)
assert poller.call_count == 1
time.sleep(0.02)
assert poller.call_count > 0
Copy link
Contributor Author

@jacksonllee jacksonllee Jun 12, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because now we have different execution environments for builds (Circle CI versus AppVeyor), this test has become a little unstable (see this test failure due to such indeterminancy). Given polling_internal=0.01 (set just a few lines above, via PollableResult), sleeping 0.02s gives a little more time for at least one poller call count.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It was a little unstable in the Linux tests, to be honest. Hopefully this change makes life better for everyone. Thank you!


def test_reset_polling_thread(self):
pollable = PollableResult(
Expand Down
2 changes: 1 addition & 1 deletion dev-requirements.txt
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ vcrpy>=1.11.0,<=1.11.99
vcrpy-unittest==0.1.6
sphinx_rtd_theme>=0.2.4,<1.0.0
numpydoc>=0.7.0,<1.0.0
feather-format
feather-format; sys_platform != 'win32' or python_version >= '3.5'
Copy link
Contributor Author

@jacksonllee jacksonllee Jun 12, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

feather-format depends on pyarrow, which doesn't have a Windows installer for Python < 3.5; see here. Not having feather-format installed skips one test only, which I think is fine (and we are running this test for Python >= 3.5 on Windows anyway).

numpy
pandas; python_version >= '3.5'
pandas<0.21; python_version == '3.4'
Expand Down