Add setup-conda action #46611

jonashaag · 2022-04-02T10:39:41Z

Smaller version of #46493 according to suggestion in #46493 (comment). Should be orthogonal to #46538.

closes CI: use conda incubator in doc build #39175 (Replace xxxx with the Github issue number)
Tests added and passed if fixing a bug or adding a new feature
All code checks passed.
Added an entry in the latest doc/source/whatsnew/vX.X.X.rst file if fixing a bug or adding a new feature.

jonashaag · 2022-04-02T10:57:30Z

.github/actions/setup-pandas/action.yml

@@ -0,0 +1,31 @@
+name: Set up pandas


This action will eventually supersede the setup action

jonashaag · 2022-04-03T08:31:36Z

cc @lithomas1 because you were worried about GHA capacity. I think this is fine now because of #46600.

.github/actions/run-tests/action.yml

mroeschke

One comment otherwise looks good!

jreback · 2022-04-04T22:56:20Z

looks ok. are we backporting these @mroeschke ?

datapythonista

Added some comments, but this is too big too be merged in my opinion. The CI is quite tricky, if we break things we can end up having errors that aren't caught in the CI (it happened several times in the past). Also, if things go wrong, we should be able to revert the windows or the mac changes, without having to also revert the others.

So, better go step by step. I'd start by adding the actions, when we're happy we add one of the builds (for mac or for windows), still keeping the one in azure for few days, until we're confident the new one is working fine, and catches the errors. And, then we open a new PR to remove the old one. Merging a PR this big can easily get us in trouble.

.github/actions/build_pandas/action.yml

datapythonista · 2022-04-04T22:48:56Z

.github/actions/build_pandas/action.yml

+      shell: bash -el {0}
+
+    - name: Build Version
+      run: pushd /tmp && python -c "import pandas; pandas.show_versions();" && popd


Why to execute the show_versions() from /tmp and not the current directory?

Idk it's copied from other CI code

I think it's save to remove the pushd and popd stuff.

Build Version seems a bit confusing, Show dependency versions or similar seems to be more descriptive of what this is doing.

.github/actions/run-tests/action.yml

ci/run_tests.sh

mroeschke · 2022-04-05T02:47:06Z

we should be able to revert the windows or the mac changes

This is a fair point. Might be good to just do macOS in 1 PR and Windows in another PR. Having them in separate workflows might be a good idea too since the paths and setup might differ a bit

Would probably need backporting when ready.

jonashaag · 2022-04-05T07:27:53Z

Added back Azure CI and enabled Windows only. I left the workflow name identical for now

jonashaag · 2022-04-05T16:33:23Z

Can some please re-run the failed jobs?

datapythonista · 2022-04-05T17:10:38Z

Can some please re-run the failed jobs?

The easiest is usually to close and reopen the PR, and everything should re-run. Let me know of that doesn't work.

.github/actions/build_pandas/action.yml

datapythonista · 2022-04-05T19:23:23Z

.github/actions/build_pandas/action.yml

-        python -m pip install -e . --no-build-isolation --no-use-pep517 --no-index
+        time python setup.py build_ext -v -j $N_JOBS
+        pip install -v -e . --no-build-isolation --no-use-pep517 --no-index
+      shell: bash -el {0}


How is this shell different to the default?

It is a login shell (important for Conda to work)

It fails if one of the commands fails, rather than just continuing.

datapythonista · 2022-04-05T19:23:30Z

.github/actions/build_pandas/action.yml

+      shell: bash -el {0}
+      env:
+        # Cannot use parallel compilation on Windows, see https://github.com/pandas-dev/pandas/issues/30873
+        N_JOBS: ${{ runner.os == 'Windows' && 1 || 3 }}


Why 3 jobs if not windows? My understanding is that GItHub runners have 2 cores.

macOS has 3, can change that back to 2 and change to 3 in a follow-up PR where we add macOS?

Or we could just hardcode Windows 1, Linux 2, macOS 3 here?

datapythonista · 2022-04-05T19:24:56Z

.github/actions/build_pandas/action.yml

+      shell: bash -el {0}
+
+    - name: Build Version
+      run: pushd /tmp && python -c "import pandas; pandas.show_versions();" && popd


I think it's save to remove the pushd and popd stuff.

Build Version seems a bit confusing, Show dependency versions or similar seems to be more descriptive of what this is doing.

.github/workflows/macos-windows.yml

ci/run_tests.sh

github-actions · 2022-05-06T00:05:01Z

This pull request is stale because it has been open for thirty days with no activity. Please update and respond to this comment if you're still interested in working on this.

jonashaag · 2022-05-22T10:31:14Z

Merged with main and reduced number of changes in this PR. It now mainly adds new actions that can be used by the other workflows in subsequent PRs, to deduplicate workflow code.

mroeschke · 2022-05-27T17:24:03Z

Would it be possible to do setup-pandas in this PR and run-tests in another PR? It would help isolate the change and make it easier for review.

Also, for setup-pandas it would be nice to get rid of the setup-env.sh script since I think only the docbuild script uses that file.

jonashaag · 2022-05-27T22:26:53Z

Also, for setup-pandas it would be nice to get rid of the setup-env.sh script since I think only the docbuild script uses that file.

And CircleCI. Shall I move docbuild to setup-pandas in this PR or in another PR?

mroeschke · 2022-05-27T23:04:14Z

Also, for setup-pandas it would be nice to get rid of the setup-env.sh script since I think only the docbuild script uses that file.

And CircleCI. Shall I move docbuild to setup-pandas in this PR or in another PR?

I think it would be okay to do it in this PR

jonashaag · 2022-05-27T23:20:25Z

Moved setup_env to CircleCI folder and change docbuild to use new action. Let's see if this works.

.github/actions/build_pandas/action.yml

mroeschke

Awesome, great cleanup. LGTM

.github/workflows/macos-windows.yml

mroeschke · 2022-06-03T20:59:20Z

Awesome, thanks for the simplifying and the cleanup @jonashaag

* Add run-tests and setup-pandas actions * Undo run-tests * Use setup-pandas in docbuild * Update docbuild-and-upload.yml * Update macos-windows.yml * Update docbuild-and-upload.yml * Update action.yml * Create action.yml * Delete .github/actions/setup-pandas directory * Review feedback * Update .github/workflows/macos-windows.yml Co-authored-by: Matthew Roeschke <[email protected]>

jonashaag commented Apr 2, 2022

View reviewed changes

jonashaag force-pushed the gha-win-mac branch from 18e5153 to 89dfbb0 Compare April 2, 2022 17:14

jonashaag mentioned this pull request Apr 2, 2022

Parallel build_ext might fail due to file collisions #30873

Closed

mroeschke reviewed Apr 3, 2022

View reviewed changes

.github/actions/run-tests/action.yml Outdated Show resolved Hide resolved

mroeschke added the CI Continuous Integration label Apr 3, 2022

mroeschke reviewed Apr 3, 2022

View reviewed changes

jreback added this to the 1.5 milestone Apr 4, 2022

datapythonista reviewed Apr 4, 2022

View reviewed changes

jonashaag mentioned this pull request Apr 5, 2022

Meta issue: CI refactoring #46643

Closed

6 tasks

jonashaag changed the title ~~Move macOS and Windows CI from Azure to GHA~~ Add Windows GHA build Apr 5, 2022

jonashaag closed this Apr 5, 2022

jonashaag reopened this Apr 5, 2022

datapythonista reviewed Apr 5, 2022

View reviewed changes

github-actions bot added the Stale label May 6, 2022

mroeschke mentioned this pull request May 7, 2022

CI: Move Windows build from Azure to GHA #46960

Merged

1 task

jonashaag changed the title ~~Add Windows GHA build~~ Add run-tests and setup-pandas actions May 22, 2022

jonashaag force-pushed the gha-win-mac branch from 006394b to 078987a Compare May 25, 2022 10:08

jonashaag force-pushed the gha-win-mac branch from d9efd64 to 6451c63 Compare May 27, 2022 20:03

jonashaag added 2 commits May 28, 2022 00:23

Add run-tests and setup-pandas actions

94b2567

Undo run-tests

4d22885

jonashaag force-pushed the gha-win-mac branch from 6451c63 to 4d22885 Compare May 27, 2022 22:24

jonashaag changed the title ~~Add run-tests and setup-pandas actions~~ Add setup-pandas action May 27, 2022

Use setup-pandas in docbuild

7a01e03

jonashaag added 6 commits May 29, 2022 19:09

Update docbuild-and-upload.yml

ac55a00

Update macos-windows.yml

5e7d802

Update docbuild-and-upload.yml

b22364f

Update action.yml

867f1e7

Create action.yml

e017dcc

Delete .github/actions/setup-pandas directory

afbb610

jonashaag changed the title ~~Add setup-pandas action~~ Add setup-co da action May 29, 2022

jonashaag changed the title ~~Add setup-co da action~~ Add setup-conda action May 29, 2022

jreback removed the Stale label May 29, 2022

mroeschke reviewed May 30, 2022

View reviewed changes

.github/actions/build_pandas/action.yml Outdated Show resolved Hide resolved

mroeschke reviewed May 30, 2022

View reviewed changes

.github/actions/build_pandas/action.yml Outdated Show resolved Hide resolved

Review feedback

8106b10

jonashaag requested a review from mroeschke May 31, 2022 07:30

mroeschke approved these changes May 31, 2022

View reviewed changes

mroeschke reviewed Jun 3, 2022

View reviewed changes

.github/workflows/macos-windows.yml Outdated Show resolved Hide resolved

Update .github/workflows/macos-windows.yml

4b3f8c4

mroeschke merged commit b1b788f into pandas-dev:main Jun 3, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add setup-conda action #46611

Add setup-conda action #46611

jonashaag commented Apr 2, 2022 •

edited by mroeschke

Loading

jonashaag Apr 2, 2022

jonashaag commented Apr 3, 2022

mroeschke left a comment

jreback commented Apr 4, 2022

datapythonista left a comment

datapythonista Apr 4, 2022

jonashaag Apr 5, 2022

datapythonista Apr 5, 2022

mroeschke commented Apr 5, 2022

jonashaag commented Apr 5, 2022

jonashaag commented Apr 5, 2022

datapythonista commented Apr 5, 2022

datapythonista Apr 5, 2022

jonashaag Apr 5, 2022

datapythonista Apr 5, 2022

jonashaag Apr 5, 2022

jonashaag Apr 5, 2022

datapythonista Apr 5, 2022

github-actions bot commented May 6, 2022

jonashaag commented May 22, 2022

mroeschke commented May 27, 2022 •

edited

Loading

jonashaag commented May 27, 2022

mroeschke commented May 27, 2022 •

edited

Loading

jonashaag commented May 27, 2022

mroeschke left a comment

mroeschke commented Jun 3, 2022

Add setup-conda action #46611

Add setup-conda action #46611

Conversation

jonashaag commented Apr 2, 2022 • edited by mroeschke Loading

Choose a reason for hiding this comment

jonashaag commented Apr 3, 2022

mroeschke left a comment

Choose a reason for hiding this comment

jreback commented Apr 4, 2022

datapythonista left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mroeschke commented Apr 5, 2022

jonashaag commented Apr 5, 2022

jonashaag commented Apr 5, 2022

datapythonista commented Apr 5, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented May 6, 2022

jonashaag commented May 22, 2022

mroeschke commented May 27, 2022 • edited Loading

jonashaag commented May 27, 2022

mroeschke commented May 27, 2022 • edited Loading

jonashaag commented May 27, 2022

mroeschke left a comment

Choose a reason for hiding this comment

mroeschke commented Jun 3, 2022

jonashaag commented Apr 2, 2022 •

edited by mroeschke

Loading

mroeschke commented May 27, 2022 •

edited

Loading

mroeschke commented May 27, 2022 •

edited

Loading