Changes to how conda activates/deactivates environments may cause install_keras() to fail. #691

psalveson · 2019-03-07T05:07:50Z

The following was produced using R 3.5.2, Rstudio server 1.2 preview, and a new install of Anaconda3 ("conda update --all" run once) on ubuntu 18.04.

I believe there is an issue with using the install_keras() function to install tensorflow/keras to a conda environment. Here's the command I used:

install_keras(method = "conda", 
              conda = "/home/peter/python/anaconda3/condabin/conda", 
              tensorflow = "gpu")

Which produces the following error:

Creating r-tensorflow conda environment for TensorFlow installation...
Collecting package metadata: ...working... done
Solving environment: ...working... done

## Package Plan ##

  environment location: /home/peter/python/anaconda3/envs/r-tensorflow

  added / updated specs:
    - python=3.7


The following NEW packages will be INSTALLED:

  ca-certificates    pkgs/main/linux-64::ca-certificates-2019.1.23-0
  certifi            pkgs/main/linux-64::certifi-2018.11.29-py37_0
  libedit            pkgs/main/linux-64::libedit-3.1.20181209-hc058e9b_0
  libffi             pkgs/main/linux-64::libffi-3.2.1-hd88cf55_4
  libgcc-ng          pkgs/main/linux-64::libgcc-ng-8.2.0-hdf63c60_1
  libstdcxx-ng       pkgs/main/linux-64::libstdcxx-ng-8.2.0-hdf63c60_1
  ncurses            pkgs/main/linux-64::ncurses-6.1-he6710b0_1
  openssl            pkgs/main/linux-64::openssl-1.1.1b-h7b6447c_0
  pip                pkgs/main/linux-64::pip-19.0.3-py37_0
  python             pkgs/main/linux-64::python-3.7.2-h0371630_0
  readline           pkgs/main/linux-64::readline-7.0-h7b6447c_5
  setuptools         pkgs/main/linux-64::setuptools-40.8.0-py37_0
  sqlite             pkgs/main/linux-64::sqlite-3.26.0-h7b6447c_0
  tk                 pkgs/main/linux-64::tk-8.6.8-hbc83047_0
  wheel              pkgs/main/linux-64::wheel-0.33.1-py37_0
  xz                 pkgs/main/linux-64::xz-5.2.4-h14c3975_4
  zlib               pkgs/main/linux-64::zlib-1.2.11-h7b6447c_3


Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done
#
# To activate this environment, use:
# > source activate r-tensorflow
#
# To deactivate an active environment, use:
# > source deactivate
#

Installing TensorFlow...
/bin/bash: /home/peter/python/anaconda3/condabin/activate: No such file or directory
Error: Error 1 occurred installing packages into conda environment r-tensorflow

The most recent versions of conda (starting with 4.4) activate and deactivate environments using

conda activate env_name
conda deactivate

Older versions required use of:

source activate env_name
source deactivate

Which has been depreciated and moved to .../anaconda3/bin/, not .../anaconda3/condabin/.
install_keras() appears to be looking in the wrong place for "activate" and "deactivate", which causes installation to fail.

There's some information here about changes to script initialization for conda 4.6.

UPDATE: Please see my comment below for a resolution.

The text was updated successfully, but these errors were encountered:

psalveson · 2019-03-07T05:15:43Z

Should this be moved to rstudio/tensorflow or rstudio/reticulate?

psalveson · 2019-03-07T06:13:45Z

I corrected the problem by creating a hardlink between .../anaconda3/bin/activate and .../anaconda3/condabin/activate (and the same for deactivate).

jjallaire · 2019-03-07T16:02:03Z

@kevinushey Could you take a look at this one? (should dovetail well w/ the other changes to conda path discovery/munging that are in flight).

kevinushey · 2019-03-07T19:54:40Z

I'll take a look (am mostly OOF this week so may not get to this until next week)

skeydan · 2019-03-10T18:37:57Z

I have conda version 4.5 and never had this problem... could this be because you specify the conda path like this

install_keras(method = "conda", 
              conda = "/home/peter/python/anaconda3/condabin/conda", 
              tensorflow = "gpu")

Can you just leave out the path to conda (as I always do)?

Also, for me with version 4.5 that anaconda3/condabin/ does not exist at all (not for a conda binary as it seems to do for you). The conda binary lives in bin instead.

psalveson · 2019-03-10T22:44:51Z

@skeydan wrote:

I have conda version 4.5 and never had this problem

I encountered the problem using the most up-to-date version of conda, which is conda 4.6.7. I'm unable to comment on earlier versions of conda.

I think you have a point regarding my use of the 'conda' argument. I'm running Rstudio server, and install_keras() is unable to find the conda binary unless I explicitly state its location. I am just now realizing that there are two copies of the conda binary: one in .../anaconda3/bin/ and one in .../anaconda3/condabin/. If I run the following:

install_keras(method = "conda", 
              conda = "/home/peter/python/anaconda3/bin/conda",
              tensorflow = "1.12.0-gpu")

Then keras/tensorflow installs as expected.

Basically, I think this is a case of user (my) error. I pointed install_keras() towards a conda binary, but not the right conda binary. install_keras() assumes that other scripts will be in the same directory as conda, but this wasn't true when I pointed install_keras() to .../anaconda3/condabin/conda.

I'm not sure why there is a directory named "condabin" that contains (only) the conda binary. I didn't add it manually. Perhaps it was added when I ran "conda update -all"?

-Peter

kevinushey · 2019-03-11T06:12:53Z

I suspect condabin is there so that users can modify their PATH to only include the conda binary and not other supporting executables. That said, I think it's reasonable for us to allow both versions to work (and correctly resolve activate if condabin/conda is used)

skeydan · 2019-03-11T07:44:40Z

@kevinushey thanks, could you put a link to those conda-related changes in reticulate here please?

@psalveson you're right, after updating anaconda to the latest version I have that condabin/conda too...

jjallaire · 2019-03-11T10:32:24Z

@kevinushey Note that the tensorflow package has it's own code for installation (as distinct from what is in reticulate) so if we make a fix for anaconda binaries we need to make it in both places. That said, I think we should soon be able to delegate to reticulate from tensorflow so we have only one codepath (but let's not do that just yet).

skeydan · 2019-03-12T07:45:44Z

Hi @psalveson after the fix to reticulate, your original way of doing things should work too :-)
Just update reticulate from github:

devtools::install_github("rstudio/reticulate")

psalveson · 2019-03-12T17:00:30Z

Thanks @skeydan

psalveson · 2019-03-12T17:29:48Z

@skeydan I seem to be having an another issue after updating reticulate. When I run:

install_keras(method = "conda", 
              conda = "/home/peter/python/anaconda3/bin/conda",
              tensorflow = "gpu")

keras_install() exits with the error message:

Error: Specified conda binary 'r-tensorflow' does not exist.

This happens regardless of whether I use the bin or condabin directories, and it happens with multiple versions of tensorflow (I've tried 1.12.0-gpu, 1.13.0-gpu, and 1.13.1-gpu).

traceback() shows:

6: stop("Specified conda binary '", conda, "' does not exist.", 
       call. = FALSE)
5: conda_binary(envname)
4: conda_create(envname, packages = python_packages, conda = conda)
3: install_tensorflow_conda(conda, version, gpu, envname, packages, 
       extra_packages)
2: install_tensorflow(method = method, conda = conda, version = tensorflow, 
       extra_packages = c(paste0("keras", version), extra_packages))
1: install_keras(method = "conda", conda = "/home/peter/python/anaconda3/bin/conda", 
       tensorflow = "gpu")

It looks like conda_binary() is expecting the path of the conda binary, but the environment name is being passed instead.

kevinushey · 2019-03-12T19:18:55Z

I'll take a look!

kevinushey · 2019-03-12T19:44:18Z

Sorry for the trouble. This was indeed a regression in reticulate and should be fixed up now (install latest with devtools::install_github("rstudio/reticulate"))

psalveson · 2019-03-12T19:59:54Z

I updated reticulate and now when I execute:

install_keras(method = "conda", 
              conda = "/home/peter/python/anaconda3/bin/conda",
              tensorflow = "gpu")

It produces the error:

Creating r-tensorflow conda environment for TensorFlow installation...
Collecting package metadata: ...working... done
Solving environment: ...working... done

## Package Plan ##

  environment location: /home/peter/python/anaconda3/envs/r-tensorflow

  added / updated specs:
    - python=3.7


The following NEW packages will be INSTALLED:

  ca-certificates    pkgs/main/linux-64::ca-certificates-2019.1.23-0
  certifi            pkgs/main/linux-64::certifi-2019.3.9-py37_0
  libedit            pkgs/main/linux-64::libedit-3.1.20181209-hc058e9b_0
  libffi             pkgs/main/linux-64::libffi-3.2.1-hd88cf55_4
  libgcc-ng          pkgs/main/linux-64::libgcc-ng-8.2.0-hdf63c60_1
  libstdcxx-ng       pkgs/main/linux-64::libstdcxx-ng-8.2.0-hdf63c60_1
  ncurses            pkgs/main/linux-64::ncurses-6.1-he6710b0_1
  openssl            pkgs/main/linux-64::openssl-1.1.1b-h7b6447c_1
  pip                pkgs/main/linux-64::pip-19.0.3-py37_0
  python             pkgs/main/linux-64::python-3.7.2-h0371630_0
  readline           pkgs/main/linux-64::readline-7.0-h7b6447c_5
  setuptools         pkgs/main/linux-64::setuptools-40.8.0-py37_0
  sqlite             pkgs/main/linux-64::sqlite-3.26.0-h7b6447c_0
  tk                 pkgs/main/linux-64::tk-8.6.8-hbc83047_0
  wheel              pkgs/main/linux-64::wheel-0.33.1-py37_0
  xz                 pkgs/main/linux-64::xz-5.2.4-h14c3975_4
  zlib               pkgs/main/linux-64::zlib-1.2.11-h7b6447c_3


Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done
#
# To activate this environment, use:
# > conda activate r-tensorflow
#
# To deactivate an active environment, use:
# > conda deactivate
#

Error: Unable to find conda binary. Is Anaconda installed?

traceback():

8: stop("Unable to find conda binary. Is Anaconda installed?", call. = FALSE)
7: conda_binary(conda)
6: conda_list(conda = conda)
5: conda_python(envname)
4: conda_create(envname, packages = python_packages, conda = conda)
3: install_tensorflow_conda(conda, version, gpu, envname, packages, 
       extra_packages)
2: install_tensorflow(method = method, conda = conda, version = tensorflow, 
       extra_packages = c(paste0("keras", version), extra_packages))
1: install_keras(method = "conda", conda = "/home/peter/python/anaconda3/bin/conda", 
       tensorflow = "gpu")

I think the problem may have something to do with the argument(s) to conda_python in step 5 above. In my case, the default conda argument ( conda = "auto") fails and conda_python() produces an error . Perhaps explicitly adding the conda argument to conda_python() would fix things?

Ex: conda_python(envname = envname, conda = conda)

No conda argument vs with conda argument:

Restarting R session...

> library(reticulate)
> conda_python("learning")
Error: Unable to find conda binary. Is Anaconda installed?

Restarting R session...

> library(reticulate)
> conda_python("learning", conda = "/home/peter/python/anaconda3/bin/conda" )
[1] "/home/peter/python/anaconda3/envs/learning/bin/python"

DIGRESSION BELOW

I'm not sure if this is related or not, but it appears that reticulate isn't able to find my anaconda installation anymore. If I restart R and run:

library(reticulate)
use_condaenv("base", conda = "/home/peter/python/anaconda3/bin/conda" )
py_config()

The output is:

Restarting R session...

> library(reticulate)
> use_condaenv("base", conda = "/home/peter/python/anaconda3/bin/conda" )
> py_config()
python:         /usr/bin/python3
libpython:      /usr/lib/python3.6/config-3.6m-x86_64-linux-gnu/libpython3.6.so
pythonhome:     /usr:/usr
version:        3.6.7 (default, Oct 22 2018, 11:32:17)  [GCC 8.2.0]
numpy:          /home/peter/.local/lib/python3.6/site-packages/numpy
numpy_version:  1.16.2

python versions found: 
 /usr/bin/python
 /usr/bin/python3
 /usr/local/bin/python3

Previously, this would have shown paths for the "base" environment of my anaconda installation of python.

UPDATE: I may be mistaken. It appears that reticulate can find my installation of anaconda as long as the environment is not the base/default environment and numpy is installed in the environment. Why is it that reticulate refuses use the the base/default environment of anaconda? I understand that it isn't recommended practice, but maybe use_condaenv() should produce an error rather than appearing to succeed?

Also, no error is produced if the condaenv doesn't exist. I'm not sure if this is new behavior or not:

Restarting R session...

> library(reticulate)
> use_condaenv(condaenv = "doesnt_exist", conda = "/home/peter/python/anaconda3/bin/conda" )
> py_config()
python:         /usr/bin/python3
libpython:      /usr/lib/python3.6/config-3.6m-x86_64-linux-gnu/libpython3.6.so
pythonhome:     /usr:/usr
version:        3.6.7 (default, Oct 22 2018, 11:32:17)  [GCC 8.2.0]
numpy:          /home/peter/.local/lib/python3.6/site-packages/numpy
numpy_version:  1.16.2

python versions found: 
 /usr/bin/python
 /usr/bin/python3
 /usr/local/bin/python3

kevinushey · 2019-03-12T21:30:44Z

See: rstudio/reticulate#441

The original concept behind use_python() is that it is advisory (i.e. the same script running on different systems might find Python in different places/environments, so all would be listed). There is a required flag you can add to make sure that the specified version of Python actually exists (an error is thrown if it doesn't)

psalveson · 2019-03-12T21:51:41Z

@kevinushey

Thank you. I'll know to add require = TRUE in the future.

psalveson · 2019-03-12T22:00:49Z

These are edits to a previous comment. Adding them here for visibility.

> library(keras)
> install_keras(method = "conda", 
+               conda = "/home/peter/python/anaconda3/bin/conda",
+               tensorflow = "gpu")

(output truncated)

Error: Unable to find conda binary. Is Anaconda installed?

traceback():

8: stop("Unable to find conda binary. Is Anaconda installed?", call. = FALSE)
7: conda_binary(conda)
6: conda_list(conda = conda)
5: conda_python(envname)
4: conda_create(envname, packages = python_packages, conda = conda)
3: install_tensorflow_conda(conda, version, gpu, envname, packages, 
       extra_packages)
2: install_tensorflow(method = method, conda = conda, version = tensorflow, 
       extra_packages = c(paste0("keras", version), extra_packages))
1: install_keras(method = "conda", conda = "/home/peter/python/anaconda3/bin/conda", 
       tensorflow = "gpu")

I think the problem may have something to do with the argument(s) to conda_python in step 5 above. In my case, the default conda argument ( conda = "auto") fails and conda_python() produces an error . Perhaps explicitly adding the conda argument to conda_python() would fix things?

Ex: conda_python(envname = envname, conda = conda)

No conda argument vs with conda argument:

Restarting R session...

> library(reticulate)
> conda_python("learning")
Error: Unable to find conda binary. Is Anaconda installed?

Restarting R session...

> library(reticulate)
> conda_python("learning", conda = "/home/peter/python/anaconda3/bin/conda" )
[1] "/home/peter/python/anaconda3/envs/learning/bin/python"

kevinushey · 2019-03-13T02:51:47Z

Thanks for continuing to test. I believe rstudio/reticulate@7edf564 should capture what you were recommending; let me know if that helps.

psalveson · 2019-03-13T04:55:20Z

@kevinushey

I updated reticulate and everything appears to be working correctly. Thanks for all your help.

skeydan · 2019-03-13T09:05:34Z

You're welcome!

kevinushey mentioned this issue Mar 11, 2019

prefer bin/conda to condabin/conda rstudio/reticulate#451

Merged

skeydan closed this as completed Mar 12, 2019

skeydan reopened this Mar 12, 2019

skeydan closed this as completed Mar 13, 2019

dfalbel mentioned this issue Aug 7, 2019

py_config and py_discover_config rstudio/reticulate#568

Closed

harshvardhaniimi mentioned this issue Feb 18, 2022

Cannot install keras on Mac M1 #1309

Open

Changes to how conda activates/deactivates environments may cause install_keras() to fail. #691

Changes to how conda activates/deactivates environments may cause install_keras() to fail. #691

Comments

psalveson commented Mar 7, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

psalveson commented Mar 7, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

psalveson commented Mar 7, 2019

Uh oh!

jjallaire commented Mar 7, 2019

Uh oh!

kevinushey commented Mar 7, 2019

Uh oh!

skeydan commented Mar 10, 2019

Uh oh!

psalveson commented Mar 10, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kevinushey commented Mar 11, 2019

Uh oh!

skeydan commented Mar 11, 2019

Uh oh!

jjallaire commented Mar 11, 2019

Uh oh!

skeydan commented Mar 12, 2019

Uh oh!

psalveson commented Mar 12, 2019

Uh oh!

psalveson commented Mar 12, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kevinushey commented Mar 12, 2019

Uh oh!

kevinushey commented Mar 12, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

psalveson commented Mar 12, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

DIGRESSION BELOW

Uh oh!

kevinushey commented Mar 12, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

psalveson commented Mar 12, 2019

Uh oh!

psalveson commented Mar 12, 2019

Uh oh!

kevinushey commented Mar 13, 2019

Uh oh!

psalveson commented Mar 13, 2019

Uh oh!

skeydan commented Mar 13, 2019

Uh oh!

psalveson commented Mar 7, 2019 •

edited

Loading

psalveson commented Mar 7, 2019 •

edited

Loading

psalveson commented Mar 10, 2019 •

edited

Loading

psalveson commented Mar 12, 2019 •

edited

Loading

kevinushey commented Mar 12, 2019 •

edited

Loading

psalveson commented Mar 12, 2019 •

edited

Loading

kevinushey commented Mar 12, 2019 •

edited

Loading