Problem importing camelot

[marco-zanon]

My Environment:
Operating system and version: Windows 10
Python version: 3.8.3
How did I install python: Anaconda default Python installation
Did I use a virtualenv?: yes, I use a virtualenv to pack all the libraries that I need for the specific project
My Rust version (rustc --version): 1.59.0
MyPyO3 version: 0.16.4

I am new to Rust.

I need to read some tables from pdf files, and did´t find any satisfactory high-level Rust crate. So I decided to use camelot (Python) which I had used successfully before, and call it from inside my Rust program.

First I was having all sorts of problems with PyO3 not finding my Python interpreter, and while researching, I found it could be solved by setting PYTHONHOME and PYTHONPATH. It did solve this problem, but broke my Anaconda installation, so I researched a bit further and found that cgranade solution to issue #1554 can solve it.

But now I have a problem with importing the camelot library. It is inside a virtua env and I reckon I should probably point to the Pyrhon version that is inside the virtual env. This is my code:

use pyo3::prelude::*;
use pyo3::types::IntoPyDict;
use which;

pub fn initialize_python() -> PyResult<()> {
    // Due to https://github.com/ContinuumIO/anaconda-issues/issues/11439,
    // we first need to set PYTHONHOME. To do so, we will look for whatever
    // directory on PATH currently has python.exe.
    let python_exe = which::which("python").unwrap();
    let python_home = python_exe.parent().unwrap();

    //virtual env 'pdf_extract' contains the camelot package
    let env_name = "pdf_extract";
    let env_file = python_home.join("envs").join(env_name);

    // The Python C API uses null-terminated UTF-16 strings, so we need to
    // encode the path into that format here.
    // We could use the Windows FFI modules provided in the standard library,
    // but we want this to work cross-platform, so we do things more manually.
    let mut env_file = env_file
        .to_str()
        .unwrap()
        .encode_utf16()
        .collect::<Vec<u16>>();
    env_file.push(0);
    unsafe {
        pyo3::ffi::Py_SetPythonHome(env_file.as_ptr());
    }

    // Once we've set the configuration we need, we can go on and manually
    // initialize PyO3.
    pyo3::prepare_freethreaded_python();

    Ok(())
}

fn main() {
    let key1 = "filepath";
    let val1 = "./Teste4.pdf";
    let key2 = "pages";
    let val2 = "3-end";
    initialize_python().unwrap();
    Python::with_gil(|py| {
        let camelot = PyModule::import(py, "camelot").unwrap();

        let kwargs = vec![(key1, val1), (key2, val2)];
        let tables = camelot.call((), Some(kwargs.into_py_dict(py))).unwrap();

        println!("{}", tables);

    })
}

And this is the full error I get:

C:\Users\mazrodrigues\Anaconda3\envs\pdf_extract\lib\site-packages\numpy\__init__.py:148: UserWarning: mkl-service package failed to import, therefore Intel(R) MKL initialization ensuring its correct out-of-the box operation under condition when Gnu OpenMP had already been loaded by Python process is not assured. Please install mkl-service package, see http://github.com/IntelPython/mkl-service from . import _distributor_init thread 'main' panicked at 'called Result::unwrap()on anErrvalue: PyErr { type: <class 'ImportError'>, value: ImportError('\n\nIMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!\n\nImporting the numpy C-extensions failed. This error can happen for\nmany reasons, often due to issues with your setup or how NumPy was\ninstalled.\n\nWe have compiled some common reasons and troubleshooting tips at:\n\n https://numpy.org/devdocs/user/troubleshooting-importerror.html\n\nPlease note and check the following:\n\n * The Python version is: Python3.8 from "C:\\Users\\mazrodrigues\\Documents\\10_Rust\\01_Treinamento\\01_Genericos\\05_py03\\v06\\target\\debug\\v01.exe"\n * The NumPy version is: "1.21.5"\n\nand make sure that they are the versions you expect.\nPlease carefully study the documentation linked above for further help.\n\nOriginal error was: DLL load failed while importing _multiarray_umath: Não foi possível encontrar o módulo especificado.\n'), traceback: Some(<traceback object at 0x000002666038F900>) }', src\main.rs:62:55 note: run withRUST_BACKTRACE=1environment variable to display a backtrace error: process didn't exit successfully:target\debug\v01.exe (exit code: 101)

By the error message it seems that there is something wrong with my Numpy installation (which is a dependency of camelot), but when I run with Python in Jupyter Notebook it works fine, so there is nothing wrong with the installation.

Can someone help me with this?

[davidhewitt]

The root of the cause is most likely that environment variables are not set up correctly. There's some previous comments about this at #1554 and #1896.

You might want to try out PyOxidizer, which can pack all of your Rust and Python code together and might help avoid you needing to figure out a solution.

[davidhewitt]

Will close this issue as there's been no further discussion in the past 4 months.

[dbsxdbsx]

@davidhewitt, now I also ran into the DLL load failed while importing issue. I am using opencv-rs on win10 with pyo3 with some cargo.toml defined like this:

[dependencies]
opencv = {version = "0.82.1"}

[dependencies.pyo3]
version = "0.19.0"
features = ["abi3-py38","extension-module"] # "abi3-py38" tells pyo3 (and maturin) to build using the stable ABI with minimum Python version 3.8

Ｉ did make opencv work out in pure rust project. But here in a lib project with pyo3, I failed when using the final produced python module. I wonder what I need to do to fix it.

I've tried to add the file path manually by:

opencv_lib_path = os.path.abspath(r'C:\tools\opencv\build\x64\vc16\bin')
sys.path.append(opencv_lib_path)

But still, not working.=_=

[davidhewitt]

@dbsxdbsx check that C:\tools\opencv\build\x64\vc16\bin is in your PATH environment variable, that may fix it?

[dbsxdbsx]

@dbsxdbsx check that C:\tools\opencv\build\x64\vc16\bin is in your PATH environment variable, that may fix it?

Yes, it is in PATH, but still not fixed.

@davidhewitt , do you mean exactly PATH but not Path? In win10, the system path env is called Path?