Auto conversion workflow?

[icweaver]

I really appreciate how PythonCall.jl avoids automatic conversions, but I was wondering if there might also be a way to enable this conditionally?

The case I have been running into is that a colleague sends me some pickle or npz file with a bunch of nested, heterogeneous Python objects which I would then like to feed into my Julia pipeline. I've been sprinkling pyconverts or Pyobjects in multiple places before passing the inputs to various Julia functions to get the expected behavior, but wasn't sure if I was approaching this the right way.

I know it would be a hit in performance in the front-end, but is there a way to accomplish this for all python objects that I load (perhaps with auto conversion rules that we specify), or maybe some workflow more in line with this package's philosophy that I am overlooking?

As a basic example, if I have the following pickle:

import pickle

data = {"entry1":[1, 2, 3], "entry2":{"entry2_1": [3, 2, 1]}}
with open("./data.pkl", "wb") as f:
    pickle.dump(data, f)

is there a way to load it in Julia as the following native object?

using PythonCall
@py import pickle

data = pywith(pickle.load, open("./data.pkl"))

# <auto conversion flag here?>

# Dict{String, Any} with 2 entries:
#  "entry1" => [1, 2, 3]
#  "entry2" => Dict("entry2_1"=>[3, 2, 1])

instead of wrapping with:

PyDict{String, Any}(data)
# PyDict{String, Any} with 2 entries:
#  "entry1" => Py[1, 2, 3]
#  "entry2" => PyDict{Py, Py}('entry2_1'=>[3, 2, 1])

So sort of like the PythonCall.jl inverse of py"<>"o for returning raw PyObjects in PyCall.jl?

[cjdoris]

There's nothing to easily achieve this right now.

You can do pyconvert(Dict{Any,Any}, x) (or with a union of types if you like) but this isn't deep, in the sense that any nested objects are still converted in the default way.

You could write your own pyconvertnative(x) to convert native Python types to native Julia ones by checking the type of x (is it None, bool, int, float, list, etc...) and converting it by hand. Could be a nice addition to PythonCall.

[icweaver]

Ok, thanks for getting back to me so quickly and for letting me know that I wasn't missing anything obvious. I will definitely have to give this a shot some time

[cjdoris]

Thinking about it, a more composible solution could be to define struct PyNative end and add a rule so that pyconvert(PyNative, x) works like you want.

It's not quite so simple though because lots of code reasonably assumes that converting to a T actually returns a T, which this wouldn't.

[icweaver]

Ah, thanks for the additional insight!

[hhaensel]

I started working on this. Would be nice, if someone with deeper knowledge could comment.

using PythonCall

const PYTYPES = setdiff(getindex.(getfield.(getfield.(methods(ispy), :sig), :types), 2), [Any])

function pyconvert_native(x)
    ispy(x) && PythonCall.pyisinstance(x, pybuiltins.dict) && return pyconvert_dict(x)

    types = setdiff(getfield.(PythonCall._pyconvert_get_rules(pytype(x)), :type), PYTYPES)
    for T in types 
        ans = PythonCall.pytryconvert(T, x)
        PythonCall.pyconvert_isunconverted(ans) || return PythonCall.pyconvert_result(T, ans)
    end
    PythonCall.pyconvert_unconverted()
end

function pyconvert_dict(::Type{T}, d) where T
    jd = pyconvert(T, d)
    kk = pyconvert_native.(keys(jd))
    vv = [ispy(v) ? PythonCall.pyisinstance(v, pybuiltins.dict) ? pyconvert_dict(T, v) : pyconvert_native(v) : v for v in values(jd)]

    T(zip(kk, vv))
end

pyconvert_dict(d) = pyconvert_dict(Dict, d)

which allows us to write the following code snippet

julia> pl = pylist([1, 2, 3]);
julia> ps = pystr("H");
julia> pd = @py dict({"a":"bb", "c": dict({"e": "f"})});

julia> pyconvert_native(pl)
3-element Vector{Int64}:
 1
 2
 3

julia> pyconvert_native(ps)
"H"

julia> pyconvert_native(pd)
Dict{String, Any} with 2 entries:
  "c" => Dict("e"=>"f")
  "a" => "bb"

julia> pyconvert_dict(d)
Dict{String, Any} with 2 entries:
  "c" => Dict("e"=>"f")
  "a" => "bb"

julia> pyconvert_dict(Dict{String, Any}, d)
Dict{String, Any} with 2 entries:
  "c" => Dict{String, Any}("e"=>"f")
  "a" => "bb"

julia> pyconvert_dict(Dict{Any, Any}, d)
Dict{Any, Any} with 2 entries:
  "c" => Dict{Any, Any}("e"=>"f")
  "a" => "bb"

[hhaensel]

We can also switch to a similar issue #172 for further discussion.

[hhaensel]

One more thought:
Could it be reasonable to cache _pyconvert_get_rules(pytype::Py) ?

[hhaensel]

I implemented caching, so my current code looks like this:

using PythonCall

const PYTYPES = push!(setdiff(getindex.(getfield.(getfield.(methods(ispy), :sig), :types), 2), [Any]), PyArray)
const PYCONVERT_NATIVE_TYPES_CACHE = Dict{PythonCall.C.PyPtr, Vector{Type}}()

function pyconvert_get_native_ruletypes(x::Py)
    tptr = PythonCall.C.Py_Type(PythonCall.getptr(x))
    get!(PYCONVERT_NATIVE_TYPES_CACHE , tptr) do
        pytype = PythonCall.pynew(PythonCall.incref(tptr))
        ans = setdiff(getfield.(PythonCall._pyconvert_get_rules(pytype), :type), PYTYPES)
        PythonCall.pydel!(pytype)
        ans
    end
end

function pyconvert_native(x)
    ispy(x) || return x

    PythonCall.pyisinstance(x, pybuiltins.dict) && return pyconvert_dict(x)

    types = pyconvert_get_native_ruletypes(x)
    for T in types 
        ans = PythonCall.pytryconvert(T, x)
        PythonCall.pyconvert_isunconverted(ans) || return PythonCall.pyconvert_result(T, ans)
    end
    PythonCall.pyconvert_unconverted()
end

function pyconvert_dict(::Type{T}, d) where T
    jd = pyconvert(T, d)
    kk = pyconvert_native.(keys(jd))
    vv = [ispy(v) ? PythonCall.pyisinstance(v, pybuiltins.dict) ? pyconvert_dict(T, v) : pyconvert_native(v) : v for v in values(jd)]

    T(zip(kk, vv))
end

pyconvert_dict(d) = pyconvert_dict(Dict, d)

If this approach looks ok, I would try to cover arrays in a similar manner to dicts.
@cjdoris BTW, is there a reason that ispy(PyArray) == false

[hhaensel]

With this in place, the following two lines add support for (automatic) DataFrame conversion:

# add DataFrame conversion rule
using DataFrames

PythonCall.pyconvert_rule_pandasdataframe(::Type{DataFrame}, x::Py) = DataFrame(PyTable(x))
PythonCall.pyconvert_add_rule("pandas.core.frame:DataFrame", DataFrame, PythonCall.pyconvert_rule_pandasdataframe, PythonCall.PYCONVERT_PRIORITY_NORMAL)

so

julia> pd_df = pd.DataFrame([[1, 2, 3], [4, 5, 6]], columns = pylist("ABC"))
Python DataFrame:
   A  B  C
0  1  2  3
1  4  5  6

julia> pyconvert_native(pd_df)
2×3 DataFrame
 Row │ A      B      C     
     │ Int64  Int64  Int64 
─────┼─────────────────────
   1 │     1      2      3
   2 │     4      5      6

[hhaensel]

@cjdoris Meanwhile I also covered arrays and implemented support for AbstractDict types during conversion, if you are interested, I can prepare a PR.

[cjdoris]

Thanks, I've been thinking a lot lately about how various aspects of PythonCall work, including conversion. I'm considering (a) removing pyconvert and just overloading Base.convert instead; and (b) converting to native types by default (e.g. list to Vector, numpy.ndarray to Array) instead of wrapper types. This would be much more similar to PyCall's behaviour, with the exception that we still won't automatically convert anything.

[bryaan]

@cjdoris Any updates on your auto conversion ideas?

[cjdoris]

Yeah sorry I've not had loads of time for PythonCall dev lately, and what time I do have has mostly been thinking about v1, trying to stabilise and finalise the API. This will include a more general pyconvert which will support to-native conversion but it's a way off. As a result I don't really want to add new functionality right now - but it looks like you can achieve what you want externally anyway, which is good.

The new package extensions stuff is going to be really useful for conversion - it will allow us to add conversion rules for all sorts, such as DataFrames in your example.

[hhaensel]

Good to hear, really looking forward to the next version. Feel free to use any of my code above, of course, and let me know if you need help.
Meanwhile I had a python dict including python lists, which needed another definition - just in case someone reads this and want to use my solution before the next version comes out.

pyconvert_get_native_ruletypes(::PyList{T}) where T = [Vector{T}, Vector]