refactor py_str macro to allow passing custom global/locals

[marius311]

This should have no change to functionality, but the new @_py_str macro can be used by other functions to pass custom globals/locals dicts into Python expression while still getting the benefit of @py_str's variable interpolation.

@_py_str is kept a macro rather than function because it makes downstream macro hygene a bit easier.

I'm using this here JuliaPy/pyjulia#381, where this can maybe be discussed more.

[stevengj]

Could you clarify why this should be a macro?

[tkf]

If we are going to use a function for this, it should take a named tuple (__module__=__module__, __source__=__source__) as the first argument so that all the information available within a usual macro is available. I think it's better to make it a single first argument rather than two arguments because future versions of julia may introduce some more information within a macro definition but it'd be bad to change the calling convention of this function.

[tkf]

Also, we should use options rather than options....

[marius311]

Could you clarify why this should be a macro?

So it stems from the fact that inside @prepare_for_pyjulia_call we do a complicated recursive walk through the whole expression to transform it. Its really nice to just be able to stick an esc around the whole transformed thing at the end here, otherwise you have to think (hard) about where to stick in esc's throughout that walk. Maybe there's a simple way, but all my playing with it it got really complex.

Given that you want to esc the whole thing, then whatever expression you get from calling into PyCall can't have bare symbols like PyObject or pyeval_ in it, they need to be turned into PyCall.PyObject and PyCall.pyeval_. You can do this by modifing things on the PyCall side, but it seems easier to just make @_py_str a macro and then hygene does this for you. That's basically my motivation.

That said, I realize now all the macroexpand(__module__, ... stuff is unnecessary so I'll get rid of it in both PRs.

[stevengj]

Why is this call to esc necessary, since there are esc calls within @_py_str?

Indeed, the call to esc here seems like it might cause problems, because it prevents the names pylocals, pyglobals, and ret from being hygienized.

[marius311]

No, pylocals etc... get hygenized by @_py_str, the esc here makes it so the interpolated variables don't also get hygenized, e.g. with the code in this PR:

julia> @macroexpand py"1+$i"
quote
    #= /home/marius/src/pyjulia/dev/PyCall/src/pyeval.jl:229 =#
    (var"#18#pyglobals", var"#19#pylocals") = (PyCall.pynamespace(Main), PyCall.pynamespace(Main))
    #= /home/marius/src/pyjulia/dev/PyCall/src/pyeval.jl:230 =#
    begin
        var"#19#pylocals"["__julia_localvar_2_1"] = PyCall.PyObject(i)
    end
    #= /home/marius/src/pyjulia/dev/PyCall/src/pyeval.jl:231 =#
    var"#20#ret" = (PyAny)(PyCall.pyeval_((string)("1+__julia_localvar_2_1"), var"#18#pyglobals", var"#19#pylocals", 258, "none"))
    #= /home/marius/src/pyjulia/dev/PyCall/src/pyeval.jl:232 =#
    begin
        PyCall.delete!(var"#19#pylocals", "__julia_localvar_2_1")
    end
    #= /home/marius/src/pyjulia/dev/PyCall/src/pyeval.jl:233 =#
    var"#20#ret"
end

[tkf]

Maybe I should've asked this before but why is insert_pyevals run at run-time? It looks like this doesn't depend on the run-time value of globals and locals (or maybe I'm missing something)? If we expand everything at macro-expansion time, perhaps esc of @py_str here and @prepare_for_pyjulia_call there can be removed? Then maybe _py_str doesn't have to be a macro?

[tkf]

Suggested change

	esc(:(PyCall.@_py_str($m, $m, $fname, $code, $(options...))))
	esc(:($PyCall.@_py_str($m, $m, $fname, $code, $(options...))))

I think you need this?

I think we better test py"" with something like

baremodule TestPyStr
using PyCall: @py_str
pystr(x) = py"$$x"
end
@test TestPyStr.pystr("1+1") == 2