Launching Python in Elixir

We are all in love with Elixir. It deals perfectly with concurrency, has easy syntax and a really helpful community. It is really a match for standard web applications. However there are situations where Elixir can have some performance problems - for example when it comes to longer calculations, including matrixes or some kind of optimization.

It is surprising that in Elixir, according to José Valim’s post, the problem is not the mathematical processing, but rather the lack of support for large, mutable structures. Due to the immutable nature of data in Elixir, you really need a lot of copying of immutable structures to perform a simple matrix operation.

When you come across such a problem, which includes large mathematical operations, what is the first solution coming to your mind?

Probably it is using other languages.

But wait... Isn’t the whole application already written in Elixir? That is why today we are going to learn how to launch other languages from Elixir.

There are a few possible approaches to this problem. In this article we are going to learn two, probably most common, of them.

The first one is using an Elixir’s dependency - :erlport. However its use case is limited, because it supports only two languages - Python and Ruby.

The second approach is to call the executable file directly using System.cmd/3. This one, however, is a bit tricky to handle, but can be used with any language exported to the executable. Because of the :erlport limitations we will be using Python to be the support language for this article.

:erlport

First things first - we need to include the :erlport dependency in the mix.exs. To do so, simply add {:erlport, "~> 0.10.1"} to the dependencies list in the previously mentioned file.

:erlport gives us access to the :python module, which implements functions crucial for our Python code to work. First of all, we will need to start a Python instance. To do so, we will use the :python.start/1 function. This function has plenty of options to configure, which can be read in the documentation. Then we will call the Python instance via :python.call/4 function to launch our Python code. That is why, to keep :python functions together and have nice, clean code, we are going to need a new module. Let’s call it PythonHelper.

For this, we will need the startinstance/2_ function. Its responsibility, as the name says, is to start a Python instance. It takes two arguments - path and version. 'path' determines the project path to Python files, basically where we store them. The version parameter determines the Python version - when you have both python 3.x and python 2.x installed on your device. This function calls :python.start/1 with previously mentioned arguments as the keyword list. It returns {:ok, pid} once the instance initialization is successful.

defmodule MyApp.PythonHelper do

 def start_instance(path, version \\ 'python') do
   :python.start([{:python_path, path}, {:python, version}])
 end

end

Then we need to call the created instance. That is the responsibility of the callinstance/4_ function. It takes the instance pid, filename as module, function and args as arguments and calls the instance. args should be a list of arguments for a Python function. callinstance/4_ will return the result of the called function.

defmodule MyApp.PythonHelper do

 def start_instance(path, version \\ 'python') do
   :python.start([{:python_path, path}, {:python, version}])
 end

 def call_instance(pid, module, function, args \\ []) do
   :python.call(pid, module, function, args)
 end

end

At the end of the day it is nice to have some function for turning off the instance, so that our PythonHelper module is complete.

defmodule MyApp.PythonHelper do

 def start_instance(path, version \\ 'python') do
   :python.start([{:python_path, path}, {:python, version}])
 end

 def call_instance(pid, module, function, args \\ []) do
   :python.call(pid, module, function, args)
 end

 def stop_instance(pid) do
   :python.stop(pid)
 end
end

The only thing left is to create our example .py file. I like putting them in the priv/python project directory, because it is easily accessible via :code.privdir/1 function. Let’s name our python file _hello.py and create a simple function returning a greeting text with our argument.

def welcome(world):
 return "".join(["Hello", " ", world.decode("utf-8")])

Testing the Python module in the shell

Now we can test the coded module in our iex shell:

iex(12)> path = [:code.priv_dir(:my_app), "python"] |> Path.join() |> to_charlist()
'/Users/my_user/Desktop/mixing_python/my_app/_build/dev/lib/shopping_list/priv/python'
iex(13)> {:ok, pid} = PythonHelper.start_instance(path, 'python3')
{:ok, #PID<0.518.0>}
iex(14)> PythonHelper.call_instance(pid, :hello, :welcome, ["World"])
'Hello World'
iex(15)> PythonHelper.stop_instance(pid)
:ok

As you can see, it is working pretty good. It is worth remembering that the result in the .py file was a string, but the value passed to Elixir is a char list. You can also notice PythonHelper.startinstance/2_ is called with the ‘python3’ argument. That is because the machine it was run on had two versions of Python installed. One of them was python 2.x, called in the terminal via python command, and the other one was python 3.x, called in the terminal via python3 command.

Creating Python instances in Elixir

When it comes to the Python instances, there are a few possibilities of creating them - one is to create them directly before each Python call and close it right after with :python.stop/1, so it doesn’t hang in the memory, like we did while testing our module in the shell. It is a good fit when you want to call the Python module once in a while. For more frequent calls it is nice to have a hanging Python instance, which will handle those calls without repeatedly opening and shutting down. To do so, we can leverage the Elixir’s GenServer behavior.

Let’s start by creating a standard Elixir’s GenServer module. In the init/1 function we are calling the same function as we did in the terminal. Since the PythonHelper.startinstance/2_ returns {:ok, pid} we do not need another {:ok, state} at the end of the init/1. Our GenServer will store the Python process PID as its state.

defmodule MyApp.PythonServer do
 use GenServer

 def start_link() do
   GenServer.start_link(__MODULE__, nil, name: __MODULE__)
 end

 def init(_) do
   [:code.priv_dir(:shopping_list), "python"]
   |> Path.join()
   |> to_charlist()
   |> PythonHelper.start_instance('python3')
 end

end

Now we will make use of our previously written module. In the callfunction/3_ we are going to use GenServer.call/3 to make a call to the GenServer process. In the handlecall/3_ we will handle the incoming message with given parameters and use the PythonHelper module to call the Python instance with the pid stored in GenServer state.

defmodule MyApp.PythonServer do
 use GenServer

 alias ShoppingList.PythonHelper

 ...

 def call_function(module, function, args) do
   GenServer.call(__MODULE__, {:call_function, module, function, args})
 end

 def handle_call({:call_function, module, function, args}, _from, pid) do
   result = PythonHelper.call_instance(pid, module, function, args)
   {:reply, result, pid}
 end
end

Remember to add PythonServer to the supervision tree in the application.ex and you are ready to go. We can run our application with iex -S mix and test the written code.

iex(1)> MyApp.PythonServer.call_function(:hello, :welcome, ["World"])
'Hello World'

This solution looks pretty solid at the first glance. The problem is that for the longer calculations we can reach the GenServer.call/3 default timeout. This can be easily solved by adding :infinity as the timeout argument to GenServer.call/3 function. However it doesn’t solve the problem of process blocking while waiting for the call response.

The solution is to use :python.cast/4 instead of :python.call/4. It is used to send a message to Python, instead of calling a function directly. To do so we need to adjust our .py file to handle those messages. In the .py file we will need to use erlport import, which is automatically injected to your python instance, so you do not need to install it via pip.

from erlport.erlang import set_message_handler, cast
from erlport.erlterms import Atom

message_handler = None

def cast_message(pid, message):
 cast(pid, message)

def handle_message(message):
 print("Received message from Elixir")
 print(message)
 if message_handler:
   result = welcome(message)
   cast_message(message_handler, (Atom(b'python'), result))

def register_handler(pid):
 global message_handler
 message_handler = pid

def welcome(world):
 return "".join(["Hello", " ", world.decode("utf-8")])

set_message_handler(handle_message)

We need to add also 3 new functions: cast_message/2, responsible for passing data back to Elixir using the cast function imported from erlport, handlemessage/1, which is responsible for handling incoming messages and _registerhandler/1, which is responsible for setting up _messagehandler, which is basically the Elixir’s sender process PID. The only thing left is to add _setmessagehandler(handlemessage)_ at the end of the file, which sets up our previously mentioned function as the messages handler.

Now we need to add :python.cast/2 to our PythonHelper module, which is pretty straightforward.

defmodule MyApp.PythonHelper do

 ...

 def cast_instance(pid, message) do
   :python.cast(pid, message)
 end

end

Adding functionalities to the PythonServer module

And the last step is to add new functionalities to the PythonServer module. We are going to use GenServer.cast/2 since we do not expect any response after calling the Python instance. However after casting the message we expect the instance to send us a message once it finishes its work. That is why we need the handleinfo/2 function_.

defmodule MyApp.PythonServer do
 use GenServer

 ...

 def cast_function(message) do
   GenServer.cast(__MODULE__, {:cast_function, message})
 end

 ...

 def handle_cast({:cast_function, message}, pid) do
   PythonHelper.cast_instance(pid, message)
   {:noreply, pid}
 end

 def handle_info({:python, message}, pid) do
   IO.inspect("Received message from Python")
   IO.inspect message
   {:noreply, pid}
 end
end

Finally, we can run our application with iex -S mix and test the written code.

iex(1)> ShoppingList.PythonServer.cast_function("World")
Received message from Elixir
b'World'
"Received message from Python"
:ok
iex(2)> 'Hello World'

This approach is really great because it is asynchronous and other operations can be done while awaiting the Python code. This solution can also be scaled by launching a pool of Python instances and monitoring if they are already taken by some calculations - binding the occupied flag when sending the cast message and unbinding it when the message comes back - and thus breaking down the traffic between them.

System.cmd/3

This approach is pretty straightforward. However handling communication might be a bit tricky. We will use System.cmd/3 function, which takes 3 arguments:

• command which is expected to be executable available in PATH,
• arguments which should be given as the list of binaries,
• opts which we will not use right now.

Now let’s write our module for calling system commands. I named it simply CmdCaller. Then write a function called callpython/3, which takes 3 arguments: _cmd, as we might want to call different python versions, path to the .py file and arguments, which will be our list of arguments to the cmd call.

defmodule MyApp.CmdCaller do

 def call_python(cmd, path, arguments) do
   case System.cmd(cmd, [path | arguments]) do
     {output, 0} ->
       output
       |> String.replace("\n", "")
       |> Jason.decode!()
     {_, 1} -> {:error, "Python code / call error"}
   end
 end
end

The trick is that calling System.cmd/3 will return us the command line output of the executable. That is why we will need to ensure later on that our .py file is printing things in the right format to the cmd. I like using JSONs and that is why I assume that the printed output will have such format. There is also error handling with case do, to ensure that the function will return {:error, reason} tuple once there is something wrong with our command call.

Then we need to adjust our .py file. We will use the click library to make the file executable from the command line. To encode the result to the JSON format we will use the json library. It is worth remembering that once we call the executable from the command line we do not need to parse data from bytes to string with decode(“utf-8”), as we did with :erlport. As you can notice the function does not return anything. It prints the result to the command line - it is because our Elixir function reads the console output of the launched command.

import click
import json

@click.command()
@click.argument('world', required=1)
def welcome(world):
 result = {
   "result":  "".join(["Hello", " ", world])
 }

 print(json.dumps(result))

if __name__ == '__main__':
 welcome()

Now we can run our application with iex -S mix and test the written code.

iex(1)> MyApp.CmdCaller.call_python("python3", "/Users/username/Desktop/my_app/priv/python/hello_2.py", ["World"])
%{"result" => "Hello World"}

This approach is much faster to implement than the first one. However it is really inconvenient to handle the communication between the Python code and the Elixir application. This solution might be used however while the Python code is rather small and you do not want to waste a lot of time coding the whole background for :erlport and digging into its documentation. Also it does not provide elasticity of the first solution, which can be adjusted to the problem size.

Summary

The first solution is really scalable and might be adjusted to the wide range of problems. However it takes a while to implement and personally I came across some problems with launching Python libraries while using :erlport. The second one is really straightforward and easy to code. However it does not provide such elasticity and scalability as the first one. It is however really nice when you need some side Python code to do its job once a while.

There are also other possible solutions:

• to create a Python server with API and callbacks, and to communicate with it via localhost,
• using gRPC - remote procedure call system by Google.

You can also consider using them, but they are rather used in the communication between bigger projects and not while using a few Python support modules.