Erlang Team Storage. How does the Elixir data engine work?

What is ETS in Elixir?

ETS (Erlang Term Storage) is a fast in-memory data storage, able to store big amounts of data. It comes built-in with Elixir and it allows to store any Elixir term.

Each table in ETS is created and owned by a separate process, which complies with the standard of agents model. This has of course its downsides: when the process terminates, the table is destroyed and the data stored in the table vanishes.

Erlang Term Storage types

ETS in Elixir has 4 table types, which differ in terms of uniqueness of key values and data order:
• :set - the default one, one key corresponds with one row, once you save a row with the same key, it is being overwritten,
• :ordered_set - very similar to :set, but keys are ordered by Elixir/Erlang term,
• :bag - one key can have multiple rows, but rows are required to be unique,
• :duplicate_bag - one key can have multiple rows and rows can duplicate.

In the beginning, it is often hard to determine which table type we need.

ETS access control types

Erlang Term Storage has also 3 access control types:
• :public - read/write functions are available to all other processes,
• :protected - default one, read is available to all other processes, write - only to owner process
• :private - read/write available to the owner processes only.

Do you want to learn more about Elixir? Take a look at our freeElixir online course which will lead you through its fundamental concepts!

Wrapping ETS functions with GenServer

Once we understand the basics, we can jump into some code. As an example we are going to build a simple GenServer capable of saving, looking for and deleting data from the ETS table.

First we are going to build an EtsService module responsible for ETS interactions. We are going to use GenServer, because it will play the role of a communication hub between our application and the ETS table.

In the init/1 function there I am initializing a new ETS table with the :ets.new/2 function. The first argument represents a table name and the second one is a list of options. I am passing there :set as the table type and read_concurrency: true, which will optimize the table to the concurrent read operations according to the ETS documentation. The GenServer state will serve as the ETS table process PID storage.

defmodule MyApp.EtsService do
 use GenServer

 @table_name :my_first_ets_table

 def start_link(_) do
   GenServer.start_link(__MODULE__, nil, name: __MODULE__)
 end

 def init(_init_state) do
   auth_table_pid = :ets.new(@table_name, [:set, read_concurrency: true])
   {:ok, auth_table_pid}
 end
end

The first thing we want to do is to make some kind of interface connecting the GenServer to the outer world. We are going to write 3 simple functions: insert_data/2, find_data/1, and delete_data/1.

defmodule MyApp.EtsService do

 …

 def insert_data(key, value) do
   GenServer.cast(__MODULE__, {:insert, {key, value}})
 end

 def find_data(key) do
   GenServer.call(__MODULE__, {:find, key})
 end

 def delete_data(key) do
   GenServer.cast(__MODULE__, {:delete, key})
 end
end

These functions are calling GenServer.cast/2 and GenServer.call/2 to communicate with the GenServer, which will handle the messages. GenServer.cast/2 is used when you don’t expect the GenServer to respond. On the other hand, GenServer.call/2 will await the GenServer response.

Then we are going to add messages handling - that is two handle_cast/2 for inserting and deleting data and one handle_call/3 for looking for data.

defmodule MyApp.EtsService do

 ...

 def handle_cast({:insert, {key, value}}, pid) do
   :ets.insert(pid, {key, value})
   {:noreply, pid}
 end

 def handle_cast({:delete, key}, pid) do
   :ets.delete(pid, key)
   {:noreply, pid}
 end

 def handle_call({:find, key}, _from, pid) do
   result = :ets.lookup(pid, key)
   {:reply, result, pid}
 end
end

And this is where the Erlang Term Storage magic begins. In this example, I used the 3 simplest possible ETS functions:
• :ets.insert/2 - used for inserting data to the table, the first argument determines the table PID or name in case of named tables, the second one is a set of data - in our case key and value we just passed,
• :ets.delete/2 - used for deleting data from the table, the first argument is the same as in :ets.insert/2, the second one determines the key of the row that should be deleted,
• :ets.lookup/2 - used for searching for the whole row with a given key; the first argument is the same as in the other function, the second one determines the key of the row we are interested in.

It is just as simple as that. Of course there are some more complicated functions - and I encourage you to look at the ETS documentation for more information and possibilities. But for purposes of showing how simple it is to use ETS in your project these three will do the job.

The results are as follows:

iex(6)> MyApp.EtsService.insert_data("this_is_my_key", "this_is_my_value")
:ok
iex(7)> MyApp.EtsService.find_data("this_is_my_key")
[{"this_is_my_key", "this_is_my_value"}]
iex(8)> MyApp.EtsService.delete_data("this_is_my_key")
:ok
iex(9)> MyApp.EtsService.find_data("this_is_my_key")  
[]
iex(10)>

As you can see, our module for wrapping ETS functions using GenServer is working fine.

There are a few things worth mentioning. The first one is that it is good to remember that the :ets.lookup/2 function will always return the list of results. It is doing so because the table is not always :set type as in our example. There could be multiple values in the table under the same key. Even if there are no matches, as in the last find_data/1 call, it will return an empty list.

The second one - after restarting the application, the table will be empty. The data lives in the memory as long as the corresponding process is alive. Once the process dies - the data disappears.

ETS in Elixir: Wrapping up

Erlang Term Storage (ETS) is a really nice feature to have. Instead of relying on external solutions, you can really just focus on your work using this built-in term storage engine. The use cases are really wide - you can use it to build a token-cache store, user navigation history module, and many others.