How Phoenix LiveView helps Spotify's artist ad platform
Learn how Spotify uses Elixir and leverages Phoenix LiveView for its internal web application.
Table of Content
- Elixir development at Spotify: A background
- Elixir at Spotify: a need as a mother of invention
- Phoenix LiveView: from proof of concept to a game-changer
- Building a case for Elixir at Spotify
- Elixir and Phoenix LiveView at Spotify: A summary
Elixir development at Spotify: A background
After big companies like Bleacher Report, Moz and Discord admitted using Elixir in production, and PepsiCo being the next giant to follow their footsteps, the world’s leading streaming platform Spotify has joined the list.
Founded in 2006 and launched 2 years later, Spotify enjoys constantly growing traffic, having handled 381 million users in the third quarter of 2021. The platform offers a suite of services for music & podcast fans (songs, full releases, podcasts & playlists), as well as for the artists and content creators (advertising, metrics, etc.). The platform’s ever-rising traffic requires the ability to handle thousands of concurrent requests every second. This, in turn, calls for a robust tech stack.
Elixir at Spotify: a need as a mother of invention
What did make the devs at Spotify bet on Elixir? As it often happens, the decision to give the functional programming language a try came with a last-minute need.
In 2021, Spotify was nearing the release of a major product: an ad platform for artists letting them promote their work. In the last week before the launch, the devs realized they needed an internal application managing the service.
The business need
“We needed an internal debugging portal”, said Spotify’s staff engineer Joel Kemp on the Thinking Elixir podcast. The portal’s task was to make the team see what was being booked by the customers, and troubleshoot when needed.
The main challenge Kemp was facing, was that in periods right before the launch teams are reluctant to give love to its internal tools. Most of their attention is spent on fixing bugs, end-to-end testing, last-minute checks, etc. But the internal app seemed critical for measuring the product’s performance and, what seemed logical, success.
The standard procedure
Up till that moment, the standard approach to this problem would have been building a single-page app in React and a JSON API backend. Such operation would demand involving two people: both a frontend and a backend developer.
But given the fact that the portal’s audience would be the Spotify devs themselves, there was no need for things like a fancy UI to please anybody else. Designing it to access the crucial-yet-basic insights into the app’s performance would be a waste of time. And - let us remember - there was virtually no time.
The whole operation, done in a standard way, seemed daunting. Kemp was in dire need of an alternative.
Phoenix LiveView: from proof of concept to a game-changer
Several months earlier, Spotify hired a new team member who was well-versed in Elixir. Back then, he hinted at the language and its advantages as an alternative to Kemp’s struggles with a JVM-built backend service handling heavy traffic, i.e. thousands of requests per second.
After a few days of research, Kemp concluded that both Elixir programming language and the BEAM virtual machine it was based on, could indeed address his pain points, e.g. indicating the service’s bottlenecks at the double workload, without a need for heavy instrumentation. The tech stack also allowed for increasing the system’s throughput right away.
Proof of concept
For Spotify’s debugging portal, Kemp decided to use yet another Elixir-based weapon: the Phoenix LiveView library. Phoenix LiveView enables real-time user experiences with server-rendered HTML.
Among its greatest pros, the staff engineer lists:
- no need for JavaScript,
- SPA feel based on bidirectional web socket communication,
- all the package of Elixir’s advantages, e.g. concurrency, and fault-tolerance.
“We ended up choosing to try Phoenix LiveView as a proof-of-concept”, wrote Joel Kemp in his blog.
Success was defined as:
- getting the internal service into production quickly on Spotify’s infrastructure,
- not having to write lots of frontend code,
- and getting contributions from others (particularly folks who had no Elixir experience).
Kemp decided to build the service alone.
How did it go?
The first feature was ready locally in 1 day.
It took another 2 days to get the service into production.
With new features implemented within 1-2 minutes, and tests written within another 5-10 minutes, Spotify’s debugging portal became a success. After 2 weeks, the tool was used daily. Today it helps both technical and non-technical employees in their work.
Elixir also turned out to be approachable for newcomers. Devs with no prior experience in functional programming appreciated its fast learning curve and were able to contribute to the project shortly afterward.
Building a case for Elixir at Spotify
Before building the platform’s internal portal with LiveView, Kemp had to do some grassroots work.
To convince people at Spotify to Elixir, Kemp wrote a 27-page-long paper explaining the benefits of BEAM. Instead of a surface-level argument, he wanted to offer them a real deal, not just jump onto a bandwagon of “the next big thing” in web development.
“It’s not enough to just compare it to the status quo”, he says on Thinking Elixir. “You also have to say what about Scala, Closure, Go and Rust, and argue why those technologies are not a good fit. All this was leading up to the Phoenix LiveView, and how much we needed a web app with no time and no people.”
He also underlines the importance of building a community to support your case, as well as of being ready to put one’s professional reputation at stake.
“It is very easy to reject a proposal with just one or two people pitching it”, he goes. “The thing is very different with a swarm of people that are advocating for it”.
Calling for a certain technology, in Kemp’s view, must not be based on technological merits solely: “Programming is socio-technical, and you have to take people into consideration, as well as what standards does it have to enforce, where is the platform going, etc”.
Elixir and Phoenix LiveView at Spotify: A summary
Spotify’s tech stack varies depending on the kind of service. The listener-facing backend services are done with Java. Pipelines - with Scala. Machine learning tasks are handled by, you guessed it, Python and the web apps, and SPA rely on React.
Elixir/Phoenix joined these technologies in 2021 when the streaming platform staff needed an internal troubleshooting tool for its artists’ advertising service. Spotify’s staff engineer Joel Kemp chose Phoenix LiveView to build the tool with, being under time pressure and facing workforce shortages.
What started as a proof of concept proved that “with minimum resources you can build the tool in less than half of the time it would have taken with 2 devs working with React & JSON API”, and gained viral popularity among the Spotify team, including both technical and non-technical employees. In short, that’s how Elixir/Phoenix became a game-changer.
The story of Kemp and Spotify shows that both technical merits and a supportive community were key in implementing Elixir Phoenix at the company.
The engineer calls introducing it “a marathon” and claims opting for a new technology like Elixir takes grit to face rejection on a weekly or daily basis.
