Elixir Programming Language - Interview
Michał Buszkiewicz, CTO at Curiosum, was a guest in Podcast POIT ("Porozmawiajmy o IT"/"Let's talk about IT") conducted by Krzysztof Kempiński. The interview took place in Polish, but the content discussed during the meeting was extremely interesting, so we decided to make English transcription for you.
We defy the myth that functional languages are more difficult than object-oriented. Read and check if Elixir is a good language for starting in programming for your project!
Hi, Michał! It’s a pleasure to have you in this podcast.
Hello! It’s great that we’re having this meeting, I’m very glad.
I was just thinking that I didn't actually add to this introduction, the title of a speaker at conferences, being a guest in podcasts - from what I can see, it is also a significant part of your business.
Yes. It was a very nice experience to show up at ElixirConf EU in Warsaw recently. We talked a bit about debugging the application. In general, the presentation showed a bit more unconventional approaches to the subject, which, as you mentioned, I like very much. The response was very positive, even among people who somewhere did not agree with these approaches, so I am very glad that we made it and that we had the opportunity to meet all together after two years.
Yes, exactly, now everyone is longing for this type of meetings.
This episode is also unique to me in that the Elixir we are going to talk about today, is a technology, or let's say this branch of technology that I have been working in for about three years, so it will be even more pleasant to talk to someone who also finds oneself in this stack.
Okay, let's get down to business. At the beginning, there is a fixed point on the agenda, i.e. a question for the guest. Michał, do you listen to podcasts? If so, do you have any favorite audios you can share with us?
I have to admit that I am the type of person who rather feels comfortable as a reader. Sometimes I listen to typically Elixir-oriented podcasts, such as Thinking Elixir, in which I also took part in the past, Elixir Mix or podcasts that look a bit more honest about programming, Running in Production, for example.
However, when it comes to reading resources, which are somewhere more in my style when it comes to finding content, I really like everything that top newsletters send me, such as ElixirWeekly. Also when it comes to such sources of knowledge on an ongoing basis, these are the things that reach me the most often.
Sure, I get it. It does not surprise me at all, because I have such podcast listeners who’s only way to experience the podcast is reading transcripts, so I perfectly understand that everyone has their preferences, some prefer to listen, some to watch, and others to read.
You know what, I think it's also about having some kind of ability to stay focused on one thread for a long time. I have to admit that this is something that I also struggle with sometimes. And the reception of the read content in such a form that somewhere I read a fragment of an article, then read another fragment, then another one, often seems to be easier for my way of mind functioning.
But, of course, I also appreciate podcasts, so I'm not closed to them, definitely.
Sure, that’s great.
Why we chose Elixir
As I said in the introduction, you are the co-founder of the software house Curiosum, and you focus on Elixir, it is the leading technology in your business. But you do not limit yourself to the projects that you implement, you are also active in the community, you run a blog, as you said, you have recently been a speaker at ElixirConf EU.
I wonder where this choice comes from? Whatever the case, Elixir is not the most popular technology among software houses, and for sure not in Poland.
It has never been the most important fact for us, but after some time after setting up the company, we have realized that we are probably the first software house in Poznań and I am not sure, but probably also in Poland, which from the very beginning focused hard on Elixir without switching to this technology from any other. This is quite important, because thanks to this we can say that we are specialists in this field, we do not focus on small topics or invest our time in technological piles in which we do not fully believe.
On the other hand, having such experience drawn from our previous professional work in creating web applications for many industries, we know perfectly well what limitations were brought by the technologies that we have used so far in terms of scalability or the ability to handle simultaneous connections reaching the number of hundreds of thousands, or in some cases, millions. We also just realized that the languages traditionally associated with the joy of developers' work and the simplicity of entering and creating applications with their help are simply not adapted to parallel and fast data processing.
All this later generates infrastructure scaling costs that investors and startup founders have no idea about in the initial phase of their product development. We decided to face it with a technology that is both productive in the application itself and ready for the challenges faced by a startup, when this application leaves the MVP phase and becomes much more.
Sure, got it. So it was a conscious choice, defining the technology problems that you had previously dealt with and a strong bet on Elixir.
OK. Now I would ask you to tell a little about the language itself. Where did it come from, who made it? Because the language itself it's not very old in terms of years on the market, but it is based on solutions that already have been in use. Please tell me what is this language is and how was it created.
The main creator of this language and, in fact, the initiator of the whole idea is Brazilian José Valim, associated for a long time with the ecosystem, the environment of the Ruby language. He has developed an interest in languages, tools more suited to the performance and scalability demands of today's applications.
He was particularly passionate about the Erlang virtual machine, which has been widely used in telecommunications to solve such problems since the 1980s, when it was created by Ericsson.
It turns out that these kinds of problems have largely returned, and it has been realized that this is actually an issue that is very relevant.
Elixir is a kind of superstructure above this VM, which is responsible for José's lack of Erlang in terms of syntax or convenience of use. It also draws inspiration from other functional languages, i.e. Haskell, Clojure, and syntactically from Ruby. From the first of them, among others, he took the construction of pattern matching, the second one gave rise to some concepts related to the organization of the code and various syntactic sugar, and the third is actually a syntactic stylistics.
However, comparing it to natural languages, I would compare Ruby and Elixir to languages that have, let’s say, similar spelling but completely different grammar, as it is common to sell Elixir as similar to Ruby, which is quite, I would say superficial, if not nonsense.
Yes, it may seem so at first glance, but underneath, completely different mechanisms.
Yes, it also results from the fact that the community overlaps a bit here. The creators themselves were also associated with the Ruby language and somewhere there contributed quite strongly to the development of its ecosystem. There are a lot of coincidences here and very few any actual connections.
Yes, I think they are rather brothers, if you can say that about technologies, but in fact they are quite different solutions. As you said, José Valim, as Ruby on Rails' core member at the time, was looking for a solution to the concurrent computing problem, he looked at it in other languages and found Erlang like you said.
As he himself said later, he loved everything that he found in this Erlang, in this virtual machine, but he also hated what was missing there, i.e. the simplicity of code creation, which in Ruby, let's say, is something that de facto defines the entire language. So he decided to use his experience with Ruby a bit, to create a new language which would run in a virtual machine, allowing to run even an Erlang code here.
OKAY. I would like to ask you, what demands does Elixir meet? Plenty of similar languages become unpublished, some die as quickly as they are created, but Elixir has somehow beaten and continues to stand out. Therefore, it should be suspected that it actually meets the need. Could you mention something like that when it comes to Elixir?
This is a kind of sweet spot between efficiency, ease of coding and the quality of work results, it can be summed up in the shortest possible time. I think there are many reasons why we chose Elixir, and not any other, functionally purist language, such as Haskell, that is on some way to the mainstream, one could say.
I believe that Elixir is responding to the age-old problem where startups initially choosing technology known for rapid application development, developer’s joy, time to deliver, eventually do not have much to offer in terms of further scaling the applications. Of course, the use of the appropriate technology, i.e. Elixir here, is not a guarantee of anything, because there are a lot of projects that are badly carried out and use it in a not fully conscious way, but the characteristics of the language and this toolset mean that it gives more possibilities and leads to better solutions.
Okay, now let's go a little bit into how this language is built. What is the structure of this language, what is the architecture of the runtime? Please shed some light on that.
Let's start with the tools shared with the Erlang language, or BEAM - a virtual machine, which in turn is a part of the OTP, or Open Telecom Platform, which is simply a set of standard tools library for the Erlang language. And this is the basis that ensures scalability and fault tolerance, which is another important feature of both Erlang and Elixir. And for this virtual machine both Erlang and Elixir applications are compiled.
The features of this virtual machine are definitely more important than what is sometimes mentioned as important in the case of Java virtual machines, i.e. some cross-platform etc. Here it is not so promising when it comes to e.g. running code compiled in one platform on another platform, but it's not the most important aspect and that's not what we focus on.
On the other hand, an important feature of the language itself is that the tasks of its own compiler are relatively sparse, given the fact that much of the key language structure is built using macros, i.e. functions that take certain data and return code. For example, the word that makes up the function: "def", is also defined as a macro, in the source code of the language itself you can see exactly what happens when defining the function, how it is registered, etc. whose subsequent macros, i.e. our code-generating functions, are expanded to a form in which there is nothing to develop. There are only elements composed of something that we see in a Kernel Special Forms module, creations handled directly by the compiler. And this character, in turn, is already compiled to bytecode.
This is, in a nutshell, the way the Elixir looks and you can say that it is quite an important feature, because it strongly determines the way of creating bookcases, some DSLs, where it is needed. And of course the whole metaprogramming aspect, which is also very important.
Exactly, it’s an interesting fact that quite a lot of Elixir is actually written in Elixir.
This is a fun thing. What you said, that this bytecode, Elixir compiled into a bytecode, later run on a virtual machine, is like standing on the shoulders of giants, we rely on what for decades, when it comes to Erlang's virtual machine, was developed and we are adding something new to it, which allows you to write software in a more pleasant, easier, more accessible way. I think an analogy that might be closer to the audience here is that it can be compared a bit to Scala and the Java virtual machine. It’s a similar concept at least.
As you said Elixir is based on a functional paradigm, it is like the core paradigm of this language. Whatever the case, this paradigm is gaining some popularity now, but it's still a niche. Object-oriented programming definitely reigns supreme when it comes to modern programming languages.
I believe that, unfortunately, it’s somewhat influenced by the fact that, in the common opinion, when you talk to someone, functional programming is perceived as very complicated, mathematical, even scientific, marked, full of such a theory - this is the reception. Do you deal with something like this when programming in Elixir? Is this what functional programming looks like?
Elixir is rather far from the kind of features that are actually more ascribed to more purist languages, such as Haskell, Elm, maybe F Sharp. Certain concepts existing in these languages were simply not transferred to Elixir on purpose. These languages are often statically typed, when we have dynamic typing in Elixir, and it is really a challenge for programmers who would use Elixir, and who have used Haskell so far. In Elixir, we have a « let it crash » approach, in contrast to such strict error catching already during compilation, e.g. in Haskell. So, die-hard fans of functional languages, quite purist, with a large dose of theorism, will rather not like Elixir.
I treat functional programming here as such a paradigm that simply emphasizes some kind of data processing from one form to another. I think this is the aspect of functional programming that is the most important in Elixir and very easy to use even by a pipe operator.
Coming back to all other languages - adored by all people who are very passionate about this mathematical aspect of functional programming, for example Haskell, it will probably remain a great language, but chosen in specific projects, to achieve a specific goal, consciously. Elixir, on the other hand, has a future that covers a wider range of applications.
Some kind of simplicity and advantages of its virtual machine in comparison with object-oriented languages, to which, you can say, some elements of declarative or functional programming, are of great importance - basically all languages known to us today have such elements. By far the biggest advantage here is the mutability of the data and the complete lack of objectivity - as strange as it sounds to someone used to OOP - it makes it very easy to reason about interactions between modules and, for example, to transfer code in both local and distributed environments.
Elixir frameworks and libraries
Yes. I think it's also practical and pragmatic. However, Erlang as a language used in telecommunications must have been very pragmatic, and here I have the impression that all the concepts of functional programming that exist in Erlang and in Elixir are also intended to serve this purpose. Here we do not want to theorize, but actually build something that will simply work in production.
And while we are talking about production and these applications, I would like to ask you, what most widely used frameworks, libraries, and precisely Elixir applications, are most significant and noteworthy?
It cannot be denied that if we talk about Elixir applications somewhere on the web, we usually think about the Phoenix Framework or one of its components. It is a framework that can be compared to Ruby on Rails, but it is definitely more modularly constructed and it is much easier to disconnect its individual elements somewhere, if necessary, give up some of them, etc. Besides, the creators of this framework more than once openly said that they used the experience of frameworks such as Ruby on Rails and that certain design solutions used in them were simply wrong, and they learned from their own or someone else's mistakes.
Quite an interesting element, not yet in the 1x version, is the Phoenix LiveView, which is somewhat reminiscent of React, although it is a comparison that should actually be covered with many stars. It is in fact the use of the Erlang virtual machine's ability to process hundreds of thousands of processes at once, to be transferred, to manage the state of the web application interface, in fact to our server, which maintains a connection with each user server for each user who, for example, displays a given page. In fact, a separate process that maintains the state and sends it to the client, i.e. the browser, via WebSockets, which is also a strong point of Phoenix.
Then, of course, we also have frameworks that are often used together with it, eg Absinthe for API construction in GraphQL, which is also a very cool thing. Of course, not to mention all the other libraries that are necessary for most apps, such as Ecto for databases, which is an interesting approach to mapping tables to structures in our language, because, of course, we are not dealing with an object-oriented language, so it cannot be an object relational mapping. Ecto tries to be as close to purely SQL concepts as possible. And this is a slightly different approach than many other ORM frameworks.
There are many libraries that allow you to interact with popular cloud services, for example with AWS, so you don't have to, break open doors or reinvent the wheel. There are many different HTTP clients that allow us to interact with some API that we need to consume in a more structured way. Generally, all libraries of the most popular issues that we deal with in web applications are available.
And, departing a bit from web, we also have Nerves, of course, a framework for building IoT applications, mainly at the level of devices such as Rasberry Pi, which is also a very cool subject. When it comes to completely different applications, sometimes I look for some completely different things and it turns out that there is, for example, a framework for two-dimensional user interfaces like Scenic. Or a framework for building some visually nice dashboards using also React like Kitto.
So there are many applications here. Many libraries that were created, for example, a few years ago and are not developed until today, work without any problems due to the fact that Elixir is a language that strongly focuses on such harmonious development. At the moment, it's a closed language when it comes to features from the original backlog. They have all been implemented, and only sometimes there are some deprecations that will really be deleted only when Elixir 2.0 is released. In general, the development of the language is rather difficult to expect that somewhere from version 1.12 to 1.13 there will be any breaking changes, meanwhile it’s just their policy. And that's cool because there are fewer surprises.
But apart from that, such an industry as IoT or embedded devices, there are also applications that you may not hear about at first glance, but which really find their place in certain places. For example, such an application in the case of navigation on boats, where the boat goes out to sea for months, so the navigation must not fail there, it must work continuously and in a predictable manner.
It turns out that there is a company that produces such advanced solutions and some time ago boasted about graphs of e.g. memory consumption, CPU consumption, etc. This is a really predictable technology, in the sense that the virtual machine gets up, the application is launched, the consumption of the resource is at some level and it stays at the same level for months, there are no fluctuations here.
In addition, on such, maybe not toys, but such boards as, for example, Rasberry, Arduino, etc. - there you can also run such a very, very slimmed-down version of a virtual machine with Elixir, it all boots up very quickly. So there are also such applications.
I mentioned this reliability a bit, you mentioned Michał about the fact that we can run such code in a concurrent and distributed manner. In your opinion, are these features called reliability, concurrency, some scattering of this code - are these the main features that sell Elixir?
Yeah, here I think any systems that require a high level of concurrency or that may require a high level of concurrency in the future, are a good place to use this technology. For example, Elixir incl. with Erlang's virtual machine, allowed Discord to handle 5 million parallel client connections. Later, it was improved a bit with the use of Rust, but it is still a result that gets respect.
And as I said, we often deal here with the let it crash approach, made possible by very efficient management of processes and their resources by the VM, which cleans them very efficiently when a process crashes in a controlled manner. And, of course, this philosophy is not that we want the users to see all the errors - these are the kind of errors that we expect at the server level and we deal with at the design stage in relation to processes.
I think the very cool thing is that reasoning about concurrency processing is very simple due to the fact that we have an actor model and interprocess communication via message passing, not data sharing. And as I mentioned, the lack of stateful objects known from OOP definitely facilitates the implementation of all issues related to transmitting code between nodes in order to run some functions remotely.
Development of Elixir technology
Exactly, José Valim is the creator of this language, as far as I remember, it was 2012, when some first versions of Elixir appeared at all. Who is currently behind the development of this technology? How is it currently taking shape?
Yes, it is currently the 6-person Elixir Core Team of which José is also a member. On the other hand, the language is, of course, supported by a number of companies that contribute to and sponsor major libraries and the most widely adapted frameworks, such as Phoenix and Nerves. Of course, the Core Team makes sure that, on the one hand, the voice of the community is heard in the best possible way when it comes to functionality and directions of development ... José himself stated during the last ElixirConf that if it was up to him, some elements of the language would look completely different, but the deciding factor was the society.
On the other hand, it is glorious that this Core Team makes sure that the language does not get too much functions that are not universally needed by the entire community to be built into its core. There is such a strong pressure that these kinds of things become separate libraries and as if they compete with each other in something like a free market.
Elixir popularity and use
Okay, tell me from your perspective, the popularity of this language how does it look like in the industry today? Of course, we can see the TIOBE index or other such indices, but I mean here, for example, the supply of developers, which may be a significant factor, for example, for companies when it comes to deciding to go for this technology. And also companies that use Elixir in their applications for production.
Starting from the end - of course, there are more and more companies using Elixir in production and they are really good players, for example Pinterest, which was able to halve the number of servers they need, 10 times less code in certain areas of the application. Moz, which greatly reduced disk usage and accelerated its API. Lonely Planet, Financial Times, Toyota, Bleacher Report, Discord …
PepsiCo, exactly. WhatsApp, which, although it is rather Erlang, is obviously based on the same infrastructure, so you can connect it here. So there are more and more of these companies, so I think this is the best proof that this language really has a lot to offer. However, when it comes to a typical startup environment, while a slogan such as Ruby on Rails is a kind of trigger, an obviousness that is immediately associated with certain advantages known for years, Elixir or Phoenix are still something that you need to sell in a more active way.
However, I would be reluctant to say that this is a niche technology. Perhaps I would say less obvious when it comes to choosing it for specific purposes. However, as I said - more and more really big players associated with the web and not only boast about using Elixir, which, however, proves its breakthrough strength.
However, returning to the labor market - it cannot be concealed that it is difficult. Most developers switch to Elixir from other languages, at a level like, say at least Regular. The supply of such people is relatively small, as talented developers find employment very quickly. It can be said that the supply does not keep up with the demand, which is definitely visible both in software houses that use this language and startups that operate on their own. Sometimes this creates a slightly unhealthy race, the result of which is not successful recruitment, both in such more product companies and software houses, so you have to take it into account.
On the other hand, another strategy that we also successfully use at Curiosum is recruiting young talented programmers who, ugly speaking, are not contaminated with object-oriented programming, and we train them internally using our proprietary methods and successfully entrust them with the tasks of developing commercial projects under the supervision of more experienced colleagues .
And these Juniors really learn quickly to build neat, orderly and efficient application architectures. It turns out that in fact the learning curve of this language is such that you don't need a developer with some kind of vast experience to be effective. Of course, we want every programmer who joins the projects on our behalf to meet the standards we set, so that the CV of a specific developer who joins a specific project is less important than the mere fact of cooperation with us and access to our knowledge base, our know-how.
Starting programming in Elixir
Of course. Just when we are learning this language, I would like to ask you how to start, where to look for materials, what to use first?
There are many books on PragProg, incl. Learn Functional Programming with Elixir, which connects Elixir itself very nicely with functional programming in general, and it is a very nice book. In the later stages, it is also nice to read books that talk about some more specific topics in this language. For example, Chris McCord's book about metaprogramming in Elixir, which is really something completely different from the concepts that are used for this purpose in other languages, this has given me a lot.
As for the other resources, I personally learned the language from the materials available on his website, in a very basic extent and although they are not very extensive, I was a programmer of other languages at that time, who felt quite comfortable with new things, it was enough to know the basics.
There is also the Elixir School, which is a course also available for free online, which deals with it a bit more. Sometimes we also recommend videos on Udemy, some courses that cover Elixir-related issues quite broadly. Generally, there are a lot of these materials. Maybe not as much as in the case of some traditional languages like Java etc., but in general I think that they are of much higher quality and there is definitely something to learn from here.
Sure. Perhaps I would add Exercism.io to that - it is a system for practicing your programming skills, of course in many languages, incl. in Elixir, where individual tasks of increasing difficulty are solved. We can of course check our solution locally, but also send it to this platform and count on the comment there, to check our solution in such a friendly environment. This is also a fun way to either start learning or train yourself by doing daily cuts, daily tests.
You mentioned that, for example, you recruit or accept people you train for Elixir. I have more experience with people who migrate to Elixir in the sense that they already have some experience, sometimes even quite extensive, in programming in other languages, usually object oriented. In your opinion, is such a transition - when someone actually has a workshop, but is so object-oriented and wants to switch to Elixir - easy, difficult, or does it even matter that someone already has a large object-oriented baggage of experience?
Of course, it makes a big difference. And it all depends on whether this transition is made by a talented programmer, a solid developer who understands the difference of paradigms and is ready not to mix them consciously, or whether it is a coder that does it out of necessity, then he’s dropped into the deep end, knowing before that PHP or some other language. We have seen many examples of functional language programming by people who completely did not understand this paradigm. So a lot depends on the skills of a particular person.
On the other hand, I mentioned an often repeated opinion about Elixir's similarity to Ruby, now talking a bit about people who make this kind of transition, just like me. You have to be careful that a person who is learning Elixir or who wants to potentially do it, does not do it a little bit of harm, because it is a bit like saying that Turkish is similar to Polish because they have similar alphabets.
Here at Elixir, of course, we have a complete non-objectivity and mutability of data, and the next most important difference is the approach to metaprogramming because both languages are very flexible but in completely different ways. In Ruby, basically everything can be patched literally at any time in run time, classes and methods can be defined as you like.
In Elixir, monkey patching is practically impossible by design, and the magic code generation takes place during compilation by this macro mechanism that I mentioned. Basically, once an app is compiled, its modules are closed to all Ruby add-ons that you can add whenever you want. This is a very important difference at the stage where, for example, as Ruby's Senior Developer, you are an ace in creating DSLs using the Missing Define Method and other tricks, and suddenly you are standing in front of a language like Elixir, in which, to embrace meta-programming, you need to understand how AST, Parser and other macro issues work. And this is the element that has probably caused more problems than the mere fact of moving away from the object-oriented paradigm to the functional paradigm.
Yes, meaning this awareness of what you are doing is key here.
Deficiencies in Elixir
OK, now maybe a bit of an uncomfortable question, but I want to ask you, what are the problems, gaps in your opinion when it comes to this language, this ecosystem?
The problems we face when it comes to the language itself, because I will no longer talk here about, let's say, completely extra-linguistic issues, sometimes it is poor documentation of some widely used libraries. Perhaps their creators sometimes lack the enthusiasm to maintain it. Sometimes the documentation actually ends with READ ME, and that's all you have to deal with. Sometimes there is indeed a lack of enthusiasm for the development of the libraries themselves. For example, the Arc library that a lot of people use for uploading files, correct me if I'm wrong, but it's still not maintained by its original creators.
Other shortcomings? Recently I was at ElixirConf with a presentation on debugging and I am generally looking for such a golden mean when it comes to tools for analyzing the code for errors that occurred in it. So there are some shortcomings. I will not even mention the situations when I see available libraries consuming some API for other languages and their lack to Elixir, because it is really a less important thing if the API we are talking about is well documented and you can simply make a good HTTP client for this purpose, so that it really is something less important.
In general, the topic of compilation times also comes up sometimes, it's something the Elixir team has been working very hard on for at least a few versions. There is a real improvement over what was e.g. in Elixir 1.8. Mainly it is related to some details regarding the tracking of dependencies between modules and what kind of dependencies should result in automatic recompilation of a given module, etc. However, here I can see such continuous and significant progress that these shortcomings will be eliminated at some point and this compile time will obviously improve. It’s really good here when it comes to answering these kinds of problems.
Elixir as the first programming language
Sure, got it. I would like to refer to what I said - is such a common belief that functional languages are more difficult than object-oriented, and therefore it may seem that starting your programming adventure with functional languages may not be a good idea. That you have to get a bit of experience and knowledge at the beginning and then possibly make such a transition. We have already said that this is obviously not true, but I would like to ask you if you think Elixir is a good language to start programming?
I will say this - I would like it to be so, but whether it is so, it basically depends on the characteristics of such intellectual potential of a person who learns programming, because there are people with very different predispositions. If the people we're talking about are typically academics, then functional programming has an interesting future for such people. I would really like to meet more and more people on my way who are untouched by such habits related to object-oriented programming.
On the other hand, there are still a lot of people who would rather start by learning to program in an object-oriented language, because as I said, I understand functional programming - at least as seen in Elixir - as very focused on such processes. Processing data from one state to another, and it's not necessarily the line of reasoning that will be the easiest for anyone to begin with.
For many people, the object-oriented paradigm will be so close to the way of defining the phenomena around us that it will simply be a paradigm in which these people will be more effective. And it's great, they will also have their place, I also support such people. I think programming is for almost everyone. Yes, it is largely dependent on what a person has predispositions to.
It is so, attitude and predisposition, in fact.
Michał Buszkiewicz was my guest, we talked about the Elixir language.
Michał, thank you very much for the interview!
Thank you too!