How to test HTTP requests in Elixir with ExVCR

As your app grows so does its integrations base. At some point, you may need to consume API from third-party services. Today you'll learn how to test them properly with the exvcr library.

Demo app

To illustrate why testing 3-rd party services is a problem, let's create a simple wrapper within the sample Address Converter app that will return the latitude and longitude of a geographical object. We're going to consume https://nominatim.openstreetmap.org free API for this purpose.

For simplicity, let's assume that the Address Converter app is already created. To interact easily with external API let's use the HTTPoison and Jason libraries.

In mix.exs:

defp deps do
  ...
  {:jason, "~> 1.0"},
  {:httpoison, "~> 1.6"}
  ...
end

and after that:

mix deps.get

The next step is to implement Nominatim API wrapper. In lib/address_converter/nominatim.ex file, let's add few lines of code:

defmodule AddressConverter.Nominatim do
  @base_url "https://nominatim.openstreetmap.org/search?format=json&q="
  @headers [{"Content-Type", "application/json"}]

  def fetch_coordinates(query) do
    with %{ body: body } <- HTTPoison.get!(@base_url <> query, @headers),
         response <- Jason.decode!(body) do
      response
      |> Enum.map(&%{ lat: &1["lat"], lon: &1["lon"] })
      |> Enum.at(0)
    else
      _ -> %{}
    end
  end
end

As you can see we are fetching data from https://nominatim.openstreetmap.org with proper query and JSON Content-Type header to easily interact with a response later on.

Thanks to the Jason library we can decode the returned body and use Enum.map to map each result's latitude and longitude. As nominatim.openstreetmap.org lists all possible geographical places that match the given query, there might a lot of mapped coordinates. For simplicity, let's assume that we only need the first one, and that's exactly why we use Enum.at(0).

Quick demo of how it works:

iex> AddressConverter.Nominatim.fetch_coordinates("Poznan, Poland")
%{lat: "52.4082663", lon: "16.9335199"}

Testing external API problem

As our wrapper is ready to rock the world, we can proceed to the test phase.

In test/address_converter/nominatim_test.exs let's create the following test case scenario:

defmodule AddressConverter.NominatimTest do
  use ExUnit.Case, async: true

  alias AddressConverter.Nominatim

  describe "fetch_coordinates/1" do
    test "for given query it should return proper coordinates" do
      cords = %{lat: "52.4082663", lon: "16.9335199"}

      assert Nominatim.fetch_coordinates("Poznan, Poland") == cords
    end
  end
end

It's as easy as checking whether the coordinates match proper values. Let's check it out in action:

$ mix test

.

Finished in 0.4 seconds
1 test, 0 failures

Side note: If you want to make sure that our code is making a real HTTP request, turn off the internet connection and run this test once again.

Ok, our test works as intended. So what's the problem?

Well, with this solution, each time we run this test it's executing a real HTTP request to fetch the data. It's not a good idea, and there are at least five reasons why:

• request/response cycle might take some time, you don't want to slow down your tests,
• it might not be a free API, you may pay for requests that are being executed during tests,
• some APIs have rate limits,
• you need an internet connection all the time even if you didn't change test and code,
• response for a given request in most cases should always be the same (we can mock it).

Meet exvcr library

In this case, the solution to our problem is to record the response in a file. We don't have to do it manually, there is a lib for that, and it's called exvcr.

Let's add this lib to deps:

defp deps do
  ...
  {:exvcr, "~> 0.11", only: :test}
  ...
end

As you can see we're only adding it to test dependencies, as most likely you'll not gonna use it in the dev/prod environment.

Let's fetch our new dependency:

mix deps.get

If you take a look at the documentation, you'll notice that currently exvcr works well with three HTTP clients:

• hackney,
• httpc,
• ibrowse.

It means that if you're trying to test a code that makes an HTTP request using one of these clients under the hood then you're good to go. In our case, we're using HTTPoison which is built on top of Hackney.

Here is an updated test code that takes leverage of exvcr:

defmodule AddressConverter.NominatimTest do
  use ExUnit.Case, async: true
  use ExVCR.Mock, adapter: ExVCR.Adapter.Hackney

  alias AddressConverter.Nominatim

  setup do
    ExVCR.Config.cassette_library_dir("fixture/vcr_cassettes")
    :ok
  end

  describe "fetch_coordinates/1" do
    test "for given query it should return proper coordinates" do
      use_cassette "nominatim" do
        response = %{lat: "52.4082663", lon: "16.9335199"}

        assert Nominatim.fetch_coordinates("Poznan, Poland") == response
      end
    end
  end
end

Let's break it down.

First, we use the ExVCR.Mock module with a proper adapter. As already mentioned, HTTPoison uses Hackney under the hood:

use ExVCR.Mock, adapter: ExVCR.Adapter.Hackney

Next, we need to configure ExVCR cassettes path:

setup do
  ExVCR.Config.cassette_library_dir("fixture/vcr_cassettes")
  :ok
end

Once we run the test, it'll create a file with recorded HTTP request/response data and save it in the fixture/vcr_cassettes folder. Can we choose the name for this file? We can, and that's exactly what happens here:

use_cassette "nominatim" do
  ...
end

use_cassette wraps the block of code to ensure that all HTTP requests and responses will be saved into the nominatim file cassette.

Ok, since we're now familiar with ExVCR and how it can be used within a test case scenario let's run our test:

$ mix test

.

Finished in 0.7 seconds
1 test, 0 failures

It works! As you should notice, there is a new file in your repo - fixture/vcr_cassettes/nominatim.json:

[
  {
    "request": {
      "body": "",
      "headers": {
        "Content-Type": "application/json"
      },
      "method": "get",
      "options": [],
      "request_body": "",
      "url": "https://nominatim.openstreetmap.org/search?format=json&q=Poznan, Poland"
    },
    "response": {
      "binary": false,
      "body": ..., // The body is a bit long to paste it here
      "headers": {
        "Server": "nginx",
        "Date": "Thu, 29 Oct 2020 17:51:44 GMT",
        "Content-Type": "application/json; charset=UTF-8",
        "Transfer-Encoding": "chunked",
        "Connection": "keep-alive",
        "Keep-Alive": "timeout=20",
        "Access-Control-Allow-Origin": "*",
        "Access-Control-Allow-Methods": "OPTIONS,GET"
      },
      "status_code": 200,
      "type": "ok"
    }
  }
]

In the end, ExVCR runs a real HTTP request to save request and response data, and then when you run the test again, it's trying to match the pattern of request to deliver response without performing HTTP request again. If it finds a match, it returns a saved response. If there is no match, again, a real HTTP request will be performed to save data into a cassette file.

To make sure that our test doesn't depend on internet connection anymore (so there is no real HTTP request), let's turn it off and run our test once again:

$ mix test

.

Finished in 0.7 seconds
1 test, 0 failures

Great! This is how you can improve tests that depend on external services with the ExVCR cassette mechanism. There is more configuration stuff in this library, as well as little details that you may want to use. Makes sure to check the documentation.

FAQ

What is ExVCR and how does it help in testing Elixir applications?

ExVCR is a library for Elixir that records and replays HTTP requests, making it easier to test applications by mocking external API calls.

How does ExVCR work with Elixir's HTTP clients?

ExVCR supports HTTP clients like Hackney, HTTPc, and ibrowse, allowing seamless integration with Elixir's HTTP request libraries, such as HTTPoison.

How can developers use ExVCR in their Elixir projects?

Developers can add ExVCR to their project dependencies and use it to mock HTTP requests in their tests, thus avoiding real API calls.

What are the benefits of using ExVCR in Elixir tests?

Benefits include faster tests, reduced external dependencies, and avoiding issues with API rate limits and internet connectivity.

What steps are involved in setting up ExVCR for Elixir tests?

Setting up involves adding ExVCR to the project's dependencies, configuring cassettes, and wrapping HTTP requests in test cases with ExVCR cassettes.

Can ExVCR be used with any Elixir application?

ExVCR is best used in applications that make external HTTP requests and need repeatable, consistent test results without real network calls.

Are there any limitations when using ExVCR in Elixir applications?

While ExVCR simplifies testing by mocking API calls, it may not cover all edge cases or simulate real-world network issues.