ETags in Akka HTTP

Page content

I have recently been involved in implementing ETags and Last-Modified header support in one of the services based on Akka-http.

I have prepared a quite comprehensive example project that shows how to implement those capabilities in Akka-http based projects.

In this post I’ll describe in a practical manner what ETags are and how to support them in your own projects.

Side note: I’ll focus on ETags and have a section on Last-Modified header at the end.

Quick introduction to ETags

ETag is basically a additional HTTP header that’s returned by the server that can be treated like a checksum of the response. The client can later use this value when sending consecutive requests to the same endpoint to indicate what version of the resource it has seen before.

Based on the value of ETag provided by the client, the server can decide not to return the HTTP body, and indicate this by returning HTTP code 304 - Not Modified.

When client receives back a 304 response this means that the resource that client has received previously is still up-to-date and there is no need to send it again by the server.

Note that this approach requires the HTTP client (or library) to keep cached responses on it’s side, and in case of 304 response, the data should be read from there.

Wikipedia has a very good article on ETags

Note also that the server sets value of the ETag header, and clients should use If-None-Match

First request

curl -v http://localhost:8080/books-etags/1
...
> GET /books-etags/1 HTTP/1.1
> User-Agent: curl/7.35.0
> Host: localhost:8080
> Accept: */*
> 
< HTTP/1.1 200 OK
< ETag: W/"1e8ec132952ddfe628c6e2ff6a66d843"
< Last-Modified: Tue, 25 Oct 2016 12:45:00 GMT
* Server akka-http/10.0.0 is not blacklisted
< Server: akka-http/10.0.0
< Date: Fri, 25 Nov 2016 10:43:43 GMT
< Content-Type: application/json
< Content-Length: 197
...

And the body follows

Second Request

curl -v -H "If-None-Match: W/\"1e8ec132952ddfe628c6e2ff6a66d843\""  http://localhost:8080/books-etags/1
...
> GET /books-etags/1 HTTP/1.1
> User-Agent: curl/7.35.0
> Host: localhost:8080
> Accept: */*
> If-None-Match: W/"1e8ec132952ddfe628c6e2ff6a66d843"
> 
< HTTP/1.1 304 Not Modified
< ETag: W/"1e8ec132952ddfe628c6e2ff6a66d843"
< Last-Modified: Tue, 25 Oct 2016 12:45:00 GMT
* Server akka-http/10.0.0 is not blacklisted
< Server: akka-http/10.0.0
< Date: Fri, 25 Nov 2016 10:47:26 GMT
<

Second response doesn’t have any body

Third request

After the resource on the server was updated:

curl -v -H "If-None-Match: W/\"1e8ec132952ddfe628c6e2ff6a66d843\""  http://localhost:8080/books-etags/1
...
> GET /books-etags/1 HTTP/1.1
> User-Agent: curl/7.35.0
> Host: localhost:8080
> Accept: */*
> If-None-Match: W/"1e8ec132952ddfe628c6e2ff6a66d843"
> 
< HTTP/1.1 200 OK
< ETag: W/"3049afc59dbff41160acfcbb0f3273ec"
< Last-Modified: Tue, 25 Oct 2016 12:45:00 GMT
* Server akka-http/10.0.0 is not blacklisted
< Server: akka-http/10.0.0
< Date: Fri, 25 Nov 2016 10:55:29 GMT
< Content-Type: application/json
< Content-Length: 197
< 
...

And the new body follows

I encourage you to download the sample project I have prepared: https://github.com/wlk/akka-http-etag-example which will allow you to try out those commands yourself, I have also added more debug logging to see how your request flows through on the server side.

Implementing ETags support

Akka-http has already implemented a conditional directive that allows us to use ETags quite effectively

So what’s left for us is to properly include it in our routing and pass correct arguments.

In my sample project there is a class BooksApi

This is most important snippet - I have written comments inline:

// There are 3 cases to consider
path("books-etags" / IntNumber) { id =>
  optionalHeaderValueByName("If-None-Match") {
    case Some(_) =>
      // First case, we get request with some value of "If-None-Match" header, right now we are unable to say if it's valid ETag
      booksService.getBookLastUpdatedById(id) match {
        case Some(lastUpdated) =>
          // "conditional" directive receives the ETag extracted from the request, and compares it to "lightweightBookETag(lastUpdated)"
          // which was calculated only based on the date of the book - we didn't have to fetch full object from the DB
          conditional(lightweightBookETag(lastUpdated), lastUpdated) {
            // "conditional" directive will return 304 if ETag or "If-Modified-Since" were valid
            // in this case we don't need to fetch anything more from DB
            complete {
              // If ETag was invalid (for example outdated), we continue, this time fetching full object from DB
              // Fetching full object from memory is more time consuming
              booksService.findById(id) match {
                case Some(book) => book
                case None       => throw new RuntimeException("This shouldn't happen")
              }
            }
          }
        case None => complete {
          // If resource doesn't exist we don't set any headers
          HttpResponse(NotFound, entity = "Not found")
        }
      }
    case None =>
      // Second case, request doesn't contain "If-None-Match" header
      // we know that we have to return 200 with full body, so we do that (alternatively we return 404 if resource wasn't found)
      booksService.findById(id) match {
        case Some(book) =>
          conditional(bookETag(book), book.lastUpdated) {
            complete {
              book
            }
          }
        case None =>
          complete {
            HttpResponse(NotFound, entity = "Not found")
          }
      }
  }
}

A word about `Last-Modified` header

In some cases ETag and Last-Modified value could serve the same purposes, even in my project I calculate ETag based on lastUpdated date, because I know that each time a resource changes, it will also update lastUpdated date.

Last-Modified is sometimes simpler to understand, but it’s not as universal as ETags, here are some cases were it won’t work but ETags would:

Collections

When collection has multiple resources, we can calculate combined ETag as concatenation of all ETags of individual resources, and then hash them with MD5 (or any other hashing algorithm)

In case of Last-Modified we don’t have a way to do this, because if we look at Max(Last-Modified) for all elements, we won’t notice for example removal of elements from collection

Modification date is not available

There are resources which don’t have information about modification date, they can change any time as well. In those cases ETags are the only option

Summary

Support for ETags and Last-Modified header is quite easy to add for Akka-http projects.

I have shown how to add this to a single endpoint, the drawback is that it makes code much more verbose as there are 3 cases that require handling, each requires different path the request needs to go through.

This probably could be eliminated by defining more generic function that can encapsulate all the logic, but I decided to leave this out for now.