Observability of a reactive system: Introducing Akka Sensors

Runnables

Any reactive system, by definition, wants to be responsive, resilient and elastic, and therefore can’t be anything but message-driven. Any action within such system is an asynchronous message that needs to be processed. This takes a form of a Runnable: a chunk of data (message) embedded into a snippet of code, that needs to be executed to process the data.

Dispatchers

To process the stream of Runnables, at the heart of any reactive system, there is one or more execution contexts. An execution context, backed by some sort of worker pool, accepts the Runnable in its inbound queue. When one of the context workers becomes available, the Runnable is executed.

(image courtesy of Freepik)
  • worker threads are the cashiers
  • CPU cores are the cash registers
  • how much time a customer spent being served by cashier? (run time)
  • how many workers are actually occupied serving customers? (active workers)

Actors

In a reactive actor-based system, actors are stateful entities, forming transaction boundaries and guarding state integrity.

  • how often are actors passivated (receive timeout triggered), as opposed to terminated explicitly?
  • how much traffic (messages per second) does an actor (class) get?
  • how much time does an actor (class) spend on processing a message?
  • how many errors and unprocessed messages are there?
  • how much time a persistence actor (class) spends persisting an event?

Existing solutions for Akka

Akka itself is free open source software at its core (Apache license), but to use the great Cinnamon telemetry library, commercial Lightbend subscription is required, which may not be affordable for every team.

Akka Sensors

Akka Sensors is a new free open source (MIT) Scala library that instruments JVM and Akka explicitly, using native Prometheus collectors, with very low overhead, for high-load production use.

akka {
actor {
default-dispatcher {
type = "akka.sensors.dispatch.InstrumentedDispatcherConfigurator"
executor = "akka.sensors.dispatch.InstrumentedExecutor"

instrumented-executor {
delegate = "fork-join-executor"
measure-runs = true
watch-long-runs = true
watch-check-interval = 1s
watch-too-long-run = 3s
}
}
}
Actor dashboard
Dispatcher dashboard

References

Akka

Akka appeared in 2009. At its core is an implementation of actor model, as known from Erlang, rewritten in Scala. Actors are stateful entities, communicating with each other asynchronously, by passing messages around. Each actor is guaranteed to process just one message a time, allowing for lock-free mutable state updates.

Prometheus

Prometheus is free open source (Apache) time-series database that is widely used to keep process metrics.

Software Engineer — λ Reactive