Khepri 0.3.0

[dumbbell]

At this point, Khepri should be considered Alpha and not production ready. The API will likely evolve and future releases will likely introduce breaking changes. The goal of this release is to make it easy for anyone to try it and possibly give feedback.

What's new in Khepri 0.3.0

Revamp of the public API

The public API was completely revamped in Khepri 0.3.0, with the goal to make it more straightforward and consistent. Hopefully it will be easier to understand for both old and new users.

The #79 pull request gives more details about the changes and how existing code can be adapted. But here are some of the highlights:

• The high- vs low-level API distinction is now gone. The public API is now exposed by khepri only. khepri_machine becomes an internal private module. As part of that khepri grew several new functions for common use cases and we will certainly add more in the future, based on the feedback.
• Unix-like path are first-class citizen: all functions taking a native path ([stock, wood, <<"oak">>]) now accept Unix-like paths ("/:stock/:wood/oak"). In the process, the syntax of Unix-like paths evolved: atoms are represented as :atom and binaries are represented as-is, binary. The main reason is that using <<binary>> for binaires was difficult to read and type. See Accept string-based paths in khepri_machine and khepri_tx #73 and khepri_path: Change syntax of string-based paths #74.
• Payload and event filter records are now private. Payload types are automatically detected now, likewise for event filters. That said, it is still possible to use functions to construct the internal structures, but it should rarely be necessary.
• khepri_tx, the module to perform Khepri calls inside transactions, will now expose the same API as khepri, except when functions don't make sense in a transaction.

Here is an example of an old code and its newer version:

• Up to Khepri 0.2.1:

  %% `khepri_machine' had to be used for "advanced" use cases, though
  %% `khepri' would have been fine in this example.
  case khepri_machine:get(StoreId, [stock, wood, <<"oak">>]) of
      %% Accessing the data in the node payload was a bit complicated,
      %% requiring to pattern-match on the node properties map inside the
      %% result map.
      {ok, #{Path := #{data := Quantity}}} when Quantity < 100 ->
          %% We would also have to construct a payload record.
          Payload = #kpayload_data{data = 500},
          {ok, _} = khepri_machine:put(StoreId, [orders, wood, <<"oak">>], Payload),
          ok
      _ ->
          ok
  end.

• Starting from Khepri 0.3.0:

  %% Now we have helpers for common use cases like simply accessing the data
  %% of a single tree node. The piece of data is returned directly, returning
  %% a default value if there is no data and bypassing error handling if we
  %% don't care.
  %%
  %% Unix-like paths are used for the demonstration. Native paths, like in
  %% the previous example, would work as well.
  Quantity = khepri:get_data_or(StoreId, "/:stock/:wood/oak", 0),
  if
      Quantity < 100 ->
          %% The payload record is automatically constructed internally. No
          %% need to mess with records.
          {ok, _} = khepri:put(StoreId, "/:orders/:wood/oak", 500),
          ok;
      true ->
          ok
  end.

See #79.

Options to play with consistency and latency

Queries, including read-only transactions, now accept a favor option to select if Khepri should favor consistency or low latencies:

• consistency: it means that a "consistent query" will be used in Ra. It will return the most up-to-date piece of data the cluster agreed on. Note that it could block and time out if there is no quorum in the Ra cluster.
• compromise: it performs "leader queries" most of the time to reduce latency (though latency will still be higher than "local queries"; see below), but uses "consistent queries" every 10 seconds to verify that the cluster is healthy. It should be faster but may block and time out like consistency and still return slightly out-of-date data.
• low_latency: it means that "local queries" are used exclusively. They are the fastest and have the lowest latency. However, the returned data is whatever the local Ra server has. It could be out-of-date if it has troubles keeping up with the Ra cluster. The chance of blocking and timing out is very small.

%% Favor low latency, even if it means out-of-date data compared to what the consensus is.
khepri:get(StoreId, Path, #{favor => low_latency})

See #64.

Asynchronous updates

Updates, including read-write transactions, now accept an async option to configure asynchronous commands:

• true to perform an asynchronous low-priority command without a correlation ID.
• false to perform a synchronous command.
• A correlation ID to perform an asynchronous low-priority command with that correlation ID.
• A priority to perform an asynchronous command with the specified priority but without a correlation ID.
• A combination of a correlation ID and a priority to perform an asynchronous command with the specified parameters.

%% Configure a correlation ID for this update.
Ref = erlang:make_ref(),
ok = khepri:put(StoreId, Path, Data, #{async => Ref}),

%% Later, consume the reply from Khepri/Ra.
receive
    {ra_event, _, {applied, [{Ref, Reply}]}} ->
        do_things(Reply)
end.

See #69.

Other changes

• Introduced support for Erlang/OTP 25 (Support Erlang/OTP 25 #81).
• Ra was updated from 2.0.2 to 2.0.9 (Update to Ra 2.0.5 #70, Update to Ra 2.0.6 #71, Update to Ra 2.0.7 #80, Update to Ra 2.0.9 #89).
• Introduced a compiled standalone functions cache (khepri_fun: Cache compiled standalone function #72). Note that this cache is never cleaned up for now!
• Numerous fixes and improvements to the function extraction code (khepri_fun) from @the-mikedavis (khepri_fun: propagate argument annotations to callee in bs_no_start_match2 validation failures #63, Infer type annotation for test_arity test instruction #66, khepri_fun: fix coverage for infix operators #67, khepri_fun: expand support for bit-string instructions #68 and probably more).

Download

Khepri 0.3.0 is available from Hex.pm: https://hex.pm/packages/khepri/0.3.0

Upgrade

Using Rebar:

1. Update your rebar.config:

   %% In rebar.config
   {deps, [{khepri, "0.3.0"}]}.

2. Run rebar3 upgrade khepri.

Using Erlang.mk:

1. Update your Makefile:

   %% In your Makefile
   dep_khepri = hex 0.3.0

2. Remove the deps/khepri directory. The new version will be fetched the next time you build your project.

Documentation

The documentation for Khepri 0.3.0 is available on Hex.pm.

Contributors

A warm thank you to contributors outside of the RabbitMQ team, this is invaluable for such a young project!

• Kian-Meng Ang (@kianmeng)
• Michael Davis (@the-mikedavis)

---

This discussion was created from the release Khepri 0.3.0.