hyflow-go research software

a geo-replicated, main-memory, highly consistent datastore

Low Latency

Hyflow-go keeps remote communication minimal by employing a novel multi-master replication scheme.

High Throughput

We spared no effort to make hyflow-go fast, and we are still working on it.

Modular Architecture

Transaction Layer

It is based on commit-time transaction certification. It is flexible and supports Multi-Version Concurrency Control (MVCC) and single-versioning. Can be used to provide either serializability or Extended Update Serializability (EUS), depending on the underlying ordering layer.

Ordering Layer

Consensus protocols are pluggable. Choose between the partial order protocols such as Alvin POB, M2Paxos or E‑Paxos, and total order protocols like Multi‑Paxos and Raft*. Of course, for distributed systems researchers, rolling your own is always an option.

Communication Layer

The lowest layer deals with cluster communication. It is backed by zeromq, nanomsg or mangos. Optimistic batching is used to increase throughput while keeping latency low. Relies on code generation for maximum serialization performance at run-time.

* planned, but not yet available.

Implementation Details

Faster Consensus

Partial Ordering of Transactions

By tracking conflicts between transactions, we can avoid ordering (serializing) non-conflicting transactions. This enhances concurrency and allows transactions to commit earlier. Some non-serializable executions are allowed, but this is not generally a problem in practice.


In the Alvin POB and E-Paxos protocols, each node is in charge of coordinating the transactions originating at that location. This reduces the client perceived latency by one round trip, as transactions don't need to relayed to the global leader for ordering.

Fast Path

In the absence of contention, Alvin POB and E-Paxos can further reduce latency by one round trip by using fast quorums. Fast quorums are larger than classic quorums, so this may not necessarily result in a net improvement for all deployments.

Proudly written in Go

Rapid Prototyping

Go is a perfect match for writing research software. It is a high-level language, has low cognitive overhead, compiles fast, has automatic memory management and built-in concurrency. All these allow researchers and developers to focus more on the research problem they're trying to solve, and less on the engineering aspect.

High Performance

Go is fast! While it is quick to write and feels like a scripting language, Go is strongly typed and compiles to native code. It offers a great level of control over memory layout and allocation. The built-in profiler allows developers to detect and optimize the critical paths in their code.

Easy Deployments

Go compiles to self-contained, static binaries. Deployment is a matter of copying the executable to the destination machine. There are no virtual machines or other dependencies to setup.

Network Matters

Fast Serialization

In an earlier implementation, encoding/decoding data for network transmission quickly added up to 100% CPU capacity, limiting performance unnecessarily. After experimenting with a number of options, we decided to rely on code generation for maximum performance. While it adds an additional step at build-time, CPU usage due to serialization is now a non-issue.

Cluster Communication

To speed up prototyping new transactional and consensus protocols, we implemented a light-weight cluster communication library. This makes common operations quick and easy to write. Cluster dynamics, quorum handling, reply management and timeouts are all seamlessly handled here.

Smart Transport

Traditional transports such as TCP and UDP sockets lack a lot of functionality. Instead we use nanomsg-like sockets, which support advanced communication patterns and advanced batching algorithms. We use a pure-Go implementation called mangos.


hyflow-go is under development at the Systems Software Research Group5 at Virginia Tech.

Team members:

  • Alex Turcu
  • Sebastiano Peluso
  • Roberto Palmieri
  • Binoy Ravindran


[1] Be General and Don't Give Up Consistency in Geo-Replicated Transactional Systems.
A. Turcu, R. Palmieri, S. Peluso and B. Ravindran. In OPODIS 2014, Cortina, Italy.  


[2] There is more consensus in Egalitarian parliaments.
I. Moraru, D. Andersen and M. Kaminsky. In SOSP 2013, Farminton, PA.  
[3] In search of an understandable consensus algorithm.
D. Ongaro and J. Ousterhout. In USENIX ATC 2014, Philadelphia, PA.  


[4] Source Code: ser , remoting , hyflow-go
[5] Hyflow project
[6] Systems Software Research Group at Virginia Tech