Motivation

In practice reactive applications - as a method to model software and programming paradigm of composable flow controlled streams - viable mainly within JVM context.

It has necessary ecosystem provided by several competing platforms: smallrye-mutiny by Red Hat, project reactor by (ex-)Pivotal, and community-maintained rxjava.

Libraries are interoperable on same VM because their APIs semantically conform to reactive-streams specification, with 3 of 4 having binary compatibility - streams may be combined directly without writing any adapter code.

On the other hand interprocess and network communication is done with Grpc and Http/2.

Http is general purpose application protocol to retrieve resources from server to client with request/response model.

Grpc is also based on Http/2 streams, that’s why It is also non-symmetric (only client can initiate requests to server), and has byte oriented flow control while Grpc encodes data as messages - protocol buffers.

Stable Grpc stack is available for wide range of programming languages, however its java impl has “awkward” streaming APIs, lacks composability and produces relatively high volume of per-message garbage.

Http/Grpc/Graphql is good choice for heterogeneous internet environment as general purpose solution, with libraries and tools available for many hardware/os/language combinations.

For specialized cloud environment which happens to be JVM centric there is another option - RSocket. It is binary, symmetric, session level protocol with multiple streaming interactions support, reactive-streams message flow control and pluggable transports.

Initially there was official rsocket/rsocket-java implementation, based on project-reactor.

Unfortunately all its core features, transports and instrumentation modules are based on spring-boot related dependencies that can’t be reused on competing platforms.

Porting rsocket/rsocket-java to each platform would be not only laborious, but also unfeasible effort given questionable performance of rsocket/rsocket-java post 1.0 release.

Instead protocol core was reimplemented with minimal, vendor-neutral dependencies so each vendor project can be built upon It - rsocket-jvm.

This way next library development becomes predictable and largely mechanical process - most of the functionality is shared, while general structure becomes clear once first implementation is completed.

rsocket-jvm was evaluated with 3 vendor libraries: project-reactor, rxjava & smallrye-mutiny.

Structure

Conceptually all components belong to 5 categories:

• API CONTRACTS.

RSocket-messages is netty-buffers only project, accompanied by RSocket-<VENDOR>, RSocket-RPC-<VENDOR> pair for each vendor library (see RSocket-reactor, RSocket-RPC-reactor for example).

API contracts plus RSocket-RPC-compiler are sufficient for implementing end-user services without exposing unnecessary runtime details.

• TRANSPORT CONTRACT.

Includes type declarations for transports - components able to send and receive raw bytes. They only depend on rsocket-messages so transports may be used with each platform library.

• PROTOCOL COMPONENTS.

Contain software building blocks implementing major features of the protocol: handshakes, keep-alive, requests leasing, session resumption, server graceful close, error handling etc.

Protocol components do not depend on vendor specific libraries so serve as basis for each platform library.

Additionally It hosts APIs (but not implementations) that enable end-user application specific code sharing across platforms: RSocket interceptors and connection listeners for metrics, requests lease handlers for load estimation, resumable session stores and session discovery.

• SHARED MODULES.

Denote projects utilized by each platform library as ready-to-use components, both public and private. Currently these are TCP, Unix sockets & GRPC transports, load estimator and metrics.

• PLATFORM LIBRARIES.

Top-level component intended for end users. It contains RSocket and RSocket-RPC implementations (plus RPC compiler) with vendor specific APIs and glue code to link protocol components together. Platform libraries are based on Protocol components and are configurable with Transports and Shared modules.

RSocket-JVM

Protocol components and Shared modules comprise core of the project that is foundational for each platform library.

rsocket-jvm assumes Java 8+ and depends on netty-buffer only.

RSocket - Reactor

Chronologically is first project that emerged as result of researching for 1 million streams (part2).

It is based on project-reactor of spring boot framework.

Comparing to original rsocket/rsocket-java, rsocket-reactor consumes 2x less CPU for same workloads, with virtually non-existent per message garbage.

Numbers below are from runs on commodity box with linux 5.4.0 and TCP/EPOLL, OpenJDK 11.0.11.

Single core request-response throughput, messages per second

08:34:32.638 rsocket-netty-io-transport-epoll-1-1 com.jauntsdn.rsocket.examples.response.client.Main client received messages: 2153067
08:34:33.638 rsocket-netty-io-transport-epoll-1-1 com.jauntsdn.rsocket.examples.response.client.Main client received messages: 2142966
08:34:34.638 rsocket-netty-io-transport-epoll-1-1 com.jauntsdn.rsocket.examples.response.client.Main client received messages: 2144672
08:34:35.638 rsocket-netty-io-transport-epoll-1-1 com.jauntsdn.rsocket.examples.response.client.Main client received messages: 2153692
08:34:36.638 rsocket-netty-io-transport-epoll-1-1 com.jauntsdn.rsocket.examples.response.client.Main client received messages: 2137679

Single core server-stream throughput, messages per second

08:39:17.839 rsocket-netty-io-transport-epoll-1-1 com.jauntsdn.rsocket.examples.stream.client.Main client received messages: 3567742
08:39:18.839 rsocket-netty-io-transport-epoll-1-1 com.jauntsdn.rsocket.examples.stream.client.Main client received messages: 3575242
08:39:19.839 rsocket-netty-io-transport-epoll-1-1 com.jauntsdn.rsocket.examples.stream.client.Main client received messages: 3577742
08:39:20.839 rsocket-netty-io-transport-epoll-1-1 com.jauntsdn.rsocket.examples.stream.client.Main client received messages: 3587117
08:39:21.839 rsocket-netty-io-transport-epoll-1-1 com.jauntsdn.rsocket.examples.stream.client.Main client received messages: 3577742

Single core channel throughput, messages per second

Client

08:39:01.915 rsocket-netty-io-transport-epoll-1-1 com.jauntsdn.rsocket.examples.channel.client.Main client received messages: 2333310
08:39:02.938 rsocket-netty-io-transport-epoll-1-1 com.jauntsdn.rsocket.examples.channel.client.Main client received messages: 2411087
08:39:03.915 rsocket-netty-io-transport-epoll-1-1 com.jauntsdn.rsocket.examples.channel.client.Main client received messages: 2333310
08:39:04.923 rsocket-netty-io-transport-epoll-1-1 com.jauntsdn.rsocket.examples.channel.client.Main client received messages: 2411087
08:39:05.943 rsocket-netty-io-transport-epoll-1-1 com.jauntsdn.rsocket.examples.channel.client.Main client received messages: 2411087

Server

08:39:00.984 rsocket-netty-io-transport-epoll-1-2 com.jauntsdn.rsocket.examples.channel.server.Main server received messages: 2411087
08:39:01.979 rsocket-netty-io-transport-epoll-1-2 com.jauntsdn.rsocket.examples.channel.server.Main server received messages: 2333310
08:39:02.969 rsocket-netty-io-transport-epoll-1-2 com.jauntsdn.rsocket.examples.channel.server.Main server received messages: 2333310
08:39:03.978 rsocket-netty-io-transport-epoll-1-2 com.jauntsdn.rsocket.examples.channel.server.Main server received messages: 2411087
08:39:04.986 rsocket-netty-io-transport-epoll-1-2 com.jauntsdn.rsocket.examples.channel.server.Main server received messages: 2411087

RSocket-Helidon

Project is based on reactive-streams-common from helidon se - toolkit for rapid development of cloud java applications.

reactive-streams-common is JDK only reactive library using Flow from java.util.concurrent instead of reactive-streams-jvm, and has tight integration with JDK’s CompletableFutures.

It has relatively modest set of operators available out of the box, but custom ones are easy to implement.

Initial evaluation with TCP transport demonstrated that rsocket-helidon performs slightly better than rsocket-reactor. At the same time helidon-common-reactive has considerably simpler internals compared to project-reactor.

Numbers below are from runs on commodity box with linux 5.4.0 and TCP/EPOLL, running OpenJDK 11.0.11.

Single core request-response throughput, messages per second

08:45:11.301 rsocket-netty-io-transport-epoll-1-1 com.jauntsdn.rsocket.examples.response.client.Main client received messages: 2574258
08:45:12.305 rsocket-netty-io-transport-epoll-1-1 com.jauntsdn.rsocket.examples.response.client.Main client received messages: 2418647
08:45:13.301 rsocket-netty-io-transport-epoll-1-1 com.jauntsdn.rsocket.examples.response.client.Main client received messages: 2317546
08:45:14.292 rsocket-netty-io-transport-epoll-1-1 com.jauntsdn.rsocket.examples.response.client.Main client received messages: 2301992
08:45:15.290 rsocket-netty-io-transport-epoll-1-1 com.jauntsdn.rsocket.examples.response.client.Main client received messages: 2317546
08:45:16.302 rsocket-netty-io-transport-epoll-1-1 com.jauntsdn.rsocket.examples.response.client.Main client received messages: 2356431

Single core server-stream throughput, messages per second

08:54:05.186 rsocket-netty-io-transport-epoll-1-1 com.jauntsdn.rsocket.examples.stream.client.Main client received messages: 3757323
08:54:06.186 rsocket-netty-io-transport-epoll-1-1 com.jauntsdn.rsocket.examples.stream.client.Main client received messages: 3767671
08:54:07.186 rsocket-netty-io-transport-epoll-1-1 com.jauntsdn.rsocket.examples.stream.client.Main client received messages: 3762671
08:54:08.186 rsocket-netty-io-transport-epoll-1-1 com.jauntsdn.rsocket.examples.stream.client.Main client received messages: 3758573
08:54:09.186 rsocket-netty-io-transport-epoll-1-1 com.jauntsdn.rsocket.examples.stream.client.Main client received messages: 3765171

Single core channel throughput, messages per second

Client

08:53:33.035 rsocket-netty-io-transport-epoll-1-1 com.jauntsdn.rsocket.examples.channel.client.Main client received messages: 2566641
08:53:34.016 rsocket-netty-io-transport-epoll-1-1 com.jauntsdn.rsocket.examples.channel.client.Main client received messages: 2488864
08:53:35.027 rsocket-netty-io-transport-epoll-1-1 com.jauntsdn.rsocket.examples.channel.client.Main client received messages: 2566641
08:53:36.008 rsocket-netty-io-transport-epoll-1-1 com.jauntsdn.rsocket.examples.channel.client.Main client received messages: 2566641
08:53:37.012 rsocket-netty-io-transport-epoll-1-1 com.jauntsdn.rsocket.examples.channel.client.Main client received messages: 2644418

Server

08:53:30.025 rsocket-netty-io-transport-epoll-1-1 com.jauntsdn.rsocket.examples.channel.client.Main client received messages: 2488864
08:53:31.013 rsocket-netty-io-transport-epoll-1-1 com.jauntsdn.rsocket.examples.channel.client.Main client received messages: 2488864
08:53:32.023 rsocket-netty-io-transport-epoll-1-1 com.jauntsdn.rsocket.examples.channel.client.Main client received messages: 2566641
08:53:33.035 rsocket-netty-io-transport-epoll-1-1 com.jauntsdn.rsocket.examples.channel.client.Main client received messages: 2566641
08:53:34.016 rsocket-netty-io-transport-epoll-1-1 com.jauntsdn.rsocket.examples.channel.client.Main client received messages: 2488864

Server host CPU usage while running 1 million streams load test as described in first part of 1 million streams:

RSocket-helidon

 PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND  
7435 maksym    20   0   15,1g   2,3g  28192 S 452,3   7,5  12:00.72 java  

RSocket-reactor

PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND  
6384 maksym    20   0   15,0g   2,6g  28088 S 538,9   8,4  13:02.96 java  

On this workload helidon-reactive-streams consumes 20% less CPU time compared to project-reactor.

RSocket-rxjava, RSocket-smallrye-mutiny

Both libraries have internals similar to helidon-commons-reactive, so throughput performance is rather close.

Examples

rsocket-jvm-interop-examples is 4 vendor libraries interop demonstration (runtimes are stripped for now).