Jaunt-RSocket-RPC, Spring-RSocket, GRPC: quantitative and qualitative comparison

September 3, 2021
RSocket java

Earlier this year we described weaknesses of Spring RSocket-java implementation that resulted in awful throughput, latency & garbage production: key metrics for core networking library.

Latest Spring-RSocket uses RSocket/RSocket-java version 1.1.1, and wraps up almost a year of progress made on the library.

The goal of this post is to re-evaluate that progress against both current jauntsdn/RSocket-RPC (not yet publicly available), and grpc/grpc-java.

Jauntsdn-RSocket-RPC vs Spring-RSocket-CBOR: quantitative comparison

jauntsdn-RSocket-RPC is remote procedure call system using RSocket-reactor for networking & APIs, and Protocol Buffers as sole data format. It relies on code generation with protobuf C++ plugin extensions, and is compatible with GRPC.

Spring-RSocket (version 2.5.3) offers reflection-heavy client and server on top of RSocket-java. Both are configured with HttpMessageEncoder and HttpMessageDecoder from spring’s http stack. Each call starts with interpolated “route” template string, followed by message encoding/decoding operators chain (for client chain size varies from 7 for request-response to 14 for request-channel).

Out of 2 serialization options suggested in their docs we choose CBOR instead of JSON as more compact so theoretically faster.

Additionally GRPC-java (1.37) only setup is used as performance base line. Both client and server are configured with static flow control window of 1_000_000.

Our objective is quick performance re-evaluation since 1 million streams. part2, so only single vCPU throughput is measured for 2 common interactions: request-stream (single infinite stream) and request-response (continuous window of requests).

Payload sizes are selected as follows:
4 (practical minimum: packed telemetry), 140 (tweet in latin) bytes

Tests were conducted on 12 vCPU commodity box running linux 5.4.0/OpenJDK 11.0.11, with non-TLS TCP transport.

Request-stream

jauntsdn-RSocket-RPC

11:48:32.984 rsocket-netty-io-transport-epoll-1-1 com.jauntsdn.rsocket.rpc.examples.stream.client.Main client received messages: 2877158
11:48:33.984 rsocket-netty-io-transport-epoll-1-1 com.jauntsdn.rsocket.rpc.examples.stream.client.Main client received messages: 2873612
11:48:34.984 rsocket-netty-io-transport-epoll-1-1 com.jauntsdn.rsocket.rpc.examples.stream.client.Main client received messages: 2875385

spring-RSocket-CBOR

2021-09-03 12:15:12.766  INFO 11603 --- [     parallel-2] s.r.stream.client.ClientApplication      : client received messages: 548060
2021-09-03 12:15:13.766  INFO 11603 --- [     parallel-2] s.r.stream.client.ClientApplication      : client received messages: 541163
2021-09-03 12:15:14.766  INFO 11603 --- [     parallel-2] s.r.stream.client.ClientApplication      : client received messages: 522535

GRPC-Java

12:32:16.273 pool-2-thread-1 com.jauntsdn.grpc.stress_test.stream.Main  client received messages: 1667608
12:32:17.273 pool-2-thread-1 com.jauntsdn.grpc.stress_test.stream.Main  client received messages: 1687596
12:32:18.273 pool-2-thread-1 com.jauntsdn.grpc.stress_test.stream.Main  client received messages: 1657052

jauntsdn-RSocket-RPC

12:01:09.603 rsocket-netty-io-transport-epoll-1-1 com.jauntsdn.rsocket.rpc.examples.stream.client.Main client received messages: 2235122
12:01:10.646 rsocket-netty-io-transport-epoll-1-1 com.jauntsdn.rsocket.rpc.examples.stream.client.Main client received messages: 2346001
12:01:11.668 rsocket-netty-io-transport-epoll-1-1 com.jauntsdn.rsocket.rpc.examples.stream.client.Main client received messages: 2235122

spring-RSocket-CBOR

2021-09-03 12:20:06.531  INFO 12357 --- [     parallel-2] s.r.stream.client.ClientApplication      : client received messages: 497637
2021-09-03 12:20:07.531  INFO 12357 --- [     parallel-2] s.r.stream.client.ClientApplication      : client received messages: 493875
2021-09-03 12:20:08.531  INFO 12357 --- [     parallel-2] s.r.stream.client.ClientApplication      : client received messages: 497627

GRPC-Java

12:36:51.114 pool-2-thread-1 com.jauntsdn.grpc.stress_test.stream.Main  client received messages: 1004188
12:36:52.114 pool-2-thread-1 com.jauntsdn.grpc.stress_test.stream.Main  client received messages: 1026791
12:36:53.114 pool-2-thread-1 com.jauntsdn.grpc.stress_test.stream.Main  client received messages: 988619

With request-streams spring-RSocket not only 5x slower (2.9M vs 0.55M) than jauntsdn-RSocket-RPC, but 2-3x slower than GRPC-java itself - “legacy” library It was aiming to supersede.

Request-response

jauntsdn-RSocket-RPC

12:03:12.019 rsocket-netty-io-transport-epoll-1-1 com.jauntsdn.rsocket.rpc.examples.response.client.Main client received messages: 1337757
12:03:13.019 rsocket-netty-io-transport-epoll-1-1 com.jauntsdn.rsocket.rpc.examples.response.client.Main client received messages: 1349939
12:03:14.019 rsocket-netty-io-transport-epoll-1-1 com.jauntsdn.rsocket.rpc.examples.response.client.Main client received messages: 1333029

spring-RSocket-CBOR

2021-09-03 12:23:49.258  INFO 12745 --- [     parallel-2] s.r.response.client.ClientApplication    : client received messages: 101036
2021-09-03 12:23:50.258  INFO 12745 --- [     parallel-2] s.r.response.client.ClientApplication    : client received messages: 102121
2021-09-03 12:23:51.258  INFO 12745 --- [     parallel-2] s.r.response.client.ClientApplication    : client received messages: 102134

GRPC-Java

12:38:15.146 pool-2-thread-1 com.jauntsdn.grpc.stress_test.response.Main client received messages: 89277
12:38:16.146 pool-2-thread-1 com.jauntsdn.grpc.stress_test.response.Main client received messages: 91057
12:38:17.146 pool-2-thread-1 com.jauntsdn.grpc.stress_test.response.Main client received messages: 91510

jauntsdn-RSocket-RPC

12:04:15.703 rsocket-netty-io-transport-epoll-1-1 com.jauntsdn.rsocket.rpc.examples.response.client.Main client received messages: 965725
12:04:16.729 rsocket-netty-io-transport-epoll-1-1 com.jauntsdn.rsocket.rpc.examples.response.client.Main client received messages: 1002753
12:04:17.784 rsocket-netty-io-transport-epoll-1-1 com.jauntsdn.rsocket.rpc.examples.response.client.Main client received messages: 1019256

spring-RSocket-CBOR

2021-09-03 12:25:13.025  INFO 13023 --- [     parallel-2] s.r.response.client.ClientApplication    : client received messages: 93324
2021-09-03 12:25:14.025  INFO 13023 --- [     parallel-2] s.r.response.client.ClientApplication    : client received messages: 92235
2021-09-03 12:25:15.025  INFO 13023 --- [     parallel-2] s.r.response.client.ClientApplication    : client received messages: 94413

GRPC-Java

12:43:11.855 pool-2-thread-1 com.jauntsdn.grpc.stress_test.response.Main client received messages: 85854
12:43:12.855 pool-2-thread-1 com.jauntsdn.grpc.stress_test.response.Main client received messages: 82733
12:43:13.855 pool-2-thread-1 com.jauntsdn.grpc.stress_test.response.Main client received messages: 82587

On request-response situation turned bleak very quickly for spring-RSocket: 10x-13x slower (1.3M vs 0.1M) than jauntsdn-RSocket-RPC. I didn’t believe the measurements at first, so had to double check - setup was correct.

On the bright side, spring-RSocket holds on par against GRPC - with one digit percent advantage.

Let’s look at these results from practical point - with current Spring, 80% (request-stream) to 90% (request-response) of CPU time is wasted processing RSocket/RSocket-java & framework internals - share of the cloud infra money that are turned into heat on datacenter hardware.

Spring-RSocket: false promise

With Spring and RSocket/RSocket-java, specialized binary, purpose-built cloud L5 protocol has glaringly worse performance over TCP than prior existing GRPC-java over http2 - general purpose internet protocol.

It leaves false impression of RSocket-as-protocol deficiency, while in reality It solely reflects engineering capacity, goals & decisions of organization behind Spring & RSocket/RSocket-java.

RSocket-as-protocol is capable of times better throughput with very small latency/GC pressure compared to GRPC - if proper implementation is in place.

When coupled with Reactive-Streams library (at least 3 of production ready quality available for JVM), RSocket enables bidirectional exchange of non-blocking message streams across processes (unix sockets) and networks (tcp, websockets, http2, udp/quic), at rate of millions messages per core, with latency of several milliseconds.

Spring’s RSocket state is caused by questionable foundation of RSocket/RSocket-java, bloated metadata encoding, switching from approach of single data format & codegen to data format abstraction plus reflection - which cannot be efficient or optimized due to substantial difference between data formats.

Note that both jauntsdn/RSocket-RPC-reactor and Spring RSocket/RSocket-java use same non-blocking primitives provided by netty & project-reactor, but have 5x-10x throughput difference - textbook example on consequences if proper design, scope & conservative dependencies are neglected.

Unification vs specialization

Unification vs specialization is a tradeoff: in particular for data, with former It is not possible to have “universal” optimizations, effective on multiple considerably different data formats - performance is traded for pluggability.

As enterprise applications framework Spring follows unification paradigm from its inception, so every piece of It is designed pluggable - at the expense of efficiency.

Unfortunately, as soon as server applications started being deployed to thoroughly metered 3rd party cloud VMs, efficiency and resources usage became primary concern.

That’s why jauntsdn/RSocket-RPC specializes on single data format - Protocol Buffers. It is compact, fast, widely supported and native for GRPC.

In case of spring-RSocket substituting codec with protobuf did not increase throughput, and also did not make It directly accessible by GRPC clients, despite proven RSocket-GRPC streams interoperability.

Jauntsdn-RSocket-RPC vs GRPC: qualitative comparison

Both jauntsdn/RSocket-RPC and GRPC are based on Protocol Buffers (protobuf). They rely on protobuf compiler’s service extensions framework to generate client & server stubs from language agnostic service and message definitions.

GRPC service APIs are callbacks based, while jauntsdn/RSocket-RPC APIs are Publisher based: which means they are composable, and represent errors and cancellations as first class concept.

Performance evaluation of RSocket-RPC was presented earlier, let’s continue with short list of qualitative differences between the two:

Summary: alternative RSocket library for high performance network applications on JVM

July 6, 2021
RSocket java

Alternative RSocket-RPC: fast application services communication and transparent GRPC bridge

May 20, 2021
RSocket java grpc

RSocket-JVM: streamlining implementation for each vendor platform

April 22, 2021
RSocket java helidon