Http2 is becoming language of the internet with little less than half of world traffic. It models requests and responses as binary frame streams multiplexed over single connection - major improvement over text based Http1 offering single shared stream.
This property enables support of different clients: browser applications over Http/REST, mobile/iot over GRPC, also mobile/iot/browser using RSocket-RPC - all served by same gateway/edge servers with common functionality (authorization, metrics, routing, load balancing etc) implemented in terms of Http2 streams.
Efficient internet protocol may be bad fit inside data center: RSocket gives option to transparently switch transport layer to less chatty one; means to keep latencies low with message level flow-control for request, requests concurrency control (leasing mechanism) for connection; programming model of composable asynchronous message streams with cancellation and error as first class citizens.
Clients do all interactions, servers are responders only
Client supported interactions are
Server initiated interactions are not supported because Http2 streams push-promise semantics is not suitable for RSocket.
RSocket has its own preface of either
RESUME frame that cant be understood by Http2.
Problem could be partially solved with Http2
SETTINGS custom parameters: their value size limit of 4 bytes
is enough for keep-alive fields and request leasing flag. Downside is lack of support by popular proxies as custom parameters
enable RSocket protocol specific features - hence requires custom solution.
That’s why each side of connection assumes initial state as follows:
- no session resumption support;
- empty setup payload;
- no requests leasing support: while technically possible, feature is better suited inside data centers than on internet;
- keep-alive interval and lifetime are 2^31-1 millis: client initiated periodic keep-alives are expensive for radio connected battery powered devices (major content consumers and primary target of this transport), instead server may send keep-alives on demand to determine peer liveness;
Carrying RSocket streams with Http2
Lets start with RSocket 0 stream frames.
Keep-alives are translated to Http2
PING frames with small data payload of 8 bytes - just enough to measure RTT.
Connection error frames are mapped to Http2
GO_AWAY with error code
no_error(0) and message containing RSocket
frame state in form
4 interactions streams are modelled after GRPC where Http2 stream is started with
HEADERS frame carrying request/response metadata,
DATA frames containing length delimited RSocket frames.
Client request streams are terminated by
end_stream flag on successful completion/error, or
RST_STREAM frame on cancellation.
Server response streams are terminated by trailer
Sequences are illustrated by example below
Client request starts with
:method POST :path contains interaction name for plain RSocket: /rsocket/fnf, /rsocket/response,/rsocket/stream,/rsocket/channel, or call name for RSocket-RPC: /service/method content-type application/rsocket+http2
followed by sequence of
DATA frames carrying length delimited RSocket frames.
Last frame is designated with
CANCEL frame of cancelled request is not encoded in
DATA, instead mapped to Http2
Server responds with
:status 200 content-type application/rsocket+http2
followed by sequence of
DATA frames, terminated by either success trailer
or error trailers containing code and message of RSocket
rsocket-status <error_code> rsocket-message <error_message>
Library relies on
netty - defacto standard for non-blocking networked applications, and is already present in RSocket core library.
netty-codec-http2 is the only additional direct dependency.
Few bullet points:
- tls/alpn and plaintext: latter is mostly for dev purposes
- optional epoll/kqueue transports - either transparently enabled if available on classpath, or enforced explicitly
- plain RSocket & RSocket-RPC support
- flow control: transport trusts requestN demand of protocol, Http2 receive windows are replenished
once available octets are less than half of
stream window sizefor stream, and
maxConcurrentStreams x stream window sizefor connection.
Client RSocketFactory can be configured by transport builder, with options set as required by transport contract outlined
Implied setup section
Mono<RSocket> client = NettyHttp2ClientTransport.builder() .address(host, port) .tls(insecureTrustManagerSslContext()) .userAgent("rsocket-transport-http2-example/0.0.1-SNAPSHOT") .headers("authorization", "Basic YWxhZGRpbjpvcGVuc2VzYW1l") // .rSocketRpc() // sets Path header to RSocket-RPC service/method instead of rsocket/<interaction_name> .maxConcurrentStreams(256) .streamWindowSize(100_000) .buildClient(clientRSocketFactory) .start();
Server can be configured in similar manner
Mono<CloseableChannel> server = NettyHttp2ServerTransport.builder() .address(host, port) .tls(selfSignedCertSslContext()) .maxConcurrentStreams(256) .streamWindowSize(100_000) .configureServer(serverRSocketFactory) .acceptor((setup, sendingSocket) -> Mono.just(new Responder())) .start();
Source code is hosted on Github: rsocket-transport-http2 repository.
For runnable example build project
./gradlew clean build
Then start server and client:
Client terminal should display received messages counter:
00:12:18.946 rsocket-transport-netty-http2-nio-1 com.jauntsdn.rsocket.transport.http2.example.client.Client Connected server on 192.168.0.52:8888 00:12:23.956 rsocket-transport-netty-http2-nio-1 com.jauntsdn.rsocket.transport.http2.example.client.Client received 238006 messages 00:12:28.955 rsocket-transport-netty-http2-nio-1 com.jauntsdn.rsocket.transport.http2.example.client.Client received 374223 messages 00:12:33.955 rsocket-transport-netty-http2-nio-1 com.jauntsdn.rsocket.transport.http2.example.client.Client received 393195 messages 00:12:38.955 rsocket-transport-netty-http2-nio-1 com.jauntsdn.rsocket.transport.http2.example.client.Client received 358807 messages 00:12:43.955 rsocket-transport-netty-http2-nio-1 com.jauntsdn.rsocket.transport.http2.example.client.Client received 376130 messages 00:12:48.954 rsocket-transport-netty-http2-nio-1 com.jauntsdn.rsocket.transport.http2.example.client.Client received 375942 messages
Lets check how single stream looks like on wire
Client request is sent after Http2 connection preface: 24 byte magic string and
Request is started with
HEADERS frame containing
:path=/rsocket/stream headers (we use plain RSocket), followed by
DATA frame with RSocket request frame. Http2 stream ids start from 3 because first id is reserved for Http1 - Http2 protocol upgrade flow.
Server responds with
HEADERS frame containing
:status=200, followed by sequence of
DATA frames holding length
delimited RSocket frames, terminated by response trailer
rsocket-status=0 denoting successful response completion.
We can evaluate transport performance by measuring interaction response latency and RPS with different concurrency limit on each run: 1, 4, 8. Lets start with request-stream as most common one.
Stream response contains 24 messages, message data is random and same string for given sequence, limited to 100 characters.
Epoll transport is enabled, tls is disabled, test host is 12 vCPU / 32 GB machine running Ubuntu 18.
Test server and client are available in rsocket-transport-http2-test module.
Request-stream test with concurrency limit = 8 can be started as follows
./perf_server.sh ./perf_client.sh request-stream 8
Results are presented below
18:32:58.407 rsocket-transport-netty-http2-epoll-1 com.jauntsdn.rsocket.transport.http2.perftest.TransportPerfClient --- request-stream , concurrency 1 --- 18:32:58.408 rsocket-transport-netty-http2-epoll-1 com.jauntsdn.rsocket.transport.http2.perftest.TransportPerfClient p50 => 126 microseconds 18:32:58.408 rsocket-transport-netty-http2-epoll-1 com.jauntsdn.rsocket.transport.http2.perftest.TransportPerfClient p95 => 147 microseconds 18:32:58.408 rsocket-transport-netty-http2-epoll-1 com.jauntsdn.rsocket.transport.http2.perftest.TransportPerfClient p99 => 164 microseconds 18:32:58.408 rsocket-transport-netty-http2-epoll-1 com.jauntsdn.rsocket.transport.http2.perftest.TransportPerfClient rps => 7813
18:34:16.801 rsocket-transport-netty-http2-epoll-1 com.jauntsdn.rsocket.transport.http2.perftest.TransportPerfClient --- request-stream , concurrency 4 --- 18:34:16.802 rsocket-transport-netty-http2-epoll-1 com.jauntsdn.rsocket.transport.http2.perftest.TransportPerfClient p50 => 289 microseconds 18:34:16.802 rsocket-transport-netty-http2-epoll-1 com.jauntsdn.rsocket.transport.http2.perftest.TransportPerfClient p95 => 395 microseconds 18:34:16.802 rsocket-transport-netty-http2-epoll-1 com.jauntsdn.rsocket.transport.http2.perftest.TransportPerfClient p99 => 459 microseconds 18:34:16.802 rsocket-transport-netty-http2-epoll-1 com.jauntsdn.rsocket.transport.http2.perftest.TransportPerfClient rps => 13446
18:37:59.846 rsocket-transport-netty-http2-epoll-1 com.jauntsdn.rsocket.transport.http2.perftest.TransportPerfClient --- request-stream , concurrency 8 --- 18:37:59.846 rsocket-transport-netty-http2-epoll-1 com.jauntsdn.rsocket.transport.http2.perftest.TransportPerfClient p50 => 547 microseconds 18:37:59.846 rsocket-transport-netty-http2-epoll-1 com.jauntsdn.rsocket.transport.http2.perftest.TransportPerfClient p95 => 657 microseconds 18:37:59.846 rsocket-transport-netty-http2-epoll-1 com.jauntsdn.rsocket.transport.http2.perftest.TransportPerfClient p99 => 842 microseconds 18:37:59.846 rsocket-transport-netty-http2-epoll-1 com.jauntsdn.rsocket.transport.http2.perftest.TransportPerfClient rps => 14439
Under concurrency 8 test, 14k streams (and 14439 x 24 = 346536 messages) per second per core are served with sub millisecond latency.
Concurrency 1 demonstrates lowest latency - little more than 160 microseconds, but throughput is underutilized: around 8k streams per second only.
Forking was motivated by series of problems making official organization repo
rsocket/rsocket-java hardly usable.
Here is brief outline of blockers that made implementation on top of It not feasible:
- out of order streams: stream request frames may have non-sequential ids: this is against both RSocket and Http2 spec.
- multiple cases of excessive frames sent on non-happy path: stream termination, calls after RSocket is terminated, few others; This makes mapping RSocket streams to Http2 harder than necessary because of Netty Http2 codec strictness;
- deficient custom ByteBuf implementation (named
TupleByteBufthere), unusable with ssl enabled transports due to bugs in write related methods;
- series of changes declared to improve performance also introduced hard to debug concurrency problems randomly manifesting under load;
Unfortunate bonus bit:
rsocket/rsocket-java (and all of the above) is part of
spring-boot:2.2.x / spring-integration:5.2.x - latest major releases.
Their users cant have RSocket with tls enabled transports, and with non-tls witness their services suddenly stop responding after few hours deployed.