openssl/doc/designs/quic-design/stream-receive-buffers.md

Stream Receive Buffers

This is a QUIC specific module that retains the received stream data until the application reads it with SSL_read() or any future stream read calls.

Receive Buffers requirements for MVP

These are the requirements that were identified for MVP:

• As packets with stream frames are received in arbitrary frames the received data must be stored until all the data with earlier offsets are received.
• As packets can be received before application calls SSL_read() to read the data the data must be stored.
• The application should be able to set the limit on how much data should be stored. The flow controller should be used to limit the peer to not send more data. Without the flow control limit a rogue peer could trigger a DoS via unlimited flow of incoming stream data frames.
• After the data is passed via SSL_read() to the application the stored data can be released and flow control limit can be raised.
• As the peer can recreate stream data frames when resending them, the implementation must be able to handle properly frames with partially or fully overlapping data with previously received frames.

Optional Receive Buffers requirements

These are optional features of the stream receive buffers implementation. They are not required for MVP but they are otherwise desirable:

• To support a single copy operation with a future stream read call the received data should not be copied out of the decrypted packets to store the data. The only information actually stored would be a list of offset, length, and pointers to data, along with a pointer to the decrypted QUIC packet that stores the actual frame.

Proposed new public API calls

int SSL_set_max_stored_stream_data(SSL *stream, size_t length);

This function adjusts the current data flow control limit on the stream to allow storing length bytes of quic stream data before it is read by the application.

OpenSSL handles sending MAX_STREAM_DATA frames appropriately when the application reads the stored data.

int SSL_set_max_unprocessed_packet_data(SSL *connection,
                                        size_t length);

This sets the limit on unprocessed quic packet data length in bytes that is allowed to be allocated for the connection. See the Other considerations section below.

Interfaces to other QUIC implementation modules

Front End I/O API

SSL_read() copies data out of the stored buffers if available and eventually triggers release of stored unprocessed packet(s).

SSL_peek(), SSL_pending(), SSL_has_pending() peek into the stored buffers for any information about the stored data.

RX Depacketizer

The Receive Buffers module obtains the stream data via the ssl_queue_data() callback.

The module uses ossl_qrx_pkt_wrap_up_ref() and ossl_qrx_pkt_wrap_release() functions to keep and release decrypted packets with unprocessed data.

Flow Control

The Receive Buffers module provides an appropriate value for the Flow Control module to send MAX_DATA and MAX_STREAM_DATA frames. Details TBD.

QUIC Read Record Layer

The Receive Buffers module needs to know whether it should stop holding the decrypted quic packets and start copying the stream data due to the limit reached. See the SSL_set_max_unprocessed_quic_packet_data() function above and the Other considerations section below. Details TBD.

Implementation details

The QUIC_RSTREAM object holds the received stream data in the SFRAME_LIST structure. This is a sorted list of partially (never fully) overlapping data frames. Each list item holds a pointer to the received packet wrapper for refcounting and proper release of the received packet data once the stream data is read by the application.

Each SFRAME_LIST item has range.start and range.end values greater than the range.start and range.end values of the previous item in the list. This invariant is ensured on the insertion of overlapping stream frames. Any redundant frames are released. Insertion at the end of the list is optimised as in the ideal situation when no packets are lost we always just append new frames.

See include/internal/quic_stream.h and include/internal/quic_sf_list.h for internal API details.

Other considerations

The peer is allowed to recreate the stream data frames. As we aim for a single-copy operation a rogue peer could use this to override the stored data limits by sending duplicate frames with only slight changes in the offset. For example: 1st frame - offset 0 length 1000, 2nd frame - offset 1 length 1000, 3rd frame - offset 2 length 1000, and so on. We would have to keep the packet data for all these frames which would effectively raise the stream data flow control limit quadratically.

And this is not the only way how a rogue peer could make us occupy much more data than what is allowed by the stream data flow control limit in the single-copy scenario.

Although intuitively the MAX_DATA flow control limit might be used to somehow limit the allocated packet buffer size, it is defined as sum of allowed data to be sent across all the streams in the connection instead. The packet buffer will contain much more data than just the stream frames especially with a rogue peer, that means MAX_DATA limit cannot be used to limit the memory occupied by packet buffers.

To resolve this problem, we fall back to copying the data off the decrypted packet buffer once we reach a limit on unprocessed decrypted packets. We might also consider falling back to copying the data in case we receive stream data frames that are partially overlapping and one frame not being a subrange of the other.

Because in MVP only a single bidirectional stream to receive any data will be supported, the MAX_DATA flow control limit should be equal to MAX_STREAM_DATA limit for that stream.