TX Packetiser ============= This module creates frames from the application data obtained from the application. It also receives CRYPTO frames from the TLS Handshake Record Layer and ACK frames from the ACK Handling And Loss Detector subsystem. The packetiser also deals with the flow and congestion controllers. Creation & Destruction ---------------------- ```c typedef struct quic_tx_packetiser_args_st { /* Configuration Settings */ QUIC_CONN_ID cur_scid; /* Current Source Connection ID we use. */ QUIC_CONN_ID cur_dcid; /* Current Destination Connection ID we use. */ BIO_ADDR peer; /* Current destination L4 address we use. */ /* ACK delay exponent used when encoding. */ uint32_t ack_delay_exponent; /* Injected Dependencies */ OSSL_QTX *qtx; /* QUIC Record Layer TX we are using */ QUIC_TXPIM *txpim; /* QUIC TX'd Packet Information Manager */ QUIC_CFQ *cfq; /* QUIC Control Frame Queue */ OSSL_ACKM *ackm; /* QUIC Acknowledgement Manager */ QUIC_STREAM_MAP *qsm; /* QUIC Streams Map */ QUIC_TXFC *conn_txfc; /* QUIC Connection-Level TX Flow Controller */ QUIC_RXFC *conn_rxfc; /* QUIC Connection-Level RX Flow Controller */ const OSSL_CC_METHOD *cc_method; /* QUIC Congestion Controller */ OSSL_CC_DATA *cc_data; /* QUIC Congestion Controller Instance */ OSSL_TIME (*now)(void *arg); /* Callback to get current time. */ void *now_arg; /* * Injected dependencies - crypto streams. * * Note: There is no crypto stream for the 0-RTT EL. * crypto[QUIC_PN_SPACE_APP] is the 1-RTT crypto stream. */ QUIC_SSTREAM *crypto[QUIC_PN_SPACE_NUM]; } QUIC_TX_PACKETISER_ARGS; _owur typedef struct ossl_quic_tx_packetiser_st OSSL_QUIC_TX_PACKETISER; OSSL_QUIC_TX_PACKETISER *ossl_quic_tx_packetiser_new(QUIC_TX_PACKETISER_ARGS *args); void ossl_quic_tx_packetiser_free(OSSL_QUIC_TX_PACKETISER *tx); ``` Structures ---------- ### Connection Represented by an QUIC_CONNECTION object. ### Stream Represented by an QUIC_STREAM object. As per [RFC 9000 2.3 Stream Prioritization], streams should contain a priority provided by the calling application. For MVP, this is not required to be implemented because only one stream is supported. However, packets being retransmitted should be preferentially sent as noted in [RFC 9000 13.3 Retransmission of Information]. ```c void SSL_set_priority(SSL *stream, uint32_t priority); uint32_t SSL_get_priority(SSL *stream); ``` For protocols where priority is not meaningful, the set function is a noop and the get function returns a constant value. Interactions ------------ The packetiser interacts with the following components, the APIs for which can be found in their respective design documents and header files: - SSTREAM: manages application stream data for transmission. - QUIC_STREAM_MAP: Maps stream IDs to QUIC_STREAM objects and tracks which streams are active (i.e., need servicing by the TX packetiser). - Crypto streams for each EL other than 0-RTT (each is one SSTREAM). - CFQ: queried for generic control frames - QTX: record layer which completed packets are written to. - TXPIM: logs information about transmitted packets, provides information to FIFD. - FIFD: notified of transmitted packets. - ACKM: loss detector. - Connection and stream-level TXFC and RXFC instances. - Congestion controller (not needed for MVP). ### SSTREAM Each application or crypto stream has a SSTREAM object for the sending part. This manages the buffering of data written to the stream, frees that data when the packet it was sent in was acknowledged, and can return the data for retransmission on loss. It receives loss and acknowledgement notifications from the FIFD without direct TX packetiser involvement. ### QUIC Stream Map The TX packetiser queries the QUIC stream map for a list of active streams (QUIC_STREAM), which are iterated on a rotating round robin basis. Each QUIC_STREAM provides access to the various components, such as a QUIC_SSTREAM instance (for streams with a send part). Streams are marked inactive when they no longer have any need to generate frames at the present time. ### Crypto Streams The crypto streams for each EL (other than 0-RTT, which does not have a crypto stream) are represented by SSTREAM instances. The TX packetiser queries SSTREAM instances provided to it as needed when generating packets. ### CFQ Many control frames do not require special handling and are handled by the generic CFQ mechanism. The TX packetiser queries the CFQ for any frames to be sent and schedules them into a packet. ### QUIC Write Record Layer Coalesced frames are passed to the QUIC record layer for encryption and sending. To send accumulated frames as packets to the QUIC Write Record Layer: ```c int ossl_qtx_write_pkt(OSSL_QTX *qtx, const OSSL_QTX_PKT *pkt); ``` The packetiser will attempt to maximise the number of bytes in a packet. It will also attempt to create multiple packets to send simultaneously. The packetiser should also implement a wait time to allow more data to accumulate before exhausting it's supply of data. The length of the wait will depend on how much data is queued already and how much space remains in the packet being filled. Once the wait is finished, the packets will be sent by calling: ```c void ossl_qtx_flush_net(OSSL_QTX *qtx); ``` The write record layer is responsible for coalescing multiple QUIC packets into datagrams. ### TXPIM, FIFD, ACK Handling and Loss Detector ACK handling and loss detection is provided by the ACKM and FIFD. The FIFD uses the per-packet information recorded by the TXPIM to track which frames are contained within a packet which was lost or acknowledged, and generates callbacks to the TX packetiser, SSTREAM instances and CFQ to allow it to regenerate those frames as needed. 1. When a packet is sent, the packetiser informs the FIFD, which also informs the ACK Manager. 2. When a packet is ACKed, the FIFD notifies applicable SSTREAMs and the CFQ as appropriate. 3. When a packet is lost, the FIFD notifies the TX packetiser of any frames which were in the lost packet for which the Regenerate strategy is applicable. 4. Currently, no notifications to the TX packetiser are needed when packets are discarded (e.g. due to an EL being discarded). ### Flow Control The packetiser interacts with connection and stream-level TXFC and RXFC instances. It interacts with RXFC instances to know when to generate flow control frames, and with TXFC instances to know how much stream data it is allowed to send in a packet. ### Congestion Control The packetiser is likely to interact with the congestion controller in the future. Currently, congestion control is a no-op. Packets ------- Packet formats are defined in [RFC 9000 17.1 Packet Formats]. ### Packet types QUIC supports a number of different packets. The combination of packets of different encryption levels as per [RFC 9000 12.2 Coalescing Packets], is done by the record layer. Non-encrypted packets are not handled by the TX Packetiser and callers may send them by direct calls to the record layer. #### Initial Packet Refer to [RFC 9000 17.2.2 Initial Packet]. #### Handshake Packet Refer to [RFC 9000 17.2.4 Handshake Packet]. #### App Data 0-RTT Packet Refer to [RFC 9000 17.2.3 0-RTT]. #### App Data 1-RTT Packet Refer to [RFC 9000 17.3.1 1-RTT]. Packetisation and Processing ---------------------------- ### Definitions - Maximum Datagram Payload Length (MDPL): The maximum number of UDP payload bytes we can put in a UDP packet. This is derived from the applicable PMTU. This is also the maximum size of a single QUIC packet if we place only one packet in a datagram. The MDPL may vary based on both local source IP and destination IP due to different path MTUs. - Maximum Packet Length (MPL): The maximum size of a fully encrypted and serialized QUIC packet in bytes in some given context. Typically equal to the MDPL and never greater than it. - Maximum Plaintext Payload Length (MPPL): The maximum number of plaintext bytes we can put in the payload of a QUIC packet. This is related to the MDPL by the size of the encoded header and the size of any AEAD authentication tag which will be attached to the ciphertext. - Coalescing MPL (CMPL): The maximum number of bytes left to serialize another QUIC packet into the same datagram as one or more previous packets. This is just the MDPL minus the total size of all previous packets already serialized into to the same datagram. - Coalescing MPPL (CMPPL): The maximum number of payload bytes we can put in the payload of another QUIC packet which is to be coalesced with one or more previous QUIC packets and placed into the same datagram. Essentially, this is the room we have left for another packet payload. - Remaining CMPPL (RCMPPL): The number of bytes left in a packet whose payload we are currently forming. This is the CMPPL minus any bytes we have already put into the payload. - Minimum Datagram Length (MinDPL): In some cases we must ensure a datagram has a minimum size of a certain number of bytes. This does not need to be accomplished with a single packet, but we may need to add PADDING frames to the final packet added to a datagram in this case. - Minimum Packet Length (MinPL): The minimum serialized packet length we are using while serializing a given packet. May often be 0. Used to meet MinDPL requirements, and thus equal to MinDPL minus the length of any packets we have already encoded into the datagram. - Minimum Plaintext Payload Length (MinPPL): The minimum number of bytes which must be placed into a packet payload in order to meet the MinPL minimum size when the packet is encoded. - Active Stream: A stream which has data or flow control frames ready for transmission. ### Frames Frames are taken from [RFC 9000 12.4 Frames and Frame Types]. | Type | Name | I | H | 0 | 1 | N | C | P | F | |------|-----------------------|---------|---------|---------|---------|---------|---------|---------|---------| | 0x00 | padding | ✓ | ✓ | ✓ | ✓ | ✓ | | ✓ | | | 0x01 | ping | ✓ | ✓ | ✓ | ✓ | | | | | | 0x02 | ack 0x02 | ✓ | ✓ | | ✓ | ✓ | ✓ | | | | 0x03 | ack 0x03 | ✓ | ✓ | | ✓ | ✓ | ✓ | | | | 0x04 | reset_stream | | | ✓ | ✓ | | | | | | 0x05 | stop_sending | | | ✓ | ✓ | | | | | | 0x06 | crypto | ✓ | ✓ | | ✓ | | | | | | 0x07 | new_token | | | | ✓ | | | | | | 0x08 | stream 0x08 | | | ✓ | ✓ | | | | ✓ | | 0x09 | stream 0x09 | | | ✓ | ✓ | | | | ✓ | | 0x0A | stream 0x0A | | | ✓ | ✓ | | | | ✓ | | 0x0B | stream 0x0B | | | ✓ | ✓ | | | | ✓ | | 0x0C | stream 0x0C | | | ✓ | ✓ | | | | ✓ | | 0x0D | stream 0x0D | | | ✓ | ✓ | | | | ✓ | | 0x0E | stream 0x0E | | | ✓ | ✓ | | | | ✓ | | 0x0F | stream 0x0F | | | ✓ | ✓ | | | | ✓ | | 0x10 | max_data | | | ✓ | ✓ | | | | | | 0x11 | max_stream_data | | | ✓ | ✓ | | | | | | 0x12 | max_streams 0x12 | | | ✓ | ✓ | | | | | | 0x13 | max_streams 0x13 | | | ✓ | ✓ | | | | | | 0x14 | data_blocked | | | ✓ | ✓ | | | | | | 0x15 | stream_data_blocked | | | ✓ | ✓ | | | | | | 0x16 | streams_blocked 0x16 | | | ✓ | ✓ | | | | | | 0x17 | streams_blocked 0x17 | | | ✓ | ✓ | | | | | | 0x18 | new_connection_id | | | ✓ | ✓ | | | ✓ | | | 0x19 | retire_connection_id | | | ✓ | ✓ | | | | | | 0x1A | path_challenge | | | ✓ | ✓ | | | ✓ | | | 0x1B | path_response | | | | ✓ | | | ✓ | | | 0x1C | connection_close 0x1C | ✓ | ✓ | ✓ | ✓ | ✓ | | | | | 0x1D | connection_close 0x1D | | | ✓ | ✓ | ✓ | | | | | 0x1E | handshake_done | | | | ✓ | | | | | The various fields are as defined in RFC 9000. #### Pkts _Pkts_ are defined as: | Pkts | Description| | :---: | --- | | I | Valid in Initial packets| | H | Valid in Handshake packets| | 0 | Valid in 0-RTT packets| | 1 | Valid in 1-RTT packets| #### Spec _Spec_ is defined as: | Spec | Description | | :---: | --- | | N | Not ack-eliciting. | | C | does not count toward bytes in flight for congestion control purposes. | | P | Can be used to probe new network paths during connection migration. | | F | The contents of frames with this marking are flow controlled. | For `C`, `N` and `P`, the entire packet must consist of only frames with the marking for the packet to qualify for it. For example, a packet with an ACK frame and a _stream_ frame would qualify for neither the `C` or `N` markings. #### Notes - Do we need the distinction between 0-rtt and 1-rtt when both are in the Application Data number space? - 0-RTT packets can morph into 1-RTT packets and this needs to be handled by the packetiser. ### Frame Type Prioritisation The frame types listed above are reordered below in the order of priority with which we want to serialize them. We discuss the motivations for this priority ordering below. Items without a line between them have the same priority. ```plain HANDSHAKE_DONE GCR / REGEN ---------------------------- MAX_DATA REGEN DATA_BLOCKED REGEN MAX_STREAMS REGEN STREAMS_BLOCKED REGEN ---------------------------- NEW_CONNECTION_ID GCR RETIRE_CONNECTION_ID GCR ---------------------------- PATH_CHALLENGE - PATH_RESPONSE - ---------------------------- ACK - (non-ACK-eliciting) ---------------------------- CONNECTION_CLOSE *** (non-ACK-eliciting) ---------------------------- NEW_TOKEN GCR ---------------------------- CRYPTO GCR/*q ============================ ] priority group, repeats per stream RESET_STREAM GCR* ] STOP_SENDING GCR* ] ---------------------------- ] MAX_STREAM_DATA REGEN ] STREAM_DATA_BLOCKED REGEN ] ---------------------------- ] STREAM *q ] ============================ ] ---------------------------- PING - ---------------------------- PADDING - (non-ACK-eliciting) ``` (See [Frame in Flight Manager](quic-fifm.md) for information on the meaning of the second column, which specifies the retransmission strategy for each frame type.) - `PADDING`: For obvious reasons, this frame type is the lowest priority. We only add `PADDING` frames at the very end after serializing all other frames if we have been asked to ensure a non-zero MinPL but have not yet met that minimum. - `PING`: The `PING` frame is encoded as a single byte. It is used to make a packet ACK-eliciting if it would not otherwise be ACK-eliciting. Therefore we only need to send it if a. we have been asked to ensure the packet is ACK-eliciting, and b. we do not have any other ACK-eliciting frames in the packet. Thus we wait until the end before adding the PING frame as we may end up adding other ACK-eliciting frames and not need to add it. There is never a need to add more than one PING frame. If we have been asked to ensure the packet is ACK-eliciting and we do not know for sure up front if we will add any other ACK-eliciting packet, we must reserve one byte of our CMPPL to ensure we have room for this. We can cancel this reservation if we add an ACK-eliciting frame earlier. For example: - We have been asked to ensure a packet is ACK-eliciting and the CMPPL is 1000 (we are coalescing with another packet). - We allocate 999 bytes for non-PING frames. - While adding non-PING frames, we add a STREAM frame, which is ACK-eliciting, therefore the PING frame reservation is cancelled and we increase our allocation for non-PING frames to 1000 bytes. - `HANDSHAKE_DONE`: This is a single byte frame with no data which is used to indicate handshake completion. It is only ever sent once. As such, it can be implemented as a single flag, and there is no risk of it outcompeting other frames. It is therefore trivially given the highest priority. - `MAX_DATA`, `DATA_BLOCKED`: These manage connection-level flow control. They consist of a single integer argument, and, as such, take up little space, but are also critical to ensuring the timely expansion of the connection-level flow control window. Thus there is a performance reason to include them in packets with high priority and due to their small size and the fact that there will only ever be at most one per packet, there is no risk of them outcompeting other frames. - `MAX_STREAMS`, `STREAMS_BLOCKED`: Similar to the frames above for connection-level flow control, but controls rate at which new streams are opened. The same arguments apply here, so they are prioritised equally. - `STREAM`: This is the bread and butter of a QUIC packet, and contains application-level stream data. As such these frames can usually be expected to consume most of our packet's payload budget. We must generally assume that - there are many streams, and - several of those streams have much more data waiting to be sent than can be sent in a single packet. Therefore we must ensure some level of balance between multiple competing streams. We refer to this as stream scheduling. There are many strategies that can be used for this, and in the future we might even support application-signalled prioritisation of specific streams. We discuss stream scheduling further below. Because these frames are expected to make up the bulk of most packets, we consider them low priority, higher only than `PING` and `PADDING` frames. Moreover, we give priority to control frames as unlike `STREAM` frames, they are vital to the maintenance of the health of the connection itself. Once we have serialized all other frame types, we can reserve the rest of the packet for any `STREAM` frames. Since all `STREAM` frames are ACK-eliciting, if we have any `STREAM` frame to send at all, it cancels any need for any `PING` frame, and may be able to partially or wholly obviate our need for any `PADDING` frames which we might otherwise have needed. Thus once we start serializing STREAM frames, we are limited only by the remaining CMPPL. - `MAX_STREAM_DATA`, `STREAM_DATA_BLOCKED`: Stream-level flow control. These contain only a stream ID and integer value used for flow control, so they are not large. Since they are critical to the management and health of a specific stream, and because they are small and have no risk of stealing too many bytes from the `STREAM` frames they follow, we always serialize these before any corresponding `STREAM` frames for a given stream ID. - `RESET_STREAM`, `STOP_SENDING`: These terminate a given stream ID and thus are also associated with a stream. They are also small. As such, we consider these higher priority than both `STREAM` frames and the stream-level flow control frames. - `NEW_CONNECTION_ID`, `RETIRE_CONNECTION_ID`: These are critical for connection management and are not particularly large, therefore they are given a high priority. - `PATH_CHALLENGE`, `PATH_RESPONSE`: Used during connection migration, these are small and are given a high priority. - `CRYPTO`: These frames generate the logical crypto stream, which is a logical bidirectional bytestream used to transport TLS records for connection handshake and management purposes. As such, the crypto stream is viewed as similar to application streams but of a higher priority. We are willing to let `CRYPTO` frames outcompete all application stream-related frames if need be, as `CRYPTO` frames are more important to the maintenance of the connection and the handshake layer should not generate an excessive amount of data. - `CONNECTION_CLOSE`, `NEW_TOKEN`: The `CONNECTION_CLOSE` frame can contain a user-specified reason string. The `NEW_TOKEN` frame contains an opaque token blob. Both can be arbitrarily large but for the fact that they must fit in a single packet and are thus ultimately limited by the MPPL. However, these frames are important to connection maintenance and thus are given a priority just above that of `CRYPTO` frames. The `CONNECTION_CLOSE` frame has higher priority than `NEW_TOKEN`. - `ACK`: `ACK` frames are critical to avoid needless retransmissions by our peer. They can also potentially become large if a large number of ACK ranges needs to be transmitted. Thus `ACK` frames are given a fairly high priority; specifically, their priority is higher than all frames which have the potential to be large but below all frames which contain only limited data, such as connection-level flow control. However, we reserve the right to adapt the size of the ACK frames we transmit by chopping off some of the PN ranges to limit the size of the ACK frame if its size would be otherwise excessive. This ensures that the high priority of the ACK frame does not starve the packet of room for stream data. ### Stream Scheduling **Stream budgeting.** When it is time to add STREAM frames to a packet under construction, we take our Remaining CMPPL and call this value the Streams Budget. There are many ways we could make use of this Streams Budget. For the purposes of stream budgeting, we consider all bytes of STREAM frames, stream-level flow control frames, RESET_STREAM and STOP_SENDING frames to “belong” to their respective streams, and the encoded sizes of these frames are accounted to those streams for budgeting purposes. If the total number of bytes of frames necessary to serialize all pending data from all active streams is less than our Streams Budget, there is no need for any prioritisation. Otherwise, there are a number of strategies we could employ. We can categorise the possible strategies into two groups to begin with: - **Intrapacket muxing (IRPM)**. When the data available to send across all streams exceeds the Streams Budget for the packet, allocate an equal portion of the packet to each stream. - **Interpacket muxing (IXPM).** When the data available to send across all streams exceeds the Streams Budget for the packet, try to fill the packet using as few streams as possible, and multiplex by using different streams in different packets. Though obvious, IRPM does not appear to be a widely used strategy [1] [2], probably due to a clear downside: if a packet is lost and it contains data for multiple streams, all of those streams will be held up. This undermines a key advantage of QUIC, namely the ability of streams to function independently of one another for the purposes of head-of-line blocking. By contrast, with IXPM, if a packet is lost, typically only a single stream is held up. Suppose we choose IXPM. We must now choose a strategy for deciding when to schedule streams on packets. [1] establishes that there are two basic strategies found in use: - A round robin (RR) strategy in which the frame scheduler switches to the next active stream every n packets (where n ≥ 1). - A sequential (SEQ) strategy in which a stream keeps being transmitted until it is no longer active. The SEQ strategy does not appear to be suitable for general-purpose applications as it presumably starves other streams of bandwidth. It appears that this strategy may be chosen in some implementations because it can offer greater efficiency with HTTP/3, where there are performance benefits to completing transmission of one stream before beginning the next. However, it does not seem like a suitable choice for an application-agnostic QUIC implementation. Thus the RR strategy is the better choice and the popular choice in a survey of implementations. The choice of `n` for the RR strategy is most trivially 1 but there are suggestions [1] that a higher value of `n` may lead to greater performance due to packet loss in typical networks occurring in small durations affecting small numbers of consecutive packets. Thus, if `n` is greater than 1, fewer streams will be affected by packet loss and held up on average. However, implementing different values of `n` poses no non-trivial implementation concerns, so it is not a major concern for discussion here. Such a parameter can easily be made configurable. Thus, we choose what active stream to select to fill in a packet on a revolving round robin basis, moving to the next stream in the round robin every `n` packets. If the available data in the active stream is not enough to fill a packet, we do also move to the next stream, so IRPM can still occur in this case. When we fill a packet with a stream, we start with any applicable `RESET_STREAM` or `STOP_SENDING` frames, followed by stream-level flow control frames if needed, followed by `STREAM` frames. (This means that `RESET_STREAM`, `STOP_SENDING`, `MAX_STREAM_DATA`, `STREAM_DATA_BLOCKED` and `STREAM` frames are interleaved rather than occurring in a fixed priority order; i.e., first there could be a `STOP_SENDING` frame for one stream, then a `STREAM` frame for another, then another `STOP_SENDING` frame for another stream, etc.) [1] [Same Standards; Different Decisions: A Study of QUIC and HTTP/3 Implementation Diversity (Marx et al. 2020)](https://qlog.edm.uhasselt.be/epiq/files/QUICImplementationDiversity_Marx_final_11jun2020.pdf) [2] [Resource Multiplexing and Prioritization in HTTP/2 over TCP versus HTTP/3 over QUIC (Marx et al. 2020)](https://h3.edm.uhasselt.be/files/ResourceMultiplexing_H2andH3_Marx2020.pdf) ### Packets with Special Requirements Some packets have special requirements which the TX packetiser must meet: - **Padded Initial Datagrams.** A datagram must always be padded to at least 1200 bytes if it contains an Initial packet. (If there are multiple packets in the datagram, the padding does not necessarily need to be part of the Initial packet itself.) This serves to confirm that the QUIC minimum MTU is met. - **Token in Initial Packets.** Initial packets may need to contain a token. If used, token is contained in all further Initial packets sent by the client, not just the first Initial packet. - **Anti-amplification Limit.** Sometimes a lower MDPL may be imposed due to anti-amplification limits. (Only a concern for servers, so not relevant to MVP.) Note: It has been observed that a lot of implementations are not fastidious about enforcing the amplification limit in terms of precise packet sizes. Rather, they just use it to determine if they can send another packet, but not to determine what size that packet must be. Implementations with 'precise' anti-amplification implementations appear to be rare. - **MTU Probes.** These packets have a precisely crafted size for the purposes of probing a path MTU. Unlike ordinary packets, they are routinely expected to be lost and this loss should not be taken as a signal for congestion control purposes. (Not relevant for MVP.) - **Path/Migration Probes.** These packets are sent to verify a new path for the purposes of connection migration. - **ACK Manager Probes.** Packets produced because the ACK manager has requested a probe be sent. These MUST be made ACK-eliciting (using a PING frame if necessary). However, these packets need not be reserved exclusively for ACK Manager purposes; they SHOULD contain new data if available, and MAY contain old data. We handle the need for different kinds of packet via a notion of “archetypes”. The TX packetiser is requested to generate a datagram via the following call: ```c /* Generate normal packets containing most frame types. */ #define TX_PACKETISER_ARCHETYPE_NORMAL 0 /* Generate ACKs only. */ #define TX_PACKETISER_ARCHETYPE_ACK_ONLY 1 int ossl_quic_tx_packetiser_generate(OSSL_QUIC_TX_PACKETISER *txp, uint32_t archetype); ``` More archetypes can be added in the future as required. The archetype limits what frames can be placed into the packets of a datagram. ### Encryption Levels A QUIC connection progresses through Initial, Handshake, 0-RTT and 1-RTT encryption levels (ELs). The TX packetiser decides what EL to use to send a packet; or rather, it would be more accurate to say that the TX packetiser decides what ELs need a packet generating. Many resources are instantiated per EL, and can only be managed using a packet of that EL, therefore a datagram will frequently need to contain multiple packets to manage the resources of different ELs. We can thus view datagram construction as a process of determining if an EL needs to produce a packet for each EL, and concatenating the resulting packets. The following EL-specific resources exist: - The crypto stream, a bidirectional byte stream abstraction provided to the handshake layer. There is one crypto stream for each of the Initial, Handshake and 1-RTT ELs. (`CRYPTO` frames are prohibited in 0-RTT packets, which is to say the 0-RTT EL has no crypto stream of its own.) - Packet number spaces and acknowledgements. The 0-RTT and 1-RTT ELs share a PN space, but Initial and Handshake ELs both have their own PN spaces. Thus, Initial packets can only be acknowledged using an `ACK` frame sent in an Initial packet, etc. Thus, a fully generalised datagram construction methodology looks like this: - Let E be the set of ELs which are not discarded and for which `pending(el)` is true, where `pending()` is a predicate function determining if the EL has data to send. - Determine if we are limited by anti-amplification restrictions. (Not relevant for MVP since this is only needed on the server side.) - For each EL in E, construct a packet bearing in mind the Remaining CMPPL and append it to the datagram. For the Initial EL, we attach a token if we have been given one. If Initial is in E, the total length of the resulting datagram must be at least 1200, but it is up to us to which packets of which ELs in E we add padding to. - Send the datagram. ### TX Key Update The TX packetiser decides when to tell the QRL to initiate a TX-side key update. It decides this using information provided by the QRL. ### Restricting packet sizes Two factors impact the size of packets that can be sent: * The maximum datagram payload length (MDPL) * Congestion control The MDPL limits the size of an entire datagram, whereas congestion control limits how much data can be in flight at any given time, which may cause a lower limit to be imposed on a given packet. ### Stateless Reset Refer to [RFC 9000 10.3 Stateless Reset]. It's entirely reasonable for the state machine to send this directly and immediately if required. [RFC 9000 2.3 Stream Prioritization]: https://datatracker.ietf.org/doc/html/rfc9000#section-2.3 [RFC 9000 4.1 Data Flow Control]: https://datatracker.ietf.org/doc/html/rfc9000#section-4.1 [RFC 9000 10.3 Stateless Reset]: https://datatracker.ietf.org/doc/html/rfc9000#section-10.3 [RFC 9000 12.2 Coalescing Packets]: https://datatracker.ietf.org/doc/html/rfc9000#section-12.2 [RFC 9000 12.4 Frames and Frame Types]: https://datatracker.ietf.org/doc/html/rfc9000#section-12.4 [RFC 9000 13.3 Retransmission of Information]: https://datatracker.ietf.org/doc/html/rfc9000#section-13.3 [RFC 9000 17.1 Packet Formats]: https://datatracker.ietf.org/doc/html/rfc9000#section-17 [RFC 9000 17.2.1 Version Negotiation Packet]: https://datatracker.ietf.org/doc/html/rfc9000#section-17.2.1 [RFC 9000 17.2.2 Initial Packet]: https://datatracker.ietf.org/doc/html/rfc9000#section-17.2.2 [RFC 9000 17.2.3 0-RTT]: https://datatracker.ietf.org/doc/html/rfc9000#section-17.2.3 [RFC 9000 17.2.4 Handshake Packet]: https://datatracker.ietf.org/doc/html/rfc9000#section-17.2.4 [RFC 9000 17.2.5 Retry Packet]: https://datatracker.ietf.org/doc/html/rfc9000#section-17.2.5 [RFC 9000 17.3.1 1-RTT]: https://datatracker.ietf.org/doc/html/rfc9000#section-17.3.1 [RFC 9002]: https://datatracker.ietf.org/doc/html/rfc9002