Every CrossByte server bounds what one peer can make it hold, and every limit is on by default. Each limit belongs to the object it is set on: one server, or one client, changing its limits changes no other in the process. Each member also says what it bounds in its own documentation.
Durations are in seconds unless a table says otherwise, and sizes in bytes: a KiB is 1,024 bytes and a MiB 1,048,576.
None of these limits bounds how often one address connects: a connection that arrives and leaves at once passes all
of them. A server open to the internet keeps a RateLimiter in admit, which is asked before any work is done for a
connection:
import crossbyte.net.RateLimiter;
import crossbyte.net.ServerWebSocket;
var server = new ServerWebSocket();
// A burst of 20 connections from one address, then one every 3 s.
var joins = new RateLimiter(20, 60);
server.admit = (address, _) -> joins.tryAcquire(RateLimiter.addressKey(address));RateLimiter.addressKey keys an IPv6 client by its /64, which is what one subscriber is given.
Behind a proxy or a load balancer a connection comes from the proxy. TrustedProxies reads the client's address from
Forwarded or X-Forwarded-For, believing only what trusted proxies wrote, as Express's trust proxy does:
import crossbyte.net.RateLimiter;
import crossbyte.net.TrustedProxies;
var proxies = new TrustedProxies(["loopback", "10.0.0.0/8"]);
var logins = new RateLimiter(5, 60);
function mayTry(peer:String, forwarded:Null<String>, forwardedFor:Null<String>):Bool {
return logins.tryAcquire(RateLimiter.addressKey(proxies.clientAddress(peer, forwarded, forwardedFor)));
}A limiter keeps its buckets in memory, which is what one long-running server wants. Where several processes serve the
requests (PHP, which starts one for each, or a server run behind a load balancer), give each limiter the same
RateLimitStore and they limit together. KVRateLimitStore keeps the buckets in a key/value store, SQLite on PHP and
natively:
import crossbyte.db.kv.KVRateLimitStore;
import crossbyte.db.kv.SQLiteKVStore;
import crossbyte.net.RateLimiter;
var shared = new RateLimiter(5, 60, new KVRateLimitStore(SQLiteKVStore.open("data/limits.db", {table: "rate_limits"})));consume says what a 429 Too Many Requests says: limit, remaining, retryAfter and resetAfter, for the
RateLimit and Retry-After headers.
A ServerSocket, and a NetHost built on one, and the HTTPServer and ServerWebSocket built on it, hold each of
these by default:
| On the server | Default | Bounds |
|---|---|---|
maxConnections |
10,000 | connections open at once; one more is closed as it is accepted, and counted in refusedConnections |
maxPendingHandshakes |
256 | TLS handshakes under way; more wait in the kernel's queue |
maxPendingHandshakesPerAddress |
16 | one address's share of those, once half are taken; past it a connection is closed as it is accepted |
maxConnectionsPerAddress |
none | connections one address holds open at once, at all times; past it a connection is closed as it is accepted |
handshakeTimeout |
10 s | each TLS handshake, from accept |
maxAcceptsPerTick |
64 | connections taken from the queue in one wake |
receiveBufferSize, sendBufferSize |
the system's | the kernel's buffers for each connection |
Each Socket it accepts holds at most maxInputBufferSize (16 MiB) unread and maxOutputBufferSize unsent (no
limit by default; HTTP and WebSocket servers set 8 MiB).
A raw connection refused at maxConnections is accepted and closed at once, so its peer reads an end of stream, or
a reset if it had sent something. Node's net.Server.maxConnections does the same: a server's places free up on the
scale of sessions, so a client left queued in the kernel would wait without a word. A ServerWebSocket answers the
upgrade with 503 instead, and an HTTPServer (its limit from HTTPServerConfig.maxConnections) answers the request.
A NetHost's maxConnections is its server's, and a reliable UDP host counts its sessions itself.
maxConnectionsPerAddress is nginx's limit_conn and HAProxy's src_conn_cur: one address may hold that many
connections open, and one more is closed as it is accepted, before any TLS work, and counted in refusedConnections.
An address is counted as RateLimiter.addressKey keys it, so an IPv6 client is counted by its /64. Many real clients
can share an address (a carrier's NAT, an office), so it is off unless set; set it before listen(). A
ServerWebSocket counts its sessions against it and an HTTPServer its connections.
// Given port:Int.
var server = new crossbyte.net.ServerSocket();
// A player's client holds one connection; a household behind one address a few.
server.maxConnectionsPerAddress = 8;
server.addEventListener(crossbyte.events.ServerSocketConnectEvent.CONNECT, e -> {});
server.bind(port, "0.0.0.0");
server.listen();Behind a proxy that forwards connections before their client has spoken, every connection has the proxy's address:
set maxPendingHandshakesPerAddress and maxConnectionsPerAddress to 0 there, and bound each client at the proxy.
Each connection is a descriptor, and a process may hold only so many (RLIMIT_NOFILE on Linux and macOS). Natively,
CrossByte raises the soft limit to the hard one as the process starts, as Go and the JVM do: a shell's soft limit of
1,024 would otherwise stop a server near a thousand connections, whatever the hard limit allows. Where the limit is
still below 4,096, the first server to listen says so, once. -D crossbyte_keep_nofile leaves the limit alone.
Past the limit an accept fails. On Linux the connection waits in the kernel's queue; macOS drops it instead, so its
client finds it closed. The server sets its listener aside for 5 ms, then twice as long after each failure that
follows, at most a second (Go's net/http schedule, and what libuv, Netty and nginx do), and takes a waiting
connection once a descriptor frees. acceptFailures counts the failures, and the first of a run is reported as an
ioError.
A socket reads what arrives into a buffer of its own, and holds at most maxInputBufferSize of it unread: 16 MiB by
default, twice the largest message any CrossByte transport takes. At the limit it stops reading
(inputOverflowPolicy = PAUSE, the default) until the application reads, and what the peer sends waits in the
kernel until TCP's window holds the peer back, so nothing is lost. With CLOSE, it closes the connection as soon as
a byte arrives past the limit. HTTP, WebSocket, RPC and NetConnection read as data arrives and never reach it.
A game server that reads one message a tick can set it to a few of its messages. Keep it above the largest message
the application waits for whole: under PAUSE, an application that waits for more than the limit before it reads
anything waits for good.
import crossbyte.events.ServerSocketConnectEvent;
import crossbyte.net.ServerSocket;
var server = new ServerSocket();
server.addEventListener(ServerSocketConnectEvent.CONNECT, e -> {
e.socket.maxInputBufferSize = 64 * 1024; // a few of this game's messages
});What is written and that the kernel will not take yet waits in the socket's output buffer, and bytesPending says
how much. Past maxOutputBufferSize the connection is closed (outputOverflowPolicy = CLOSE) with an ioError whose
errorId is IOErrorEvent.OVERFLOW_ERROR_ID, which a NetConnection reports as Reason.Overflow; a WebSocket
session's limit, a reliable UDP session's queue for its window and a local connection's maxQueuedBytes end the same
way.
A NetConnection says so earlier: outputHighWater and outputLowWater, with onOutputHigh and onOutputDrained,
as Netty's write buffer water marks and Node's highWaterMark and drain do. A server sending each client a stream of
updates can skip them for a client that has fallen behind and send it the whole state once it has caught up:
// Given connection:crossbyte.net.NetConnection, sendDelta:Void->Void, sendSnapshot:Void->Void.
var behind = false;
connection.outputHighWater = 256 * 1024;
connection.outputLowWater = 32 * 1024;
connection.onOutputHigh = () -> behind = true;
connection.onOutputDrained = () -> {
behind = false;
sendSnapshot();
};
// Each tick:
if (!behind) {
sendDelta();
}Over TCP the marks are checked once a pass's sends have gone to the system, so a burst the system takes whole is not a backlog; over WebSocket, reliable UDP and local IPC as each send returns. With no marks set a send costs nothing more.
An idle connection a ServerSocket accepted, with a listener for what arrives, holds about 1.3 KB of heap natively
and 1.6 KB on the jvm (a client about 2 KB), plus the kernel's buffers. Its buffers keep their storage while the
connection is busy, so a message read or written allocates nothing, and let it go once the connection has read and
written nothing for one of its runtime's sweeps, every five seconds: after one 16 KB message each way, a connection
holds 1.5 KB once quiet (1.7 KB on the jvm). A buffer grown past 64 KB lets go as soon
as it empties: natively and on the jvm its storage goes to a pool its runtime keeps, by size, which the next buffer
to grow that far takes from, so a connection sending or receiving a large burst each pass reuses the same storage
rather than allocating it again (over 20 seconds of 1 MB bursts, about 6 KB for each megabyte, where growing anew
took 2.4 MB). The pool's sizes are a quarter apart (64, 80, 96, 112 and 128 KB, and so on to 32 MB), so a buffer
holds at most a quarter more than it needs, and a buffer takes at once the size its last burst needed. The pool lets go
of what nobody has taken for ten to fifteen seconds. A WebSocket session's buffers, and a message past 64 KB, do
the same. On Node a socket keeps its buffers' storage.
The built-in poll walks every connection on every pass of the runtime: natively on Linux about 77 ns a connection,
so 10,000 idle connections cost a pass 0.77 ms, and a POLL runtime at rest 5% of a core at 12 ticks a second and
17% at 60 (in WSL, 4 CPUs). A busy server passes once per batch of arrivals, so the same 10,000 with traffic spread
over 1,000 wakes a second spend most of a core in the poll alone. Past about 10% of a core (10,000 connections at
130 passes a second, or 2,000 at 650), the crossbyte-libuv backend, whose pass costs what is ready rather than what
is open, starts to pay. On Windows, 8,000 idle connections cost 3% of a core on the built-in poll (select) and
nothing measurable on libuv.
Much of what a connection costs is in the kernel: what has arrived and not been read, and what has been written and
not acknowledged. Left alone, the system grows both with use (Linux to 6 MB received and 4 MB sent a connection,
Windows by the connection's bandwidth), so a peer that sends and does not read, or reads nothing, makes the kernel
hold that much for it. receiveBufferSize and sendBufferSize fix them, on a Socket before it connects or on a
ServerSocket for every connection it accepts (natively and on the jvm):
var server = new ServerSocket();
server.receiveBufferSize = 64 * 1024; // before listen(): the window starts there
server.sendBufferSize = 64 * 1024;What the system will not take waits in the socket's own output buffer instead, where bytesPending counts it and
maxOutputBufferSize bounds it. A smaller buffer is cheaper and slower over a long path: a connection moves at most
one buffer per round trip.
A ServerWebSocket holds the TCP limits above, and these:
| On the server | Default | Bounds |
|---|---|---|
maxPendingHandshakes |
256 | connections still upgrading, TLS and the HTTP upgrade together; more wait in the kernel's queue |
maxPendingHandshakesPerAddress |
16 | one address's share of those, once half are taken; past it a connection is closed as it is accepted |
handshakeTimeout |
10 s | each upgrade, from accept |
maxHeaderSize |
16 KiB | the upgrade request's head, answered 431 past it |
maxConnections |
10,000 | sessions open at once, answered 503 past it |
maxConnectionsPerAddress |
none | sessions one address holds, from accept to close; past it a connection is closed as it is accepted |
maxMessageSize |
1 MiB | one message arriving, its frames together, refused on a frame's header with 1009 |
maxOutputBufferSize |
8 MiB | what waits for a peer that is not reading, closed with 1011 past it |
idleTimeout |
60 s | silence from the peer, with a ping sent every pingInterval (30 s) meanwhile |
closeTimeout |
5 s | the closing handshake |
A peer's pings are answered one at a time, the newest only (RFC 6455 lets a pong answer only the latest ping), so a
peer that pings and reads nothing is owed one pong however many it sends. refusedConnections counts the
connections the connection limits closed, and handshakeFailures the upgrades that never finished.
What a session holds. An idle session a server accepted holds about 4 KB of heap natively and 3 KB on the jvm,
plus the kernel's socket buffers; a client about the same. Its upgrade request is kept as the head it arrived as,
and its headers are read from that again if they are asked for after the session opened. A session that has heard
nothing and sent nothing for one pingInterval (30 s) lets go of what its buffers held for earlier messages: after
one 16 KB message each way, a session holds 4.6 KB once quiet. Output that waited for a slow peer
goes as soon as it drains.
What a message costs. Natively, a message sent or received allocates nothing but the text a listener asks for,
which is a String of its own and safe to keep; a listener that reads data allocates nothing. Its bytes and its
event are the session's own, filled again for the next message, and valid only during the listener's call (see
Event). A message past 64 KB is read into storage from the pool its runtime keeps for large buffers (see "What a
connection holds"), given back once the listener returns. A client draws its masking keys from a pool of random
bytes, 8 KB at a time.
One message to many sessions. PreparedMessage makes a message ready once, encoded and framed as a server sends
it, and each session copies the frames into what it sends, where a sendText per session would encode, frame and
copy it again for each:
// Given server:ServerWebSocket, room:Array<crossbyte.net.WebSocket>.
import crossbyte.net.PreparedMessage;
var update = PreparedMessage.text('{"type":"move","x":12,"y":40}');
server.broadcast(update); // every session the server has open
server.broadcast(update, room); // or the ones the application choseWho receives (rooms, topics, areas of interest) stays the application's choice. Preparing copies the bytes, so the
buffer it was made from is the caller's again at once, a message event's payload included; the message itself never
changes, and may be kept and sent again from any runtime. Made with compress, it also holds a compressed form,
compressed once, for the sessions that agreed to permessage-deflate. A secure session still encrypts on its own, so
TLS shares only the framing; and a client masks each frame with a key of its own, so a client's sendPrepared
shares the encoding and frames it itself. Each session still writes once a pass, however many messages it was sent
in it. A broadcast throws nothing for one session: one not open, or closing, is passed over, and one past its
maxOutputBufferSize is closed with 1011.
What a server you call can cost you is bounded per request, on the URLRequest, so one caller's settings never
reach another's requests:
URLRequest |
Default | Bounds |
|---|---|---|
readTimeout |
30 s | time with nothing arriving; natively, on the jvm and the interpreter also the wait for a load thread past URLLoader.maxConcurrentLoads |
responseHeaderTimeout |
300 s | the response's head, from the request having gone; bytes trickling in do not move it |
loadTimeout |
none | the whole load, from load() to COMPLETE |
maxBodySize |
64 MiB | the body on the wire; a larger Content-Length is refused before it is read |
maxDecompressedSize |
64 MiB | what the body decodes to |
maxResponseHeaderSize |
64 KiB | the status line and header fields, 1xx responses included |
maxRedirects |
10 | redirects followed before the load fails |
0 or less lifts any of them. An idle timeout alone does not bound a request: a server sending a byte at a time
resets it with every byte, so give a request to a server you do not trust a loadTimeout. A custom HTTPBackend
reads the same limits from its HTTPRequestContext; the loader cancels a request past its loadTimeout through
the context's cancelToken.
Names. A socket's connect by name, and every client request, looks the name up on one of four threads the process keeps for it. A system lookup cannot be stopped, so that is the most a wedged resolver can hold. A caller waits 30 s for the answer at most (less within a request's idle timeout, and a cancel ends the wait at once). An answer is kept for 30 s and a failure for 5, and callers asking for a name already being looked up share the lookup.
An HTTPServer holds the TCP limits above, with its own maxConnections in HTTPServerConfig.
HTTPServerConfig.phpMaxResponseSize (8 MiB) bounds a PHP script's response, whose CGI header block is held to
64 KiB and 100 lines besides. phpMaxExchanges (64) bounds the requests a runtime has with its PHP backend at once;
more wait their turn within phpTimeout, and past 1,024 waiting a request is refused.
A ReliableDatagramServerSocket opens a session for a CONNECT, on the strength of a source address any sender can
write. What that costs, and what it sends to addresses nobody has proved, is bounded:
| On the server | Default | Bounds |
|---|---|---|
maxPendingConnections |
256 | sessions waiting to finish their handshakes |
joinValidation |
UNDER_PRESSURE |
when a join must show it receives at its address before a session is opened |
joinValidationThreshold |
64 | pending sessions past which UNDER_PRESSURE validates |
maxResetsPerSecond (static) |
1,000 | FINs every server in the process sends, together, to addresses with no session |
allowRebind |
off | whether a session follows its player to a new address |
Each session holds at most ReliableDatagramSocket.maxOutputBufferSize (256 KiB) waiting for its congestion window,
past which it is ended with an ioError saying why. A game that sends more at once (a level, one large reliable
message) raises the limit, or sets it to 0 for none. reliable-udp.md explains the handshake and
what a session and a message cost.
A PeerConnectionHost creates nothing for a datagram until it has proved who sent it: a STUN check is answered only
once its MESSAGE-INTEGRITY checks against a connection's credentials, and DTLS is routed by an address ICE has
proved. So there is no reflection and no half-open table to fill. What follows bounds a peer that has been through
signalling, and can send whatever it likes once connected, sharing a runtime with every other peer on it.
| On the connection | Default | Bounds |
|---|---|---|
readyTimeout |
30 s | connect until the whole stack is up, and an ICE restart; 0 for no deadline |
maxPeerChannels |
512 | channels the peer opened that are open at once, what libwebrtc's 1,024 streams give one side; past it an OPEN is refused (no ACK, the stream reset) and counted in refusedChannels; 0 for no limit |
maxLabelSize |
1 KiB | the label, and the protocol, of a channel the peer opens, in UTF-8 bytes; refused past it likewise; 0 for the wire's 65,535 |
IceAgent.MAX_REMOTE_CANDIDATES |
64 | the peer's candidates, advertised and learned from where its checks arrive; past it a check from a new place goes unanswered |
IceAgent.MAX_LEARNED_LOCAL_CANDIDATES |
64 | places the peer's answers say it saw this end's checks come from; never paired, since a check leaves from their base |
Under the data channels, an SCTP association holds at most 2 MiB the application has not been given (the window it advertises), 1 MiB of one message being reassembled in at most 2,048 pieces, and 18,432 pieces in all: fragments and whole messages waiting their turn, every stream together, the most an honest peer can make it hold. Past either total it gives back half of what it holds, dropping unfinished messages.
What one packet can make it do is bounded by the packet. One SACK is read a packet, and 256 of its gap blocks. A
FORWARD TSN finds the streams it gives up on by the TSN each starts at, and walks a stream by the shorter of the
range it names and what the stream holds. HEARTBEATs and stream-reset requests draw one packet of answers. No more
than 16,384 fragments go past the peer's cumulative acknowledgement; the rest wait, counted in bufferedAmount.