server: bound unauthenticated connections in number and in time (#16237)

A connection that never logs in costs whatever its transport costs, for as
long as it keeps itself alive: the only limit was the 30s idle timeout, which
any message resets. Nothing bounded how many such connections one machine
holds, on any transport. The shape sshd_config answers with LoginGraceTime
and MaxStartups.

Every connection is admitted among the unauthorized ones before its identity
handshake, in create_tcp_connection, and holds that place until it
authorizes or ends: the count of live places is the bound, not a ledger
beside the connections: the resource bound. One address may hold sixteen, a
quarter of the room; a further connection from it is refused before the
handshake. That share is a fairness cap against the cheapest flood, one host
with one address, not a security boundary: any pool of addresses passes it,
and the global limit is what holds. With 64 held in all, a further arrival
is refused too, and the oldest connection is told to go, unless one is on
its way out already: the handshake is raced against that eviction and ends
at once, and the session loop has it as a branch of its select, so the place
opens as soon as the connection has actually gone and not on a timer tick.
The newcomer is not let in on a place still occupied; the controller retries
on its own with backoff, and by then the place is free. At most one
connection is ever on its way out, so a burst of refused arrivals clears no
more room than a single one, and the retry that takes the freed place counts
against its address's share: one address turns out at most as many
connections as it may hold.

One deadline, from the moment the connection starts, a branch of the session
loop's select rather than a check on the TestDelay tick: a connection not
authorized after 180s is closed, however alive it keeps itself, a wrong
password, a pending 2FA, an accept prompt or an admin-terminal credential
prompt left unanswered. The controller reconnects on its own and the prompt
comes back. It closes with the Timeout reason the idle path uses, and that
path still ends a connection that says nothing for 30s. There is no shorter
deadline for the first login request: an admin-terminal controller shows
its credential prompt before sending one, and a peer that wanted to dodge
such a deadline would only have to send a login request, so it would bound
nothing.

The peer address is normalized with try_into_v4 before admission, the same
form Connection::start keys the whitelist on, so an IPv4 peer and its
IPv4-mapped IPv6 form are one address and not two shares.

The WebRTC answerer's slot keeps bounding peer connection setup up to the
open data channel; from there this covers it like every other transport.

Tests cover the registry and the live bound: an address over its share is
refused while others are admitted; at the limit the newcomer is refused, the
oldest is told to go, nobody else is while it is on its way out, and its
place frees only when it has; an address at the limit turns out no more
connections than its share and is then refused without evicting anyone; and
with the limit held by 64 connections stalled in the handshake, one more
arrival is refused while the oldest handshake ends at once and only then is
there a place again.


Claude-Session: https://claude.ai/code/session_019aokqJuhjvB3kijXtAg5Ns

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>

src/server.rs

@@ -118,6 +118,15 @@ pub struct Server {
 pub type ServerPtr = Arc<RwLock<Server>>;
 pub type ServerPtrWeak = Weak<RwLock<Server>>;
 
+#[cfg(test)]
+pub fn new_for_test() -> ServerPtr {
+    Arc::new(RwLock::new(Server {
+        connections: HashMap::new(),
+        services: HashMap::new(),
+        id_count: 1000,
+    }))
+}
+
 pub fn new() -> ServerPtr {
     let mut server = Server {
         connections: HashMap::new(),
@@ -204,7 +213,48 @@ pub async fn create_tcp_connection(
     meta: ConnectionMeta,
 ) -> ResultType<()> {
     let mut stream = stream;
+    // The address the connection layer keys on, whitelist and admission alike.
+    let addr = hbb_common::try_into_v4(addr);
     let id = server.write().unwrap().get_new_id();
+    // Admitted before the identity handshake, so a peer that stalls in it, or after it without
+    // logging in, holds its place the whole time; an address over its share is turned away.
+    let Some(unauthorized) = admit_unauthorized(id, addr.ip()) else {
+        bail!("too many unauthenticated connections from {}", addr.ip());
+    };
+    tokio::select! {
+        handshake = identity_handshake(&mut stream, secure) => handshake?,
+        _ = unauthorized.evicted() => {
+            bail!("evicted to make room for a newer unauthenticated connection");
+        }
+    }
+
+    #[cfg(target_os = "macos")]
+    {
+        use std::process::Command;
+        if let Ok(task) = Command::new("/usr/bin/caffeinate")
+            .arg("-u")
+            .arg("-t 5")
+            .spawn()
+        {
+            super::CHILD_PROCESS.lock().unwrap().push(task);
+        }
+        log::info!("wake up macos");
+    }
+    Connection::start(
+        addr,
+        stream,
+        id,
+        Arc::downgrade(&server),
+        meta,
+        unauthorized,
+    )
+    .await;
+    Ok(())
+}
+
+/// Our signed identity goes out and, when `secure`, the controller's reply keys `stream`.
+/// Separate so it can be raced against the connection's eviction.
+async fn identity_handshake(stream: &mut Stream, secure: bool) -> ResultType<()> {
     let (sk, pk) = Config::get_key_pair();
     if secure && pk.len() == sign::PUBLICKEYBYTES && sk.len() == sign::SECRETKEYBYTES {
         let mut sk_ = [0u8; sign::SECRETKEYBYTES];
@@ -267,19 +317,6 @@ pub async fn create_tcp_connection(
         }
     }
 
-    #[cfg(target_os = "macos")]
-    {
-        use std::process::Command;
-        if let Ok(task) = Command::new("/usr/bin/caffeinate")
-            .arg("-u")
-            .arg("-t 5")
-            .spawn()
-        {
-            super::CHILD_PROCESS.lock().unwrap().push(task);
-        }
-        log::info!("wake up macos");
-    }
-    Connection::start(addr, stream, id, Arc::downgrade(&server), meta).await;
     Ok(())
 }