Skip to content

hub: prevent concurrent duplicate outgoing connections to the same SKI#85

Merged
uhl merged 5 commits into
enbility:devfrom
andig:fix/double-connection-outgoing-dedup
Jul 6, 2026
Merged

hub: prevent concurrent duplicate outgoing connections to the same SKI#85
uhl merged 5 commits into
enbility:devfrom
andig:fix/double-connection-outgoing-dedup

Conversation

@andig

@andig andig commented Jun 15, 2026

Copy link
Copy Markdown
Contributor

connectFoundService only dedupes outgoing dials via isSkiConnected, which becomes true after registerConnection (i.e. after the websocket dial + SHIP handshake). Two concurrent outgoing dials to the same SKI can both pass that check, both establish, and then keepThisConnection tears one down mid-discovery. Some remotes then stop answering measurement reads on the surviving connection until the whole service is restarted.

Reserve the SKI for the duration of the dial via a new connectionsInitiating set (guarded by muxCon), so a concurrent outgoing dial returns early instead of racing to a second handshake. The incoming-vs-outgoing case is still handled by keepThisConnection.

This is a real-live issue observed as a stalled EEBus meter on a Vaillant heatpump in evcc-io/evcc#30889. PR is done by Claude but lgtm.

Comment thread hub/hub_connections_client.go Outdated
@uhl uhl self-requested a review June 17, 2026 05:45
@coveralls

coveralls commented Jun 29, 2026

Copy link
Copy Markdown

Coverage Status

coverage: 92.972% (+0.02%) from 92.952% — andig:fix/double-connection-outgoing-dedup into enbility:dev

Comment thread hub/hub_connections_client.go Outdated
Comment thread hub/hub_connections_client.go Outdated
uhl added a commit that referenced this pull request Jul 6, 2026
Fix evcc-io/evcc#31465

Found while chasing an intermittent double-connection related test hang
in evcc (evcc-io/evcc#31493) that reproduces on top of #85's dedup fix.

`sendSpineData` only calls `CloseConnection` when
`IsDataConnectionClosed()` already reports the connection closed:

```go
if isClosed, err := c.dataWriter.IsDataConnectionClosed(); isClosed {
    c.CloseConnection(false, 0, "")
    return err
}
...
err = c.dataWriter.WriteMessageToWebsocketConnection(shipMsg)
if err != nil {
    logging.Log().Debug("error sending message: ", err)
    return err
}
```

If the actual websocket write fails for any other reason (e.g. the
remote already closed the connection before this side's writer noticed),
the error is logged and swallowed, but the connection is never torn
down. The `ShipConnection` stays registered as the current writer for
that SKI, so every subsequent send silently fails forever and the SKI is
never freed for a fresh connection attempt.

This surfaced concretely while stress-testing a double-connection race
scenario: after one side's connection was implicitly invalidated, the
other kept retrying sends against it every couple of seconds
indefinitely ("Error sending spine message: connection closed for remote
SKI ...", repeating), and no reconnect was ever attempted.

Fix: also call `CloseConnection(false, 0, "")` when the write itself
errors, mirroring the existing `IsDataConnectionClosed` branch. Updated
`TestSendSpineData_WriteError` to expect the resulting
`HandleConnectionClosed` call.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
@uhl

uhl commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

Sorry for the churn here — the nil-check removal was on me, I asked for it thinking NewHub covered every construction path, but CI now shows it doesn't.

The latest run against ec48b663 is red across the board: panic: assignment to entry in nil map at hub/hub_connections_client.go:165 (the h.connectionsInitiating[ski] = true line), which takes down the entire hub package test binary. The same commit is the one that removed the if h.connectionsInitiating == nil { ... } guard added in 3e2cff27, which is where CI was last green.

CI history confirms the trigger:

  • 3e2cff27 (nil-check present): both workflows green
  • ec48b663 (nil-check removed): both workflows red

Some test path exercises connectFoundService on a *Hub where connectionsInitiating is nil — the most likely culprit is the &Hub{} literal in hub/hub_merge_test.go:17 (newMergeHub), which deliberately bypasses NewHub because the merge primitive only needs muxReg + remoteServices. There may be similar patterns elsewhere.

Two options to unstick this:

  1. Restore the defensive nil-init from 3e2cff27 inside connectFoundService. Simplest, and matches what CI proved works.
  2. Fix the &Hub{} literal(s) in test code to include connectionsInitiating: make(map[string]bool). Cleaner separation, but future &Hub{} literals will need to remember the field.

Either works for me — happy to defer to your preference. Sorry again for sending you down this path.

@andig

andig commented Jul 6, 2026

Copy link
Copy Markdown
Contributor Author

I'll push 2, its consistent with the rest of the codebase.

andig added 2 commits July 6, 2026 14:42
…ating

ReportMdnsEntries schedules a delayed timer that can reach
connectFoundService asynchronously, which writes to
connectionsInitiating. The bare &Hub{} literals in hub_mdns_test.go
predate that field and left it nil, causing a flaky
"assignment to entry in nil map" panic that took down the whole
package's test binary.
Replaces the ten repeated &Hub{...} literals with a single helper so
future fields (like the connectionsInitiating fix) only need to be
added once.
@andig

andig commented Jul 6, 2026

Copy link
Copy Markdown
Contributor Author

Actually, did best of both ;)

@uhl uhl self-requested a review July 6, 2026 13:39

@uhl uhl left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good now

@uhl uhl merged commit 3abd41d into enbility:dev Jul 6, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants