Fix deadlock between thread-safe inproc senders

[Cyridian]

Two thread-safe sockets that send to each other at the same time can deadlock.

MailboxSafe.Send takes the owning socket's lock to write a command. But a sender flushing a pipe already holds its own socket's lock, and when the flush wakes a passive reader on the peer it sends ActivateRead into the peer's MailboxSafe - which takes the peer's socket lock. So with a ServerSocket and a ClientSocket connected over inproc and both sending at once, each thread holds its own socket's lock and waits for the other's.

The fix gives MailboxSafe a private lock for senders, the same way IOThreadMailbox already works: senders never touch the socket's lock, so the cycle can't form.

The receive side needs a little care, because a receiver that has to wait used to wait on the socket's lock itself (Monitor.Wait(m_sync)). Now it:

• re-reads the command pipe under the private lock, so a command flushed just before it got there isn't missed;
• releases the socket lock at every recursion depth (the socket lock can be held more than once, e.g. BindRandomPort calls Bind), waits on the private lock, and then takes the socket lock back after leaving the private lock - so the socket lock is never acquired while the private lock is held;
• restores every level even if the thread is interrupted while doing so, and re-raises the interrupt afterwards, so the caller's Unlock still matches.

Close passes through both locks so no sender is still mid-send.

The new test InprocConcurrentSendsInBothDirectionsDoNotDeadlock has a Server and a Client send 20,000 messages to each other at the same time while draining. On current master it hangs every time I run it (3 out of 3); with the fix it passes reliably (200 out of 200). If it ever regresses, the test fails its Join and may then hang during disposal, so the CI hang timeout is what will catch it.

[drewnoakes]

Thank you!

[drewnoakes]

https://github.com/zeromq/netmq/releases/tag/4.0.4.4