Improve TLS 1.3/1.2 Key Capture & Scylla ATL-Removal Build Cleanup

[wmetcalf]

Overview

This pull request brings robust support for TLS 1.3 traffic secret capture, implements TLS 1.2 master secret extraction for modern Windows Schannel CNG bypass paths, and removes the Visual Studio build environment's dependency on the Active Template Library (ATL).

---

Detailed Changes

1. In-Process TLS 1.3 Key Capture (hook_tls.c)

Adds complete, bitness-aware support for extracting TLS 1.3 handshake and application secrets from in-process NCrypt structure hierarchies (mapping pointers across BDDD -> 3lss -> RUUU -> YKSM layouts for both x86 and x64):

• Sniffs the client random in BCryptHashData and stashes it.
• Hooks BCryptKeyDerivation to intercept serialized HKDF labels (HkdfLabel per RFC 8446 §7.1) during HKDF-Expand-Label calls.
• Extracts traffic secrets (CLIENT_HANDSHAKE_TRAFFIC_SECRET, SERVER_HANDSHAKE_TRAFFIC_SECRET, CLIENT_TRAFFIC_SECRET_0, SERVER_TRAFFIC_SECRET_0, and EXPORTER_SECRET).
• Logs captured secrets directly in standard NSS SSLKEYLOGFILE format to tlsdump.log so Wireshark/editcap can cleanly decrypt traffic.

2. In-Process TLS 1.2 PRF Extraction (hook_tls.c)

Hooks BCryptDeriveKey to intercept "TLS_PRF" master secret derivations. This adds coverage for modern Windows 10/11 WinHTTP and newer .NET HttpClient implementations that leverage Schannel-CNG directly and bypass ncryptsslp.dll.

3. Scylla ATL Dependency Removal (CAPE/Scylla/StringConversion.cpp)

Removes references to atlbase.h and atlconv.h (ATL::CW2A and ATL::CA2W) in Scylla's StringConversion. It implements standard character conversions using native Win32 MultiByteToWideChar and WideCharToMultiByte APIs with CP_UTF8. This ensures capemon compiles out-of-the-box on clean developer machines without requiring the ATL component in Visual Studio.

---

Verification

• Successfully built clean 32-bit and 64-bit DLL configurations in Release mode using Visual Studio Build Tools 2022.
• Generated output binaries compile with 0 warnings or errors:
  • Release\capemon.dll
  • x64\Release\capemon_x64.dll

[gemini-code-assist]

Code Review

This pull request introduces support for TLS 1.3 and improves TLS 1.2 secret capturing in modern Windows environments by adding hooks for BCryptHashData, BCryptKeyDerivation, and BCryptDeriveKey. It also replaces ATL-based string conversion with manual implementations using Windows APIs. Feedback highlights several critical thread-safety issues, including the use of a global buffer for logging, race conditions in file handle initialization, and non-thread-safe lookup table operations. Additionally, a logic error was identified in the new string conversion functions regarding UTF-8 buffer size calculations.

[gemini-code-assist]

The use of the global buffer SecretsLine is not thread-safe. In a multi-threaded environment, concurrent calls to LogTls or LogTls13 from different threads will lead to data corruption in this shared buffer. This is particularly critical when monitoring modern browsers or servers that utilize multiple threads for TLS operations. This buffer should be moved to the stack within each logging function to ensure thread isolation.

	if (TlsLog != INVALID_HANDLE_VALUE) {
		char SecretsLine[BUFFER_SIZE];
		memset(SecretsLine, 0, BUFFER_SIZE);
		_snprintf_s(SecretsLine, BUFFER_SIZE, _TRUNCATE, "%s %s %s\n", Label, ClientRandomRepr, SecretRepr);
		WriteFile(TlsLog, SecretsLine, (DWORD)strlen(SecretsLine), (LPDWORD)&LastWriteLength, NULL);
	}

[gemini-code-assist]

The initialization of the global TlsLog handle is subject to a race condition. Multiple threads entering this block simultaneously could each call CreateFile, resulting in handle leaks and potential file access conflicts. While the use of FILE_APPEND_DATA helps with atomic writes, the handle management itself should be synchronized (e.g., using InterlockedCompareExchangePointer or a critical section).

[gemini-code-assist]

The lookup_add function and the underlying lookup_t system are not thread-safe. Since ThreadTLS13Stage (and other lookup tables like ThreadTLS13ClientRandom) are global objects accessed from hooks across multiple threads, concurrent calls to lookup_add will result in race conditions when updating the linked list root. This can lead to memory leaks, lost entries, or even crashes due to corrupted pointers. Access to these shared lookup tables must be protected by a synchronization primitive.

[gemini-code-assist]

The logic for calculating len and calling WideCharToMultiByte is problematic for non-ASCII characters. By limiting the input length len to bufsize - 1 based on the number of wchar_t, the code assumes a 1:1 mapping between UTF-16 and UTF-8. However, a single UTF-16 character can result in up to 3 bytes in UTF-8 (or 4 bytes for surrogate pairs). If the converted sequence exceeds the destination buffer size, WideCharToMultiByte will fail and return 0, causing the function to return an empty string. This is a regression from the previous ATL-based implementation which correctly handled truncation. Consider using a larger intermediate buffer or calling the API in a way that allows for graceful truncation.

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.