Support for relocation of RIP-relative instructions in `FunctionHook`

[alexrp]

This is quite a difficult problem to solve, but it is possible. Instructions that use RIP-relative displacements have to be rewritten to equivalent sequences using absolute addresses. The tricky part is picking a volatile scratch register to use in the rewritten code that won't conflict with the original code.

[alexrp]

Until this problem is properly solved, we could factor the displacements used in prologue instructions into our code placement. That should be fairly simple to do.

[alexrp]

This feature is a major undertaking, to the point that it will probably be the main theme of a major release. And that's assuming I ever get around to it...

[TechInterMezzo]

Until this problem is properly solved, we could factor the displacements used in prologue instructions into our code placement. That should be fairly simple to do.

I tried to implement your suggestion. I don't know if I did it right but it seems to work for my case.

        if (Environment.Is64BitProcess)
        {
            var baseAddr = (byte*)target + JumpInstructionSize;

            var low = baseAddr + int.MinValue;

            if (low > baseAddr)
                low = null; // Underflow, i.e. the bottom of the address space is reachable.

            var high = baseAddr + int.MaxValue;

            if (high < baseAddr)
                high = (byte*)null - 1; // Overflow, i.e. the top of the address space is reachable.

            foreach (var insn in prologue)
            {
                if (insn.IsIPRelativeMemoryOperand)
                {
                    var dispAddr = (byte*)insn.IPRelativeMemoryAddress + insn.Length;
                    var dispLow = dispAddr + int.MinValue;
                    if (dispLow > dispAddr)
                    {
                        dispLow = null;
                    }
                    var dispHigh = dispAddr + int.MaxValue;
                    if (dispHigh < dispAddr)
                    {
                        dispHigh = (byte*)null - 1;
                    }
                    low = (byte*)Math.Max((nuint)low, (nuint)dispLow);
                    high = (byte*)Math.Min((nuint)high, (nuint)dispHigh);
                }
            }

            placement = CodePlacement.Range(low, high);
        }