Repository navigation
Conversation
PHP code:
```php
<?php
function sum($n) {
$s = 0;
for ($i = 0; $i < $n; $i++) {
$s += $i;
}
return $s;
}
var_dump(sum(10000000));
```
Old x86-64 code:
```asm
TRACE-1$sum$4:
subq $0x30, %rsp
movabsq $EG(jit_trace_num), %rax
movl $1, (%rax)
cmpb $4, 0x58(%r14)
jne jit$$trace_exit_0
cmpb $4, 0x68(%r14)
jne jit$$trace_exit_1
cmpb $4, 0x78(%r14)
jne jit$$trace_exit_2
movq 0x50(%r14), %rax
movq 0x60(%r14), %rcx
movq 0x70(%r14), %rdx
.L1:
cmpq %rax, %rdx
jge jit$$trace_exit_3
movq %rdx, %rsi
addq %rcx, %rsi
jo jit$$trace_exit_4
leaq 1(%rdx), %rdx
movabsq $EG(vm_interrupt), %rcx
movb (%rcx), %cl
testb %cl, %cl
jne jit$$trace_exit_5
movq %rsi, %rcx
jmp .L1
```
Note that this code uses an extra register and 2 extra moves (for %rcx)
to carry the result. This is because the trace exit wants the old
result. But we can avoid the extra register and extra moves if we simply
reconstruct the old value upon exit.
This patch implements that idea, which results in the following new code:
```asm
TRACE-1$sum$4:
subq $0x30, %rsp
movabsq $EG(jit_trace_num), %rax
movl $1, (%rax)
cmpb $4, 0x58(%r14)
jne jit$$trace_exit_0
cmpb $4, 0x68(%r14)
jne jit$$trace_exit_1
cmpb $4, 0x78(%r14)
jne jit$$trace_exit_2
movq 0x50(%r14), %rax
movq 0x60(%r14), %rcx
movq 0x70(%r14), %rdx
jmp .L2
; ... nops for alignment
.L1:
leaq 1(%rdx), %rdx
movabsq $EG(vm_interrupt), %rsi
movb (%rsi), %sil
testb %sil, %sil
jne jit$$trace_exit_5
.L2:
cmpq %rax, %rdx
jge jit$$trace_exit_3
addq %rdx, %rcx
jno .L1
subq %rdx, %rcx
jmp jit$$trace_exit_4
```
Notice how the addition now no longer needs extra moves and how the
result is kept in %rcx all the time. The cold path under `jno` restores
the old value right before jumping to the trace exit.
AWS x86_64 (c6id.metal)
Laravel 12.11.0 demo app - 50 iterations, 50 warmups, 100 requests (sec)
Symfony 2.8.0 demo app - 50 iterations, 50 warmups, 100 requests (sec)
Wordpress 6.9 main page - 50 iterations, 20 warmups, 20 requests (sec)
bench.php - 50 iterations, 20 warmups, 2 requests (sec)
|
dstogov
reviewed
Sep 29, 2026
dstogov
left a comment
dstogov
left a comment
Member
There was a problem hiding this comment.
I see, this makes better code for provided example - increment of induction variable.
However, I'm not sure if this will make improvement in general.
This will emit more cold code, even if register reuse is not desired.
On the other hand this will decries register pressure.
Please check if this will work for "$x = $x + $x". Undoing this might be problematic.
Anyway, I don't care. Merge this if you like.
A better solution would be executing the "roolback" code by de-optimizer.
I know LuaJIT somehow does this (may be deoptimizer interprets the trace) , but I'm not sure if it's possible to do the same with IR.
Member
Author
This seems to work. Checked with the following code and <?php
function sum($n) {
$s = 1;
for ($i = 0; $i < $n; $i++) {
$s = $s + $s;
}
return $s;
}
var_dump(sum(30));This generates the exact same assembly code as before.
This might increase cold code generation, but the non-cold code generation may decrease because of the reduction in register usage can decrease the number of moves too. |
Member
|
Problem with cold code is that it's emitted immediately after the trace, so it still reduces code locality across traces. I agree that we could avoid the cold code entirely by undoing the operation in the deoptimizer. PoC with an LLM: diff --git a/ext/opcache/jit/zend_jit.h b/ext/opcache/jit/zend_jit.h
index c4080d86bc6..b977036ee4c 100644
--- a/ext/opcache/jit/zend_jit.h
+++ b/ext/opcache/jit/zend_jit.h
@@ -191,6 +191,12 @@ void zend_jit_restart(void);
#define ZREG_ZVAL_ADDREF (4<<4)
#define ZREG_THIS (5<<4)
+/* The slot holds the result of the ADD/SUB opline the exit point points to
+ * (in the register, spill slot or VM stack slot described by the other bits).
+ * The deoptimizer reconstructs the old value of op1 by reverting the operation
+ * with op2 of the opline. See zend_jit_math_long_long(). */
+#define ZREG_UNDO_MATH (1<<7)
+
#define ZREG_NONE -1
#endif /* HAVE_JIT_H */
diff --git a/ext/opcache/jit/zend_jit_ir.c b/ext/opcache/jit/zend_jit_ir.c
index 2da9978a00c..64ea2d6ce0a 100644
--- a/ext/opcache/jit/zend_jit_ir.c
+++ b/ext/opcache/jit/zend_jit_ir.c
@@ -821,6 +821,7 @@ void *zend_jit_snapshot_handler(ir_ctx *ctx, ir_ref snapshot_ref, ir_insn *snaps
ir_ref var = i - 2;
ZEND_ASSERT(var < t->exit_info[exit_point].stack_size);
+ uint8_t undo = t->stack_map[t->exit_info[exit_point].stack_offset + var].flags & ZREG_UNDO_MATH;
if (t->stack_map[t->exit_info[exit_point].stack_offset + var].flags == ZREG_ZVAL_COPY) {
ZEND_ASSERT(reg != ZREG_NONE);
if ((exit_flags & ZEND_JIT_EXIT_FIXED)
@@ -854,34 +855,34 @@ void *zend_jit_snapshot_handler(ir_ctx *ctx, ir_ref snapshot_ref, ir_insn *snaps
/* spill slot on a VM stack */
if ((exit_flags & ZEND_JIT_EXIT_FIXED)
&& (t->stack_map[t->exit_info[exit_point].stack_offset + var].reg != ZREG_NONE
- || t->stack_map[t->exit_info[exit_point].stack_offset + var].flags != ZREG_TYPE_ONLY)) {
+ || t->stack_map[t->exit_info[exit_point].stack_offset + var].flags != (ZREG_TYPE_ONLY | undo))) {
exit_point = zend_jit_duplicate_exit_point(ctx, t, exit_point, snapshot_ref);
addr = (void*)zend_jit_trace_get_exit_addr(exit_point);
exit_flags &= ~ZEND_JIT_EXIT_FIXED;
}
t->stack_map[t->exit_info[exit_point].stack_offset + var].reg = ZREG_NONE;
- t->stack_map[t->exit_info[exit_point].stack_offset + var].flags = ZREG_TYPE_ONLY;
+ t->stack_map[t->exit_info[exit_point].stack_offset + var].flags = ZREG_TYPE_ONLY | undo;
} else {
if ((exit_flags & ZEND_JIT_EXIT_FIXED)
&& (t->stack_map[t->exit_info[exit_point].stack_offset + var].reg != IR_REG_NUM(reg)
- || (t->stack_map[t->exit_info[exit_point].stack_offset + var].flags & ~(ZREG_LOAD|ZREG_STORE|ZREG_LAST_USE)))) {
+ || (t->stack_map[t->exit_info[exit_point].stack_offset + var].flags & ~(ZREG_LOAD|ZREG_STORE|ZREG_LAST_USE|ZREG_UNDO_MATH)))) {
exit_point = zend_jit_duplicate_exit_point(ctx, t, exit_point, snapshot_ref);
addr = (void*)zend_jit_trace_get_exit_addr(exit_point);
exit_flags &= ~ZEND_JIT_EXIT_FIXED;
}
t->stack_map[t->exit_info[exit_point].stack_offset + var].reg = IR_REG_NUM(reg);
- t->stack_map[t->exit_info[exit_point].stack_offset + var].flags &= (ZREG_LOAD|ZREG_STORE|ZREG_LAST_USE);
+ t->stack_map[t->exit_info[exit_point].stack_offset + var].flags &= (ZREG_LOAD|ZREG_STORE|ZREG_LAST_USE|ZREG_UNDO_MATH);
}
} else {
if ((exit_flags & ZEND_JIT_EXIT_FIXED)
&& (t->stack_map[t->exit_info[exit_point].stack_offset + var].reg != ZREG_NONE
- || t->stack_map[t->exit_info[exit_point].stack_offset + var].flags != ZREG_TYPE_ONLY)) {
+ || t->stack_map[t->exit_info[exit_point].stack_offset + var].flags != (ZREG_TYPE_ONLY | undo))) {
exit_point = zend_jit_duplicate_exit_point(ctx, t, exit_point, snapshot_ref);
addr = (void*)zend_jit_trace_get_exit_addr(exit_point);
exit_flags &= ~ZEND_JIT_EXIT_FIXED;
}
t->stack_map[t->exit_info[exit_point].stack_offset + var].reg = ZREG_NONE;
- t->stack_map[t->exit_info[exit_point].stack_offset + var].flags = ZREG_TYPE_ONLY;
+ t->stack_map[t->exit_info[exit_point].stack_offset + var].flags = ZREG_TYPE_ONLY | undo;
}
} else if (!(exit_flags & ZEND_JIT_EXIT_FIXED)) {
int32_t idx = _add_trace_const(t, ctx->ir_base[ref].val.i64);
@@ -4477,6 +4478,85 @@ static int zend_jit_store_reg(zend_jit_ctx *jit, uint32_t info, int var, int8_t
return 1;
}
+/* Returns the IS_LONG value of the stack slot 'var' of an exit point */
+static ir_ref zend_jit_deopt_slot_lval(zend_jit_ctx *jit, const zend_jit_trace_stack *stack, int var, const zend_jit_exit_const *constants)
+{
+ int8_t reg = STACK_REG(stack, var);
+ uint8_t flags = STACK_FLAGS(stack, var);
+
+ ZEND_ASSERT(STACK_TYPE(stack, var) == IS_LONG);
+ if (flags == ZREG_CONST) {
+ return ir_CONST_LONG((zend_long)constants[STACK_REF(stack, var)].i);
+ } else if (flags & ZREG_SPILL_SLOT) {
+ ZEND_ASSERT(reg != ZREG_NONE);
+ return ir_LOAD_L(ir_ADD_OFFSET(ir_RLOAD_A(reg), STACK_REF(stack, var)));
+ } else if (reg != ZREG_NONE) {
+ return zend_jit_deopt_rload(jit, IR_LONG, reg);
+ } else {
+ /* The value is in the VM stack slot */
+ ZEND_ASSERT(!(flags & ~(ZREG_LOAD|ZREG_STORE|ZREG_LAST_USE)) || flags == ZREG_TYPE_ONLY);
+ return jit_Z_LVAL(jit, ZEND_ADDR_MEM_ZVAL(ZREG_FP, EX_NUM_TO_VAR(var)));
+ }
+}
+
+/* Reconstructs the old value of op1 of the ADD/SUB opline the exit point
+ * points to. See ZREG_UNDO_MATH. */
+static ir_ref zend_jit_deopt_undo_math(zend_jit_ctx *jit, const zend_op *opline, const zend_jit_trace_stack *stack, int var, const zend_jit_exit_const *constants)
+{
+ uint8_t flags = STACK_FLAGS(stack, var);
+ uint8_t opcode = (opline->opcode == ZEND_ASSIGN_OP) ? opline->extended_value : opline->opcode;
+ ir_ref res, op2;
+
+ ZEND_ASSERT(flags & ZREG_UNDO_MATH);
+ ZEND_ASSERT(opcode == ZEND_ADD || opcode == ZEND_SUB);
+ ZEND_ASSERT(var == EX_VAR_TO_NUM(opline->op1.var));
+ ZEND_ASSERT(STACK_TYPE(stack, var) == IS_LONG);
+
+ /* The slot describes where the result of the operation is */
+ if ((flags & ~ZREG_UNDO_MATH) == ZREG_TYPE_ONLY) {
+ /* The result was spilled to the VM stack slot it is bound to */
+ uint32_t res_var = (opline->opcode == ZEND_ASSIGN_OP) ? opline->op1.var : opline->result.var;
+
+ res = jit_Z_LVAL(jit, ZEND_ADDR_MEM_ZVAL(ZREG_FP, res_var));
+ } else if (flags & ZREG_SPILL_SLOT) {
+ ZEND_ASSERT(STACK_REG(stack, var) != ZREG_NONE);
+ res = ir_LOAD_L(ir_ADD_OFFSET(ir_RLOAD_A(STACK_REG(stack, var)), STACK_REF(stack, var)));
+ } else {
+ ZEND_ASSERT(STACK_REG(stack, var) != ZREG_NONE);
+ res = zend_jit_deopt_rload(jit, IR_LONG, STACK_REG(stack, var));
+ }
+
+ if (opline->op2_type == IS_CONST) {
+ zval *zv = RT_CONSTANT(opline, opline->op2);
+
+ ZEND_ASSERT(Z_TYPE_P(zv) == IS_LONG);
+ op2 = ir_CONST_LONG(Z_LVAL_P(zv));
+ } else {
+ op2 = zend_jit_deopt_slot_lval(jit, stack, EX_VAR_TO_NUM(opline->op2.var), constants);
+ }
+
+ return (opcode == ZEND_ADD) ? ir_SUB_L(res, op2) : ir_ADD_L(res, op2);
+}
+
+static int zend_jit_store_undo_math(zend_jit_ctx *jit, const zend_op *opline, const zend_jit_trace_stack *stack, int var, const zend_jit_exit_const *constants, bool set_type)
+{
+ zend_jit_addr dst = ZEND_ADDR_MEM_ZVAL(ZREG_FP, EX_NUM_TO_VAR(var));
+ ir_ref src = zend_jit_deopt_undo_math(jit, opline, stack, var, constants);
+
+ if (jit->ra && jit->ra[var].ref == IR_NULL) {
+ zend_jit_def_reg(jit, ZEND_ADDR_REG(var), src);
+ } else {
+ jit_set_Z_LVAL(jit, dst, src);
+ if (set_type &&
+ (Z_REG(dst) != ZREG_FP ||
+ !JIT_G(current_frame) ||
+ STACK_MEM_TYPE(JIT_G(current_frame)->stack, EX_VAR_TO_NUM(Z_OFFSET(dst))) != IS_LONG)) {
+ jit_set_Z_TYPE_INFO(jit, dst, IS_LONG);
+ }
+ }
+ return 1;
+}
+
static int zend_jit_store_spill_slot(zend_jit_ctx *jit, uint32_t info, int var, int8_t reg, int32_t offset, bool set_type)
{
zend_jit_addr src;
@@ -5099,6 +5179,9 @@ static int zend_jit_math_long_long(zend_jit_ctx *jit,
uint32_t old_res_info;
int32_t exit_point;
const void *exit_addr;
+ uint32_t undo_var = (uint32_t)-1;
+ ir_ref undo_ref = IR_UNUSED;
+ uint8_t undo_flags = 0;
if (opline->opcode == ZEND_ADD
&& Z_MODE(op2_addr) == IS_CONST_ZVAL && Z_LVAL_P(Z_ZV(op2_addr)) == 1) {
@@ -5114,47 +5197,39 @@ static int zend_jit_math_long_long(zend_jit_ctx *jit,
SET_STACK_REF(stack, EX_VAR_TO_NUM(opline->result.var), ir_CONST_DOUBLE((double)ZEND_LONG_MIN - 1.0));
exit_point = zend_jit_trace_get_exit_point(opline + 1, 0);
SET_STACK_INFO(stack, EX_VAR_TO_NUM(opline->result.var), old_res_info);
- } else if ((op == IR_ADD_OV || op == IR_SUB_OV) && !same_ops && Z_MODE(op1_addr) == IS_REG) {
- /* Undo the operation on the cold path for deoptimization,
- * instead of keeping op1 alive across it.
- * This allows reusing its register for the result. */
- uint32_t op1_var = EX_VAR_TO_NUM(opline->op1.var);
- ir_ref old_ref = STACK_REF(stack, op1_var);
-
- ir_ref if_ov = ir_IF(ir_OVERFLOW(ref));
- ir_IF_TRUE_cold(if_ov);
-
- /* undo: opposite operation */
- ir_ref old_op1;
- if (op == IR_ADD_OV) {
- old_op1 = ir_SUB_L(ref, op2);
- } else {
- old_op1 = ir_ADD_L(ref, op2);
- }
-
- SET_STACK_REF_EX(stack, op1_var, old_op1, STACK_FLAGS(stack, op1_var));
-
- exit_point = zend_jit_trace_get_exit_point(opline, 0);
- exit_addr = zend_jit_trace_get_exit_addr(exit_point);
- if (!exit_addr) {
- return 0;
- }
- jit_SIDE_EXIT(jit, ir_CONST_ADDR(exit_addr));
-
- /* After the snapshot is built. */
- SET_STACK_REF_EX(stack, op1_var, old_ref, STACK_FLAGS(stack, op1_var));
- ir_IF_FALSE(if_ov);
- exit_point = -1;
} else {
+ if ((op == IR_ADD_OV || op == IR_SUB_OV)
+ && !same_ops
+ && Z_MODE(op1_addr) == IS_REG
+ && op1 > 0
+ && ref > 0) {
+ /* On overflow the VM re-executes this opline and needs the old
+ * value of op1. Instead of keeping op1 alive across the guard
+ * (which prevents reusing its register for the result), let the
+ * deoptimizer reconstruct it from the result and op2. */
+ undo_var = EX_VAR_TO_NUM(opline->op1.var);
+ undo_ref = STACK_REF(stack, undo_var);
+ undo_flags = STACK_FLAGS(stack, undo_var);
+ if (undo_flags & (ZREG_LOAD|ZREG_STORE)) {
+ /* The old value is already in memory. Just don't keep it alive. */
+ SET_STACK_REF_EX(stack, undo_var, IR_NULL, undo_flags);
+ } else if (undo_ref == op1) {
+ SET_STACK_REF_EX(stack, undo_var, ref, undo_flags | ZREG_UNDO_MATH);
+ } else {
+ undo_var = (uint32_t)-1;
+ }
+ }
exit_point = zend_jit_trace_get_exit_point(opline, 0);
}
- if (exit_point >= 0) {
- exit_addr = zend_jit_trace_get_exit_addr(exit_point);
- if (!exit_addr) {
- return 0;
- }
- ir_GUARD_NOT(ir_OVERFLOW(ref), ir_CONST_ADDR(exit_addr));
+ exit_addr = zend_jit_trace_get_exit_addr(exit_point);
+ if (!exit_addr) {
+ return 0;
+ }
+ /* The snapshot is created here */
+ ir_GUARD_NOT(ir_OVERFLOW(ref), ir_CONST_ADDR(exit_addr));
+ if (undo_var != (uint32_t)-1) {
+ SET_STACK_REF_EX(stack, undo_var, undo_ref, undo_flags);
}
may_overflow = 0;
} else if ((res_info & MAY_BE_ANY) == MAY_BE_DOUBLE) {
@@ -17466,6 +17541,22 @@ static void zend_jit_preserve_parent_regs(zend_jit_ctx *jit,
zend_jit_trace_stack *parent_stack = parent_vars_count == 0 ? NULL :
parent->stack_map +
parent->exit_info[exit_num].stack_offset;
+ int undo_op2_var = -1;
+
+ if (ssa) {
+ /* The register of op2 of a ZREG_UNDO_MATH slot is needed even if
+ * the variable itself is not used by the trace */
+ for (i = 0; i < parent_vars_count; i++) {
+ if ((STACK_FLAGS(parent_stack, i) & ZREG_UNDO_MATH)
+ && !ssa->vars[i].no_val) {
+ const zend_op *opline = parent->exit_info[exit_num].opline;
+
+ if (opline->op2_type != IS_CONST) {
+ undo_op2_var = EX_VAR_TO_NUM(opline->op2.var);
+ }
+ }
+ }
+ }
for (i = 0; i < parent_vars_count; i++) {
if (STACK_FLAGS(parent_stack, i) != ZREG_CONST
@@ -17482,7 +17573,7 @@ static void zend_jit_preserve_parent_regs(zend_jit_ctx *jit,
} else {
ZEND_UNREACHABLE();
}
- if (ssa && ssa->vars[i].no_val) {
+ if (ssa && ssa->vars[i].no_val && i != undo_op2_var) {
/* pass */
} else {
ir_ref ref = ir_RLOAD(type, reg);
diff --git a/ext/opcache/jit/zend_jit_trace.c b/ext/opcache/jit/zend_jit_trace.c
index 45936ba01f4..0d04120fee6 100644
--- a/ext/opcache/jit/zend_jit_trace.c
+++ b/ext/opcache/jit/zend_jit_trace.c
@@ -2920,7 +2920,7 @@ static zend_jit_reg_var* zend_jit_trace_allocate_registers(zend_jit_trace_rec *t
&& STACK_FLAGS(parent_stack, i) != ZREG_ZVAL_COPY
) {
/* We will try to reuse register from parent trace */
- RA_IVAL_FLAGS(i) = STACK_FLAGS(parent_stack, i);
+ RA_IVAL_FLAGS(i) = STACK_FLAGS(parent_stack, i) & ~ZREG_UNDO_MATH;
count += 2;
} else {
RA_IVAL_FLAGS(i) = ZREG_LOAD;
@@ -3606,6 +3606,26 @@ static int zend_jit_trace_deoptimization(
ZEND_ASSERT(reg != ZREG_NONE);
ZEND_ASSERT(check2 == -1);
check2 = i;
+ } else if (STACK_FLAGS(parent_stack, i) & ZREG_UNDO_MATH) {
+ if (ssa && ssa->vars[i].no_val) {
+ /* pass */
+ } else {
+ uint8_t type = STACK_TYPE(parent_stack, i);
+
+ ZEND_ASSERT(opline && type == IS_LONG);
+ if (!zend_jit_store_undo_math(jit, opline, parent_stack, i, constants,
+ STACK_MEM_TYPE(parent_stack, i) != type)) {
+ return 0;
+ }
+ if (stack) {
+ if (jit->ra && jit->ra[i].ref) {
+ SET_STACK_TYPE(stack, i, type, 0);
+ SET_STACK_REF(stack, i, jit->ra[i].ref);
+ } else {
+ SET_STACK_TYPE(stack, i, type, 1);
+ }
+ }
+ }
} else if (STACK_FLAGS(parent_stack, i) & ZREG_SPILL_SLOT) {
if (ssa && ssa->vars[i].no_val) {
/* pass */
@@ -8114,6 +8134,20 @@ static void zend_jit_dump_exit_info(zend_jit_trace_info *t)
fprintf(stderr, "(zval_try_addref)");
} else if (STACK_FLAGS(stack, j) == ZREG_ZVAL_COPY) {
fprintf(stderr, "zval_copy(%s)", zend_reg_name(STACK_REG(stack, j)));
+ } else if (STACK_FLAGS(stack, j) & ZREG_UNDO_MATH) {
+ fprintf(stderr, "(undo_math=");
+ if ((STACK_FLAGS(stack, j) & ~ZREG_UNDO_MATH) == ZREG_TYPE_ONLY) {
+ fprintf(stderr, "type_only");
+ } else if (STACK_FLAGS(stack, j) & ZREG_SPILL_SLOT) {
+ if (STACK_REG(stack, j) == ZREG_NONE) {
+ fprintf(stderr, "spill=0x%x", STACK_REF(stack, j));
+ } else {
+ fprintf(stderr, "spill=0x%x(%s)", STACK_REF(stack, j), zend_reg_name(STACK_REG(stack, j)));
+ }
+ } else {
+ fprintf(stderr, "%s", zend_reg_name(STACK_REG(stack, j)));
+ }
+ fprintf(stderr, ")");
} else if (STACK_FLAGS(stack, j) & ZREG_SPILL_SLOT) {
if (STACK_REG(stack, j) == ZREG_NONE) {
fprintf(stderr, "(spill=0x%x", STACK_REF(stack, j));
@@ -8688,6 +8722,62 @@ int ZEND_FASTCALL zend_jit_trace_hot_side(zend_execute_data *execute_data, uint3
return ret;
}
+/* Returns the IS_LONG value of the stack slot 'i' of an exit point */
+static zend_long zend_jit_trace_exit_slot_lval(zend_execute_data *execute_data, const zend_jit_trace_info *t, const zend_jit_registers_buf *regs, const zend_jit_trace_stack *stack, uint32_t i)
+{
+ ZEND_ASSERT(STACK_TYPE(stack, i) == IS_LONG);
+ if (STACK_FLAGS(stack, i) == ZREG_CONST) {
+ return (zend_long)t->constants[STACK_REF(stack, i)].i;
+ } else if (STACK_FLAGS(stack, i) & ZREG_SPILL_SLOT) {
+ ZEND_ASSERT(STACK_REG(stack, i) != ZREG_NONE);
+ return *(zend_long*)((uintptr_t)regs->gpr[STACK_REG(stack, i)] + STACK_REF(stack, i));
+ } else if (STACK_REG(stack, i) != ZREG_NONE) {
+ return (zend_long)regs->gpr[STACK_REG(stack, i)];
+ } else {
+ /* The value is in the VM stack slot */
+ return Z_LVAL_P(EX_VAR_NUM(i));
+ }
+}
+
+/* Reconstructs the old value of op1 of the ADD/SUB opline the exit point
+ * points to. See ZREG_UNDO_MATH. */
+static zend_long zend_jit_trace_exit_undo_math(zend_execute_data *execute_data, const zend_jit_trace_info *t, uint32_t exit_num, const zend_jit_registers_buf *regs, const zend_jit_trace_stack *stack, uint32_t i)
+{
+ const zend_op *opline = t->exit_info[exit_num].opline;
+ uint8_t opcode = (opline->opcode == ZEND_ASSIGN_OP) ? opline->extended_value : opline->opcode;
+ zend_ulong res, op2;
+
+ ZEND_ASSERT(opcode == ZEND_ADD || opcode == ZEND_SUB);
+ ZEND_ASSERT(i == EX_VAR_TO_NUM(opline->op1.var));
+ ZEND_ASSERT(STACK_TYPE(stack, i) == IS_LONG);
+
+ /* The slot describes where the result of the operation is */
+ if ((STACK_FLAGS(stack, i) & ~ZREG_UNDO_MATH) == ZREG_TYPE_ONLY) {
+ /* The result was spilled to the VM stack slot it is bound to */
+ uint32_t res_var = (opline->opcode == ZEND_ASSIGN_OP) ? opline->op1.var : opline->result.var;
+
+ res = (zend_ulong)Z_LVAL_P(EX_VAR(res_var));
+ } else if (STACK_FLAGS(stack, i) & ZREG_SPILL_SLOT) {
+ ZEND_ASSERT(STACK_REG(stack, i) != ZREG_NONE);
+ res = *(zend_ulong*)((uintptr_t)regs->gpr[STACK_REG(stack, i)] + STACK_REF(stack, i));
+ } else {
+ ZEND_ASSERT(STACK_REG(stack, i) != ZREG_NONE);
+ res = (zend_ulong)regs->gpr[STACK_REG(stack, i)];
+ }
+
+ if (opline->op2_type == IS_CONST) {
+ zval *zv = RT_CONSTANT(opline, opline->op2);
+
+ ZEND_ASSERT(Z_TYPE_P(zv) == IS_LONG);
+ op2 = (zend_ulong)Z_LVAL_P(zv);
+ } else {
+ op2 = (zend_ulong)zend_jit_trace_exit_slot_lval(execute_data, t, regs, stack, EX_VAR_TO_NUM(opline->op2.var));
+ }
+
+ /* Unsigned arithmetic: the operation overflowed */
+ return (zend_long)((opcode == ZEND_ADD) ? res - op2 : res + op2);
+}
+
int ZEND_FASTCALL zend_jit_trace_exit(uint32_t exit_num, zend_jit_registers_buf *regs)
{
uint32_t trace_num = EG(jit_trace_num);
@@ -8751,6 +8841,8 @@ int ZEND_FASTCALL zend_jit_trace_exit(uint32_t exit_num, zend_jit_registers_buf
} else {
ZVAL_COPY(EX_VAR_NUM(i), val);
}
+ } else if (STACK_FLAGS(stack, i) & ZREG_UNDO_MATH) {
+ ZVAL_LONG(EX_VAR_NUM(i), zend_jit_trace_exit_undo_math(execute_data, t, exit_num, regs, stack, i));
} else if (STACK_FLAGS(stack, i) & ZREG_SPILL_SLOT) {
ZEND_ASSERT(STACK_REG(stack, i) != ZREG_NONE);
uintptr_t ptr = (uintptr_t)regs->gpr[STACK_REG(stack, i)] + STACK_REF(stack, i); |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PHP code:
Old x86-64 code:
Note that this code uses an extra register and 2 extra moves (for %rcx) to carry the result. This is because the trace exit wants the old result. But we can avoid the extra register and extra moves if we simply reconstruct the old value upon exit.
This patch implements that idea, which results in the following new code:
Notice how the addition now no longer needs extra moves and how the result is kept in %rcx all the time. The cold path under
jnorestores the old value right before jumping to the trace exit.