references to python objects can be randomly lost in lua 5.2 and 5.3

[thisismypassport]

[NOTE - edited as I'm no longer sure I understand this fully]

Usually immediately, but sometimes after a few retries or some time, the below example raises an error because the dict created from the lua table was not able to find the key in the table (since it became unequal to itself) and so put nil instead

import lupa.lua52 as lual 

lrt = lual.LuaRuntime()

while True:
    lrt.globals().K = lambda: object()
    lrt.globals().dict = lambda o: dict(o)

    lrt.execute("""
    k = K()
    t = {[k]=1}
    a = dict(t)
    """)

    if (tuple(lrt.globals().a.values()) != (1,)):
        print(lrt.globals().a)
        raise 0

Reproduces in lua 5.2 and 5.3, but not 5.4 or up.

I have investigated this quite a bit, but I'm not quite sure I understand this...

Still, it seems related to the PYREFST table - which is used with luaL_ref/unref, but is also a weak table (__mode = 'v').

This means:

• Being a weak table, lua is free to nil-out arbitrary entries in it while objects get collected.
• luaL_ref can get the length of the table, which in lua is defined to return an arbitrary index whose next entry is nil, not necessarily the last one.

I originally said this means luaL_ref can return a wrong index, but I'm no longer sure that's correct - I don't see anything wrong with luaL_ref returning an index in the middle of the table, as long as it is currently nil (which it should be, one way or the other)

However, it might be useful to know that some things that cause the issue to no longer reproduce are:

• Making the table strong (naturally a full leak)
• Making luaL_ref not rely on table length (instead tracking it itself manually)
• Using lua 5.4 and up, which are less likely to return a non-last value for the table length (but only less likely - still possible and still legal)

Note that there is also a leak here, thinking about it - there is no guarantee that luaL_ref will ever find the niled-out-due-to-gc entries, especially in lua 5.4 and up.

[thisismypassport]

(Another piece of the puzzle might be that lua appears to [per atomic in lgc.c] to empty weak tables before calling finalizers)