segfault in mcachefs_metadata_do_remove_hash

[rickysarraf]

Description
First of all, thank you for mcachefs. I believe this can serve as a caching layer for my needs where the backing store is a remote share. I am using the latest mcachefs from the git repository. I have run into frequent crashes today. It mostly looked like a memory leak but seemed to die eventually even before OOM could come into action.

If you need more information, please do mention.

OS: Debian Unstable
mcache: git HEAD

Stacktrace

           PID: 251736 (mcachefs)
           UID: 0 (root)
           GID: 0 (root)
        Signal: 11 (SEGV)
     Timestamp: Tue 2020-03-10 17:50:28 IST (15min ago)
  Command Line: mcachefs /mnt/chutzpah /home/rrs/.cache/mcachefs/chutzpah -o rw,auto_unmount,uid=1000,gid=1000,allow_other,cache=/home/rrs/.cache/mcachefs/mcachefs-cache-chutzpah,metafile=/home/rrs/.cache/mcachefs/mcachefs-metafile-chutzpah,journal=/home/rrs/.cache/mcachefs/mcachefs-journal-chutzpah,dev,suid
    Executable: /usr/local/bin/mcachefs
 Control Group: /system.slice/home-rrs-.cache-mcachefs-chutzpah.mount
          Unit: home-rrs-.cache-mcachefs-chutzpah.mount
         Slice: system.slice
       Boot ID: 2f55b26eef1a4cef828d61d938c2e69a
    Machine ID: c879ebc407d84ea997f9e5564dd7aab6
      Hostname: priyasi
       Storage: /var/lib/systemd/coredump/core.mcachefs.0.2f55b26eef1a4cef828d61d938c2e69a.251736.1583842828000000000000.lz4
       Message: Process 251736 (mcachefs) of user 0 dumped core.
                
                Stack trace of thread 251736:
                #0  0x00005568bab1cee8 mcachefs_metadata_do_remove_hash.part.0 (mcachefs + 0x8ee8)
                #1  0x00005568bab1f2ab mcachefs_metadata_remove_children (mcachefs + 0xb2ab)
                #2  0x00005568bab227c5 mcachefs_metadata_populate_vops (mcachefs + 0xe7c5)
                #3  0x00005568bab177e4 main (mcachefs + 0x37e4)
                #4  0x00007fd25fdb4bbb __libc_start_main (libc.so.6 + 0x26bbb)
                #5  0x00005568bab179ba _start (mcachefs + 0x39ba)

Mount service

Mar 10 18:01:48 priyasi mount[252915]: INF|7f413d4ffb80|200310:180148:159|mcachefs-config.c:170:set_default_config|Normalized mountpoint : home_rrs_.cache_mcachefs_chutzpah
Mar 10 18:01:48 priyasi mount[252915]: INF|7f413d4ffb80|200310:180148:159|mcachefs-config.c:194:mcachefs_dump_config|mcachefs Configuration :
Mar 10 18:01:48 priyasi mount[252915]: INF|7f413d4ffb80|200310:180148:159|mcachefs-config.c:195:mcachefs_dump_config|* Mountpoint /home/rrs/.cache/mcachefs/chutzpah
Mar 10 18:01:48 priyasi mount[252915]: INF|7f413d4ffb80|200310:180148:159|mcachefs-config.c:196:mcachefs_dump_config|* Source /mnt/chutzpah
Mar 10 18:01:48 priyasi mount[252915]: INF|7f413d4ffb80|200310:180148:159|mcachefs-config.c:197:mcachefs_dump_config|* Cache /home/rrs/.cache/mcachefs/mcachefs-cache-chutzpah
Mar 10 18:01:48 priyasi mount[252915]: INF|7f413d4ffb80|200310:180148:159|mcachefs-config.c:198:mcachefs_dump_config|* Metafile /home/rrs/.cache/mcachefs/mcachefs-metafile-chutzpah
Mar 10 18:01:48 priyasi mount[252915]: INF|7f413d4ffb80|200310:180148:159|mcachefs-config.c:199:mcachefs_dump_config|* Journal /home/rrs/.cache/mcachefs/mcachefs-journal-chutzpah
Mar 10 18:01:48 priyasi mount[252915]: INF|7f413d4ffb80|200310:180148:159|mcachefs-config.c:200:mcachefs_dump_config|* Backup Threads 0
Mar 10 18:01:48 priyasi mount[252915]: INF|7f413d4ffb80|200310:180148:159|mcachefs-config.c:201:mcachefs_dump_config|* Write Back Threads 0
Mar 10 18:01:48 priyasi mount[252915]: INF|7f413d4ffb80|200310:180148:159|mcachefs-config.c:202:mcachefs_dump_config|* Metadata Threads 0
Mar 10 18:01:48 priyasi mount[252915]: INF|7f413d4ffb80|200310:180148:159|mcachefs-config.c:211:mcachefs_dump_config|* Extra arg : mcachefs
Mar 10 18:01:48 priyasi mount[252915]: INF|7f413d4ffb80|200310:180148:159|mcachefs-config.c:211:mcachefs_dump_config|* Extra arg : -o
Mar 10 18:01:48 priyasi mount[252915]: INF|7f413d4ffb80|200310:180148:159|mcachefs-config.c:211:mcachefs_dump_config|* Extra arg : rw,auto_unmount,uid=1000,gid=1000,allow_other,dev,suid
Mar 10 18:01:48 priyasi mount[252915]: INF|7f413d4ffb80|200310:180148:159|mcachefs-config.c:211:mcachefs_dump_config|* Extra arg : /home/rrs/.cache/mcachefs/chutzpah
Mar 10 18:01:48 priyasi mount[252915]: INF|7f413d4ffb80|200310:180148:159|mcachefs-metadata.c:285:mcachefs_metadata_open|Opening metadata file '/home/rrs/.cache/mcachefs/mcachefs-metafile-chutzpah' (256 entries per metadata block, hash size=4 bytes)
Mar 10 18:01:45 priyasi systemd[1]: Mounting /home/rrs/.cache/mcachefs/chutzpah...
Mar 10 18:02:27 priyasi mount[252914]: Segmentation fault (core dumped)
Mar 10 18:02:27 priyasi systemd[1]: home-rrs-.cache-mcachefs-chutzpah.mount: Mount process exited, code=exited, status=139/n/a
Mar 10 18:02:27 priyasi systemd[1]: home-rrs-.cache-mcachefs-chutzpah.mount: Failed with result 'exit-code'.
Mar 10 18:02:27 priyasi systemd[1]: Failed to mount /home/rrs/.cache/mcachefs/chutzpah.

Mount Config

rrs@chutzpah:/home/rrs /mnt/chutzpah  fuse.sshfs auto_unmount,noauto,x-systemd.idle-timeout=60,x-systemd.automount,_netdev,IdentityFile=/home/rrs/.ssh/id_rsa_chutzpah_lenovo_pwless_ssh,idmap=user,allow_other,default_permissions,uid=1000,gid=1000,follow_symlinks,cache=yes,kernel_cache 0 0
rrs@lenovo:/home/rrs /mnt/lenovo  fuse.sshfs auto_unmount,noauto,x-systemd.idle-timeout=60,x-systemd.automount,_netdev,IdentityFile=/home/rrs/.ssh/id_rsa_chutzpah_lenovo_pwless_ssh,idmap=user,allow_other,default_permissions,uid=1000,gid=1000,follow_symlinks,cache=yes,kernel_cache 0 0

/mnt/chutzpah /home/rrs/.cache/mcachefs/chutzpah        fuse.mcachefs   auto_unmount,noauto,x-systemd.idle-timeout=60,x-systemd.automount,_netdev,uid=1000,gid=1000,allow_other,cache=/home/rrs/.cache/mcachefs/mcachefs-cache-chutzpah,metafile=/home/rrs/.cache/mcachefs/mcachefs-metafile-chutzpah,journal=/home/rrs/.cache/mcachefs/mcachefs-journal-chutzpah

[rickysarraf]

I found the cause. This happened because host chutzpah was not accessible. Infact, it was not accessible because I was on a different network.

But the whole purpose of mcachefs is the same; the ability to be able to use the cache while roaming. I hope the lag/leak was because of the crashing mcache process and that it doesn't do lookups when roaming.

Because, otherwise, (side)effectively it blocks the kernel and brings everything to a stall.