I “kind of” solved at least parts of the problem by simply not turning wifi off. It’s more stable than before, but not stable as such.
Edit: I played around a bit and I think the problem is somewhere in the power supply. My workbench PC can drive the board just fine for hours, just plug it into the USB port, connect serial monitor and let it do its thing. However, if I’m plugging it into my router or a usb power supply, it stops working after a while again.
My theory is, that the PSU/router thinks the device is dead, because it draws too little power? That would explain, why not shutting down wifi improves the stability - it draws more power.
Now, that is just my naive theory, I’m not entirely sure, how to validate it. Maybe I’ll just add a useless busy-loop to increase power draw or add some power consuming components.