Sounds to me like maybe the sample rate is changing away from what your application expects (44.1 vs. 48 kHz). I used to have this problem with Pulse quite often, but I don’t remember if I fixed it before switching to PipeWire.
Pulse has a “preferred sample rate” setting in its config files, but it seems to only really be a suggestion, and Pulse tends to ignore it.