They could be low level. I certainly remember when a Ruin Guard was a big scary monster that could kill my whole party.
And later when I was playing pretty seriously I still stayed out of Abyss Floor 12 because it wasn’t worth the effort (literally - I never got enough stars to get a reward from it even when I technically defeated it)