Same here. The issue for me, I think, is that my feelings about music are the same as my feelings about chocolate; it's fine, but I don't see why everyone makes such a fuss about it.
My lack of interest in dancing is just an extension of that.
I did country dancing (essentially line dancing, but done in a circle and without the country and western theme) a couple of times at primary school in the 80s. I quite enjoyed that, but as you say, it was about following the instuctions and patterns rather than reacting to the music.