Also window managers started compositing which moved 1/3 of what X was doing to the window manager. Then applications started doing their own rendering which moved another 1/3 of what X was doing to the applications. All that is left over is basically the low-level IO which had gotten greatly simpler over the years and could basically be packaged into a few libraries (mesa and libinput primarily) and some complex mutli-hop IPC which was completely unnecessary.