I don’t think you can, or should, do away with moderators. There needs to be a way to respond to illegal and abusive material quickly, before many people have to see it and it can propagate.
But I do think a good improvement would be improved transparency in moderator decisions, and accountability. It shouldn’t be that hard to implement a way for the community to remove and appoint moderators. The harder part would probably be safeguarding this mechanisms against trolls and hijackers.