As far as I know, MuPDF is not that heavy, and can view both PDFs and EPUBs (and others).
I personally use zathura, which is a very, very light weight document viewer, has vi style key bindings, and has plugins for viewing PDF, EPUB, CB, and others. Works pretty well in a keyboard centric desktop environment (I use Hyprland).