Not necessarily. You probably want to optimize the kernel and a few packages. Then there are some apps where you want to build them with specific features. Then there’s a bunch of stuff that takes forever to build where a binary would be convenient. Flags and optimizations aren’t that important for KDE frameworks or Firefox.
Offering binaries is a really nice middle ground. Gentoo makes it so easy to build custom packages from source but it’s always been all or nothing. I don’t want to wait 2-3 hours building updated libraries or Firefox every time there’s a patch.
Personally, I would be interested in a distro that had binary packages, easy builds like Gentoo and something like Arch’s AUR.