If he meant exactly what you said, I agree. But, there is an alternate interpretation of what the guy was saying:
You tend to get different kinds of customers with different price ranges. The ones who can afford to spend money generally don’t give a crap about what you’re billing them for, and they just want the work done properly.
The ones who aim to get a “good deal” tend to be less hands-off and more critical about the work done/supplies used and billed for. Frugal customers take extra time and sanity to field questions/suggestions, and sometimes, it’s just not worth dealing with.
If raising his fee filters out the latter category, it’s hard to blame him. I wouldn’t want to deal with penny-pinchers either, and simply being more expensive than the competition is an effective deterrent.