Generally, using a resistor (potentiometer or otherwise) to control LED brightness isn't recommended because of power loss, although this might be okay if you're only powering a small length of strip. That said, I'm not sure why your setup isn't working. Are you connecting each channel to its own potentiometer, or all of the channels to a single pot?