I don’t know how you can separate the idea of colonialism from the formation of the US. Seems arbitrary and without any philosophical basis.
As it stands currently, black Americans have better standards of living, in almost all developmental indices, than native Americans. So by any scientific inquiry, native Americans have had it and still have it much harder than black Americans.