Full speed can work for most hardwoods with a handheld router and a small enough diameter bit but not for all. Some woods are prone to burning if the bit is allowed to dwell just a little. I really like what Chris said...
these are just guidlines, not the rule book
It's all about feed and speed. You can set the tool to a certain "recommended" speed but depending on the work piece material, your own feed speed, and the cutter, you can be dead on or quite a bit off.
I think for handheld applications, it's best to test the material and cutter combination (the same material and same set up) to determine the ideal tool speed. (This goes for all cutting tools.) I think the tool should turn just fast enough to not chatter under similar conditions.
The cut will be nicer - smooth and not burnt and the cutter will last longer. It lasts longer and you don't burn the material because you are actually cutting instead of dwelling. If you're dwelling, you are creating heat from friction. If you're cutting, you are dissipating heat through what you are taking off - it absorbs heat from the cutting surface.
People who are used to using their tools generally have a feel for how fast to turn them. People who aren't used to using their tools are lucky if they get it right the first try.
Tom