Blessing of Kings: Theorycraft vs Trial-and-Error gives a quick and dirty way of figuring out how to do more DPS. The only thing I’d add is that you should also take into account resistances – there’s no point in throwing out a high power fire spell against a fire-immune mob.
Here’s a general algorithm for doing high DPS:
1. Rank your spells from highest DPS to lowest DPS.
2. Cast the highest DPS spell.
3. If you cannot cast the highest DPS spell again (because of a cooldown), cast the next highest.
4. Repeat steps 2 and 3, going down the list as necessary.
It seems very simple, but following this will provide you with a decent spell rotation for every class. It won’t be the optimum rotation, and you might have to drop a spell because it costs too much mana, or does too much threat, or has a really short range. The work comes in ranking your spells correctly. (The main trick here is that when calculating DPS for a non-burst fight, you only look at the cast-time of the spell, not the lifetime of the spell. For direct damage they’re the same thing, but not for DoTs.)