It's pretty accepted wisdom these days that if you're going to write a book with magic in, you need to put constraints on what that magic can do or the story won't be interesting. Magic wielders can make problems disappear with a wave of their hand or a well-placed word. You'll get deus ex machinas cropping up on every page. The villain of the piece will be so powerful your heroes don't stand a chance. Readers will expect rules and get frustrated when they can't find any. The story will end on page 10. And so on.
The topic came up briefly in #scifichat on Twitter a couple weeks ago. I might even have raised it myself, I don't remember. I do remember having to clarify what I meant, though, and the act of clarifying go me thinking. For me, magic rules exist on a gradient like so much else in the world of the Writing Process. I also apparently define "constraints" pretty loosely. For me, it's not just about who can do magic or how magic happens or where magic's found. It's kind of a mix of all of that, and a little more. Bear in mind this is from a reader's perspective. I have no doubt that even the looser types of constraints are pretty rigorous from the writer's end.
The loosest constraints on magic are found in books where magic is in the background or where the reader doesn't need a detailed description of how magic works to understand the story. The castle flies, but we don't need to know why or how. The elves in the forest just kind of are. The heroine steps through a doorway and finds herself on another continent. People can cast spells in multiple ways and magic rarely if ever has negative consequences for the caster. Ambient natural magic, the sort that humans don't control, also fits here. It's hard to tell if the writer's created rules or is just working from what feels right, but there's no question the story works.
The next level up from that are magic systems that are nearly identical to the ones listed above, except there are a handful of things magic can't do, or can't do well, or people can't do magic until they're trained. Maybe magic doesn't allow time travel or maybe it doesn't work well on animals. Maybe you get magical training on your parents' knees, or maybe you go to school for it. Magic's still pretty much in the background and the story can almost but not quite be told without it.
Then we start seeing systems where magic can do just about anything, but only certain people can make it work. Maybe they've got fairy blood. Maybe they're wizards. Maybe it's anyone with blue eyes or red hair. I'd rank systems where magic can do anything but involves intricate spellwork or specific conditions at this level too, along with magic that does anything but is hard to control. And systems that involve magical backlash are probably here too—where casting a spell diminishes personal energy or causes pain, and magicians can do anything as long as they can handle the kickback. Making it harder for the characters to do magic makes it harder for them to achieve their goals and I think, though I'm not sure, that this is about where most of the action-oriented fantasies come in.
After the broad application levels, the potential of magic starts narrowing. Maybe different people do different sorts of magic, so the guy's who good with fire is going to suck at or be unable to read the future, and the guy who reads the future can't make a potion to save his life. Maybe magic can only apply to living things, or inorganics, or the elements. Maybe spells end after thirteen hours.
From there, magic systems narrow further by imposing more and more limitations, to the point where blue-eyed, red-headed daughters of seventh sons can only control earth magic under the full moon at midnight if they've had twenty years of training and eaten special herbs beforehand. I think when you get that complicated and make magic almost impossible to do that the fantasy becomes less fun, but maybe that's me. I'd be delighted to be proved wrong.
I think a similar series of gradients probably applies to science fiction. We've got hard science fiction, after all, and various "soft" sci-fis until we hit space operas like Star Wars which can sometimes be classed as fantasy. It's probably even possible to sync the two gradients up and create one that encompasses all of speculative fiction. I am, however, too lazy to do that, but maybe I've inspired you?
Oh, and if anyone has thoughts on this, positive, negative, or simply thoughts, let me know. I like discussions. :)