I suppose all genres have blurry edges, but it's certainly true that fantasy is hard to pin down. Here are a few of my own thoughts on a narrow definition of what is fantasy, and what it is not. These are, of course, merely my own opinions, colored by my own interests, and I'm fully aware that other people may draw their genre borders in other places with other criteria. Feel free to add your two cents to this post if you like!
I'll start with the sci fi/fantasy grouping. I think many people (especially those who aren't especially interested in either genre!) tend to differentiate the two based on their setting or accoutrements. Science fiction is set in the future or in space and includes spaceships, fancy technology, robots, aliens, and so on. Fantasy is often set in pre-industrial societies, and involves magic, mythological creatures, and the sorts of characters you find in fairy tales. These differences are becoming increasingly fuzzy as more and more books include both magic and technology, both mythological creatures and robots, the scenery of past, present, and future… But I don't think the differences in setting are substantive markers of genre anyway. Rather, the difference between sci fi and fantasy has more to do with the role the magic/technology plays in the story. Sci fi explores the impact or implications of the technology (which has at least a token explanation of how it's consistent with natural laws) as a major focus of the plot, while fantasy's magic (which is counter to or beyond the laws of nature) is more part of the setting or framework the characters have to work in. Star Wars, for example, is fantasy, not sci fi, because the technology is purely magical, simply an unexplained fact of the universe, and the issues the characters face are the basic good versus evil quests. On the other hand, I would categorize The Story of Doctor Dolittle by Hugh Lofting as science fiction because the Doctor's ability to speak with animals has a scientific explanation (it doesn't matter that it's a fictional scientific explanation) and the stories often explore the limits and uses of that new communication technology.
Speaking of talking animals, another category that often gets labelled "fantasy" is stories peopled by animals. The Redwall series by Brian Jacques is a representative example of the type - but I don't believe that this sort of book really has much to do with the fantasy genre. Other than the purely decorative accident that the characters are woodland creatures, this is simply a novel set in a fictional historical setting. It's historical fiction with more fiction than history, true, but it isn't magical. If you replaced all the mice and rats with humans, nothing about the story would change, and it would have nothing fantastical about it. Watership Down by Richard Adams and the Warriors series by Erin Hunter have even less to do with fantasy, as the animals that are the characters don't even wear clothes and walk on their hind legs, yet both works are frequently listed as fantasy. I assume the idea is that to imagine that animals can interact with each other as if they had societies is supposed to be fantastical, but to me that's simply fiction, just as any set of imagined characters facing imagined conflicts is fiction.
Redwall's fictional historical setting brings me to folk tales such as Robin Hood. Folk tales are lumped in with fairy tales, and fairy tales are fantasy… But Robin Hood, according to my ideas, is really not fantasy at all. Again, it's historical fiction. At most it's a bit of a tall tale.
And finally, another category that isn't fantasy is anime. Some anime is fantasy, of course, but anime is a medium, not a subject. To throw all anime into the fantasy section of the library or bookstore is a silly as saying that all movies are sci fi or all short stories belong in the mystery section (or all cartoons are for children, a categorization that we're just beginning to break out of here in the US). And yet anime seems to be a standard topic at fantasy conventions. I guess that has more to do with an assumption that there's a sufficiently large overlap among fans of anime and fans of fantasy. Still, I'm a fan of both fantasy and block printing, but I would never expect there to be a convention dedicated to both simultaneously. (Hmm… that would be pretty cool, though, wouldn't it!)
So if I'm full of strictures on what fantasy is not, perhaps I'd better try to say what I think fantasy is. I could talk about the magic, and the impossible creatures, and goodness knows fantasy is as prone as any genre to its own conventions… but I guess at its most basic and its best I think fantasy is a genre that's about allowing the imagination free rein, so that both the storyteller and the reader can be unbound by expectations about reality.
[Picture: Pandora Dreaming, wood block print by AEGN, 2005.]