I went in Chapters last week, which is something I do infrequently. I get my books from other sources as much as possible, thank you. But I also tend to avoid Chapters because their shelving system in Science Fiction and Fantasy makes absolutely no sense. There's one author who has fantasy novels under Fantasy, Science Fiction, and Horror. Yes, horror. On account of there being werewolves and ghosts and vampires in the novels. Those urban fantasies weren't alone in the Horror section either. Not by a long shot.
I've seen this in other bookstores too, that people don't really know where to put the urban fantasy. I mean, it's got fantasy built into the name of the genre! But UF contains vampires, werewolves, demons, witches, ghosts, and zombies, and people who don't know better assume that means the novels are scary. And sure, they are in places, like the fight scenes and climax, the places where the protagonist's life is at stake. I wouldn't put any of the UF I've read on the same level as Stephen King, though. At least not for the fear factor.
As for how UF can end up in science fiction? A lot of it's set in alternate realities and near futures, and if your concept of fantasy is Tolkien, Martin, Jordan, and the like, then UF doesn't really fit in Fantasy. Also, I'd imagine that, at least in the big chain stores, that many of the employees either aren't big readers or aren't expected to be familiar with the contents of all the sections. It's different with indies, I've noticed.
I'm also sure the tried and true Confused Bookseller Shelving Method comes into play. If you don't know where to put the book automatically (based on the front cover, the author, the buzz, the demographic who's buying it), read the blurb on the back. Key words will jump out at you, like murder, death, detective, blood, crime, which usually means Mystery, or twenty-something, boyfriend, big city, identity, which often means chick lit, or, in the case of urban fantasy, paranormal, detective, magic, vampire, werewolf, demon, witch, race against time, over her head. If the blurb still confuses you, take a wild guess and it's probably literary fiction anyway.* I'm extrapolating for this next statement, but I'd guess that the action/mystery blurbs of UF novels sound closer to the action/mystery blurbs of sci-fi than they do to the drawn out quests and elves and lyrical fantasy worlds of the rest of the fantasy section.
If I had total control over the shelving of urban fantasy, I'd let my anal retentiveness reign free.** I'd create a whole new section of the bookstore (Urban Fantasy) because there are certainly enough UF novels out right now to fill a bookcase or two. I'd put the other fantasy subgenres under Fantasy, and I'd browbeat every employee until they knew what was UF and what wasn't.
I take my UF seriously, okay?
Then again, there are problems with that system. Where do you put Charles de Lint and Emma Bull and Neil Gaiman, who write fantasy novels set in modern cities, but without the action/mystery storylines? You're going to get UF readers who want Hamilton, Harrison, and Gaiman, and you're going to get non-UF fantasy readers who want Gaiman as well. Do you cop out and shelve Gaiman (etc) in both sections? Or do you draw the line and tell customers to stuff it, this is your store and you'll do what you like? And then you get into the thorny issue of the urban fantasy/paranormal romance divide…
Urban fantasy in 2011 is a teenager, and nobody quite knows how to deal with it. Writers are pushing limits, doing things differently, and readers are expecting different things than they were 10 years ago, when the current UF model came into prominence. It takes on so many masks (epic fantasy, comedy, noir, sci-fi, murder mystery, romance) and it sometimes layers those masks so you get two or three or even four in the same book, and how is anyone supposed to come up with a definition for a genre that changes on a daily basis like that? Really? But like any teenager, UF will eventually come into its full identity and … maybe we'll know where to shelve it then?
* This, friends, is the real reason covers and blurbs are so important. It helps booksellers put the books where you want them to. ;-)
** Like I do at home, where my CDs are arranged in chronological order by era. Renaissance before Baroque before Classical before Romantics before Dixieland before Swing.