When I'm NOT otherwise too swamped to organize (and to be honest, most of the time I'm too swamped to organize--not pictured are the ancillary book piles on the floor), I seem to have come up with a system where I first group the books by overarching category, then within that category I alphabetize them. I have category groups for literary stuff, poetry, plays, old textbooks, children's/MG/YA (all one group), grown-up fiction, nonfiction, writing books, language books, and graphic novels. Within each of those, I usually try to keep them alphabetized by author's last name.
Oh, and I have one special stash of books right by my desk which are Frequently Used Titles such as the AP Stylebook and a Welsh dictionary.
Unfortunately (well, not for me, but unfortunately for anyone else), the groups themselves aren't in any particular order--but I did try to group together categories that make sense together. The children's/MG/YA books are in the same bookcase as the grown-up fiction books. Literary, poetry, and plays are in the same bookcase. And graphic novels and nonfiction are in the same case. Buuuut...old textbooks are crammed in with all the fiction, and writing books actually live in a couple of different spots. And then there's the pile of Books What People Done Lent Me That I Haven't Read Yet and the pile of Review Copies I Was Supposed to Read An Embarrassingly Long Time Ago and the box of Stuff to Donate.
And that's just the stuff in my office. Elsewhere in the house are other groupings for art books and travel books and random crap like old high school yearbooks, and stacks of books my husband bought for the purposes of prepping classes or going to seminars. We're definitely book people!