Ask them what they think being a web designer means. It's a very hard title to pin down.

SOme people think it's just making pretty layouts in Photoshop.

Ask them for examples of bad UI designs and good UI designs. Good web designers are also good UI designers. If they just grunt "Apple is good", toss 'em in the reject pile - they're lazy, unimaginative, and likely inflating their experience. If they talk about washing machines or toasters, you're looking at a good candidate.

Ask them if they own a colour calibration device. If they do, that's a major positive mark in their favour. If they don't even know what one looks like, toss 'em in the reject pile.

Ask them what the positives and negatives are for each browser. If they blather on about Firefox being open source and supporting open standards blah blah blah toss 'em in the reject pile - just because someone knows HTML doesn't make them a good web designer.

My favourite interviewing question is to ask when they first used a computer, and what got them interested in pursing a computer-based career. I prefer to work with people who have found computers to be fun from an early age rather than just a way to pay the rent. YMMV, of course.

(disclaimer: I've worked with web designers, but all of my interviewing experience has been with software engineers and QA people)

Pick a website, ask them to tell you how to improve it. If it's not YOUR website, that's fine - b/c then you don't look like you're asking for work for free, but you still get an idea of what s/he would do. Provide whiteboard for sketching.

Of course all of this is dependent on what you are looking for in a "web designer" - my husband is an information architect, which means he designs the user interaction for his company's website(s). There are people who are web designers who execute this plan. If you have an IA, you don't need a web designer who is ALSO an IA, but you may be looking for both. It's good to write down what your specific expectations are, I suppose.

Write down what you need. Some people are web designers, some are flash designers, some ar flash developers, some people are information architechts, some people are code-crunchers and developers, and there's a mile of difference between all of these but some rare birds are all three!

and some people can't count.

Did the first round of interviews today. One was not so stellar, the other one was sort of OK, he's more flash oriented rather than HTML+CSS, but when I asked him for his favorite sites he showed us some real clunkers.
I like the "how would you fix this site" idea. Have 3 more interviews to do, I'll try it out.
Thanks for all the answers!