Some blog hosting services (tumblr, wordpress.com I think) let you host a domain on their service. If you want your own server the cheepest is probably NearlyFreeSpeech, the easiest probably dreamhost.

Recent AskMe with more options on hosting.

As for domains: there are often domain name sales for certain top level domains (.com, .net, .us, etc) from the various sellers, and you'll get a better rate (per year) if you buy a couple years out (usually).

Also, if you don't want to plunk down the cash now, many free blogging services offer free subdomains, allowing for a personalized URL, but without the hassle of registering and whatnot.

I use HostGator for my site (unicornbacon.com). It's something like $15 a year for the domain and $9.99 a month for the site, it comes with tons of goodies for the non-expert, including Wordpress and other blog software, and I am completely pleased with their customer service. It's cheap and fairly idiot-proof (which I need on both counts).

I registered my domains through godaddy (cheap, but kind of sleazy and perhaps too big to have really good customer service, but that was before I knew better). I have my actual sites hosted via lunarpages. So far, my only complaint about them is that they don't offer command-line shell access. But that sort of access is pretty hard to come by at consumer rates, and their customer service has been pretty darn good. I don't know whether or not they offer blog/etc goodie options these days.

My recommendation for a relative tech newbie starting a blog is blogspot. Just because it's blogspot doesn't mean it can't have its own web presence. E.g., poet Ron Silliman has created a huge phenomenon with his blogspot blog: http://ronsilliman.blogspot.com/

I switched to hostgator about a year ago for hosting and have been pleased. I mostly use joker.com for domains. I do have a couple that are still registered elsewhere but I like joker the best.

I'm actually pretty pleased with tumblr right now. Getting ready to start using it on my personal site instead of textpattern (which I was in love with when it first came out but over the years I've come to a place where I just don't need that much CMS.)