Actually the text widget does support the image. Just do it like the following:
<a href="http://www.buzzcomix.net/in.php?id=drmike" title="Vote for us on BuzzComic" target="_blank"><img src="http://www.buzzcomix.net/vote/vote-small.gif" alt="BuzzComix" /></a>
I do it with my webcomic and a Blogshares link as well.
The RSS link, I would just add it into a text widget. I know the Bookmarks won't take the 'feed:` part of the link.
Yes, you can delete categories from your blog. Dashboard -> Manage -> Categories. Sometimes though you'll wind up with an empty space in the category list on some Posts that were labeled with that category so you might want to keep an eye out for that.
hope this helps,