There are a couple different approaches you could take for this. One would be to just use the Appearance → Background page. If you want to set a background image and anchor it to the right side of the page, upload a background image to Appearance → Background first and then you will see some display options appear—select the options for "Right" and "Tiled Vertically."
Note that if you set a background image it will be anchored to the right side of the browser viewport and so it will change when the size of the browser window changes.
If you wanted to anchor a background image to the right side of the main wrapper of the content in The Columnist theme, you could add something like this to your Appearance → Custom Design → CSS editor:
background: url(http://s.wordpress.org/about/images/logos/wordpress-logo-32-blue.png) 100% 0% repeat-y;
Change the url() value to your own image link, and adjust the padding-right value to match the width of your image or until you like where the image appears.