Will the custom upgrades apply to the newly hosted site?
No, the Custom Design upgrade is tailored to only work on WordPress.com in its currently designed form. But you'll be able to edit CSS on a self-hosted WordPress site (or you can use the Custom CSS module of our popular Jetpack plugin).
My only reason for doing this would be to facilitate the sale of my book, which I understand is not acceptable on wp.com. Am I correct on this?
You can use WordPress.com to create a site for your book, so that's fine. It is true that we don't have e-commerce built in on WordPress.com, so if you want a shopping cart functionality on your site, then self-hosted WordPress would be a better fit. See more about the differences between dotcom and dotorg: http://en.support.wordpress.com/com-vs-org/