but I am going to use a different website to host it
To point the domain you registered at WordPress.com to another server (hosting service), please follow the instructions here: http://en.support.wordpress.com/domains/change-name-servers/#changing-domain-name-servers
Note you will need to obtain the name server values from your current host. You should be able to find them on their support documentation, or contact them and ask for the information. (Name servers may be referred as "nameservers" or "NS records" instead.)
Can I just cancel the entire thing?
Canceling a domain means you completely give up ownership of the domain and can take up to 35 days before it will become available to register somewhere. If you would like to completely cancel your ownership of the domain name, please reply to confirm.
We are only able to offer a refund if we receive notification within 48 hours of the purchase.
Hope this helps - let us know if you have any more questions!