If it doesn't relate to your post specifically, it's usually spam. If it links to something dodgy/wrong/commercial it's spam. If it has links to sites that are unrelated to your post, it's usually spam.
Here are some previous forum posts that might help:
http://en.forums.wordpress.com/topic/how-do-i-know-if-comments-are-spam?replies=10
http://en.forums.wordpress.com/topic/repeated-comments?replies=5
http://en.forums.wordpress.com/topic/was-it-spam?replies=7
If anything in a comment doesn't make any sense (like random numbers, or something that doesn't relate directly to your post) zap it.
Also you need to 'teach' Akismet (the program that filters the spam) how to work better, as it is obviously letting some through that it shouldn't. So when you get any that are spam, you should mark them as spam.
And here's what to do if the wrong comments go into the spam filter (Akismet):
http://en.forums.wordpress.com/topic/wordpress-user-comments-banished-to-spam?replies=49
I hope some (or hopefully all!) of that helps, tearlily. :)