One thing to remember is that stats are never going to be instantaneously updated. That just isn't possible with the number of users and sites that WordPress.com has. There will always be a delay between when someone subscribes and when they show up listed in your totals. Now, the issue we have been dealing with was that the delay was taking WAY longer than it should. I am hearing that this issue should be fixed and that the cached info that the stats page uses should be updating.
Another thing to remember is that you are NOT notified if someone unsubscribes/unfollows you. So when we are talking about sites being off by 1-5 followers, that can easily explain that sort of thing. Sadly, some people just don't know good content when they see it and decide to stop reading what we all put so much energy into writing!
Adding those two reasons together, and you can see that at any given time, the followers number for an active blog is likely to be off by a few people. PLEASE remember that when making error/bug reports. With error/bug reports, we're looking for the larger all-site problems. Those are the indications that something is seriously wrong. The smaller "off by a couple" should correct themselves as caches are updated etc.