I've suspected this for a little while now. 
I'm seeing data in Google's own tools that indicates to me that Google is very curious if you are popular. Not from a backlink perspective but from the number of direct visits to your site. I've got data to back this up too. Love to know if you are seeing similar correlations. 
Follow Me On Twitter @jimboot to join me live as I audit your site on Facebook every weekday 7:30am AESDT