The only "risk" is that you'll get bad neighbors who don't visit your town, clear their visit marks, or tap all their buildings leaving nothing for you to tap when you visit. Of course the solution to that is to dump them and find new neighbors.
Also, I find that if my neighbors DON'T vandalize my 4 buildings - the Police Station, Town Hall, Springfield Elementary, and Moe's - that my Righteousness actually goes DOWN.
You're also missing out on things you can only get by visiting neighbors. The Friend Prizes take 40 forevers to get with just Other Springfield (which does NOT tap back). And almost all events are made easier by visiting neighbors to get bonus supplies. In the current event you can pick up 270 Intel per day by visiting 30 neighbors, and every neighbor who visits your town and taps means 3 more Intel for a possible 300 additional per day (not that that's actually gonna happen, everyone doesn't visit every day).
I recommend having at least 30 neighbors. You can find plenty of people looking for neighbors in the Add Me thread in the Official Game Forum.
