Any advice for dealing with incorrect location information hurting Google places

I have a service based business and I moved. Shortly after moving my address for Google I dropped from top 3 in most of my keywords to basically not existing.

Two problems I can think of. One is mismatched location info on various directories. Problem is many of these aren't directories we manually added ourselves to. I'm not sure I'm finding all the places we are listed to try and fix.

Secondly we now live outside of town. It's service based and we still service the same town in the same way but google seems to think it's very important the gardener lives within a few blocks instead of a few km. Not sure anything can be done about this or if it's real, but it sure seems to be. Not sure why google prioritizes location so much when you don't have a location to visit but are service only.

Any tips? Anyone dealt with something similar and have any good ideas? Do I just have to endlessly search for incorrect information and try to find a way to change it?