Ave. page load time ~ 12 sec. Is this normal?

Hi folks

I run a blog on a wordpress site and recently got hit with the June update. Traffic is down about 15%.

I started poking around and noticed my ave. page load times shown in Google Analytics is about 12 seconds. I remember it used to be under a couple seconds but since I signed up with AdThrive it's gone up.

In speaking to the ad network they say it's normal to see this when you have 3rd party ads.

From reading some threads on here it seems many are saying pagespeed is only a tiny tiny ranking factor and not to worry about it. The alternative would be to get rid of my ads but I'd lose out on a couple grand per month of income.

Anyone else have a similar experience and any advice? Is it something that I really need to get sorted?

thanks

PS - hosting is from wpx.net (pro) but I don't think that's relevant much when it's 3rd party ads slowing things down.