How to undo crap SEO

Hi all, just looking for a bit of advice! I'm a web developer and took over a company that I used to do jobs for about 4 years ago. I'm doing ok now and have built up enough reputation / word of mouth to keep the engines stoked. The website is a bit naffed though, seo wise. The previous owner had spent years paying a person in the Caman Islands to post comment spam all over the place, among other shady things, so it has about 25k of crap backlinks. All of them are pre 2018, when I took it over, but it's meant I haven't bothered with SEO because the waters are too muddied. I'd like to clear it up, and I've disavowed a whole load of domains with google, but I'm not sure what the easiest route would be.

Do you think it's worth ditching the domain and rebranding? I do have some good links that I'd like to keep, not just for SEO purposes but because people do find me through them. Will the naff links naturally disappear and stop being relevant after a few years? Is there a good, systematic way to get them removed so they don't fog up the data if I were to make a real effort to build some good ones? Just wondering what people's thoughts are in this group.