What pages to include in your XML sitemap?

I'm currently looking at crawl budget and improving our crawlability, how important is this, or is it only important when you're working with a huge site?

My site has 1,471 pages, is that too small to see crawl improvements if I remove unimportant pages?

I've seen times where it's made a huge difference but that seems to be with sites that have 5,000+ pages and then then exclude old content/low traffic pages.

Thanks,