SEO on WordPress - with plugins or without plugins?

Hi guys, not really an SEO expert here, but rather a web designer and developer.
I am thinking about the idea to leave SEO plugins behind. Over time I came to the conclusion that those plugins are the holy grail for people with limited technical skills. To me, the only benefit I can gather is that those plugins make things happen quickly. However, in some cases a bug in an SEO plugin can potentially mess up rankings, not to mention how bloated some of those plugins are. That's why I want to have more control over and "break free" from the bloat.

More than advice, I'm looking for opinions from real SEO experts - do you think there is a benefit in using SEO plugins? If yes, which is your go to SEO plugin? And finally, is there anyone here who has already left behind SEO plugins and are you happy with that choice?