Sure, okay. But there's no reason they couldn't be. Most of classification rules come out of the executive branch, and could be changed by the executive branch to facilitate such a thing. You could have the FBI do it. Or the Secret Service/Treasury. Or some guys from the the Archives. Doesn't really matter.
As an example, it is possible to get searched while leaving my worksite. If the security guard finds a classified doc on me, it's not a good argument to just point the finger at the guard and claim he doesn't have the need to know what was in that document so he should get in trouble too. It's true he doesn't need to know what exactly is in the document, but he does have a need to know if I am carrying anything marked classified on my person as I exit the building. That's his job. If he reads it and gets himself into trouble, then that's on him.
Point is that it is possible to have a very small number of people get cleared to help screen all documents that ended up at a former POTUS/VPOTUS's house as a standard part of transition. The screeners have a job of finding potential classified docs in a box in a garage. They have a "need to know" enough to view cover pages and headers to complete their task. They don't have to take possession. Just report back to the former POTUS, "Hey, in box L5 there are three coverpages which appear to have documents inside them marked 'TS/SCI'.". Make a report of as much so that if the former POTUS does nothing to rectify the situation with the proper custodian of those documents, then it is clearly on him.
It definitely isn't solely on an official that left office if something or several somethings got misfiled into the wrong box. A mistake was made in a likely frenetic situation, and it might not be their fault at all. It's hard to say when exactly, but at some point they are absolutely responsible for what they accepted into their possession at their home. Sometime measured in months or weeks, not years.
Edit: It might have to be voluntary on the part of the former official, since it would mean the team rifling through what should MOSTLY be personal documents. But I would tend to think that most (maybe Trump excepted) would be willing to have the help and the reduced liability.