The preacher should be able to remember his sermon very well that his outline could be blown off the pulpit by the wind and he’s still able to preach the whole sermon without missing anything.
If your outline is so complicated and all your points are difficult to remember even if you’re the one who wrote them, you cannot expect the congregation to remember anything. You should be able to recall your outline as you go along. The less you rely on your notes, the more the people are likely to remember the sermon. You should labor to make your sermon simple and uncomplicated.