They’d like to allow search engines and block (non-paying) visitors, but they took a lazy approach to it.

The correct approach would indeed be to identify paying visitors (user+password) and search engines correctly (secret key), then they can reliably shut down everyone else.

But that would require Google to cooperate and I suspect they don’t want to set a precedent where they let a website dictate how they get content. They like to deal from an all or nothing position.

Of course there are other methods, such as making public just enough about the article to be relevant in searches, but I don’t know why they don’t do that. Probably lowers their SEO effectiveness if I were to guess.