Good article Philipp. This is a very important topic right now, I was just interviewed about hat yesterday. The NYT doesn't really get it! It's not about the spider, it' about the user: Ask yourself how often do you search for metaphors? On the net, a rose is a rose is a rose once again. Like it should it be.

Besides, what's the problem, not being listed or not being understood at all? If you write in english you can get lot's of non-native english speakers like myself who do not undertand english puns or metaphors in most cases. So if you want to reach out to everybody you try to write a non elitist text that can be grasped by almost anybody.
But that' not all. How do most people read online? They skim pages, headlines and articles. So if the article is not straight in the face enough people won't read it.

The NYT offers one solution that is self evident:
"Some news sites offer two headlines. One headline, often on the first Web page, is clever, meant to attract human readers. Then, one click to a second Web page, a more quotidian, factual headline appears with the article itself."

The explanation ist still wrong. Why do they think we got headline 1 – headline 6 in html?
So headline 1 ist the site title, headline 2 the extravagant article title and headline 3 the descriptive title for instance.
Or like I do in many cases: Germany: Google Monopoly – the benevolent King.
You get the metaphor and the subject matter in one line.

I think you nee a new breed of editors, who are technically savvy and can write. It's not even about SEO really.