I don't find it very remarkable that when one person makes up a term that's pretty easy to understand, other people in the same conversation use the same term.
In the context of this conversation, it was a response to someone talking about malevolent human therapists, and worried about AIs being trained to do the same things. So that means it's text where one of the participants is acting malevolently in those same ways.
For me, hearing this fantastical talk of "malevolent data" is like hearing people who know little about chemistry or engines saying "internal combustion cars are fine long as we don't run them on 'carbon-filled-fuel'". Otherwise, see my comment above.
Sure, it's not literally impossible. There are ICE cars that run on hydrogen. But you can't practically adapt an existing gasoline car to run on hydrogen. My point is that mobilizing terminology gives people with no knowledge of details the illusion they can speak reasonably about the topic.
> But you can't practically adapt an existing gasoline car to run on hydrogen.
You can do it pretty practically. Figuring out a supply is probably worse than the conversion itself.
> My point is that mobilizing terminology gives people with no knowledge of details the illusion they can speak reasonably about the topic.
"mobilizing terminology"? They just stuck two words together so they wouldn't have to say "training data that has the same features as a conversation with a malevolent therapist" or some similar phrase over and over. There's no expertise to be had, and there's no pretense of expertise either.
And the idea of filtering it out is understandable to a normal person: straightforward and a ton of work.
In the context of this conversation, it was a response to someone talking about malevolent human therapists, and worried about AIs being trained to do the same things. So that means it's text where one of the participants is acting malevolently in those same ways.