Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

How do you evaluate quality ? Also I suspect the performance between models would varry between datasets. Heck it would vary on same model/source if you included that your mother was being held hostage and will be killed unless you summarize the source correctly :).

I think you are still stuck with try if it works for you and hope it generalizes beyond your evaluation.



I think summarization quality can only be a subjective criterion measured using user studies and things like that.

The task itself is not very well-defined. You want a lossy representation that preserves the key points -- this may require context that the model does not have. For technical/legal text, seemingly innocuous words can be very load-bearing, and their removal can completely change the semantics of the text, but achieving this reliably requires complete context and reasoning.


There are actually some clever approaches to eval abstractive summarization.

Examples: https://eugeneyan.com/writing/evals/#summarization-consisten...


>evaluate quality

[information content of summary] / [information content of original] for summaries of a given length cap?




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: