Mysql would fall back to a disk based sort once the data set is too large.

Perhaps it's his client that is OOM-ing? Back in the day, I remember the PHP mysql client loading the entire result set into a local buffer unless you explicitly asked for the output to be streamed.

Edit: Found reference to PHP behavior I mentioned: http://php.net/manual/en/mysqlinfo.concepts.buffering.php