There isn't a uniform temperature across the entire exchanger. There's a smooth gradient extending from one end to the other. If the outside is hotter, then the inbound air gradually cools as it gives up heat to the outbound air which is gradually warming.