Right, even in the idealized models you need to spend a little bit of energy to write down the input and to read out the output of a computation. But the size of the input/output text is typically very small compared to the number of bit operations done during a computation, so if you can "unscramble" the entire intermediate state back to a low-entropy configuration that will get rid of nearly all the heat.

In less idealized models you can't even to do intermediate computations completely losslessly, so those computers will use some energy, but it can approach 0 by working more slowly.