The "displeases us" you quoted is for "the part of a person's life which concerns only himself". For example, "No person ought to be punished simply for being drunk."
For actions "which concerns others" Mill writes:
"The evil consequences of his acts do not then fall on himself, but on others; and society, as the protector of all its members, must retaliate on him; must inflict pain on him for the express purpose of punishment, and must take care that it be sufficiently severe. In the one case, he is an offender at our bar, and we are called on not only to sit in judgment on him, but, in one shape or another, to execute our own sentence: in the other case, it is not our part to inflict any suffering on him, except what may incidentally follow from our using the same liberty in the regulation of our own affairs, which we allow to him in his."
Yes, but in the first chapter, Mill is clear that even public speech falls into sphere of liberty, the part of person's life "that affect only himself":
"But there is a sphere of action in which society, as distinguished from the individual, has, if any, only an indirect interest; comprehending all that portion of a person’s life and conduct which affects only himself, or if it also affects others, only with their free, voluntary, and undeceived consent and participation. When I say only himself, I mean directly, and in the first instance: for whatever affects himself, may affect others through himself; and the objection which may be grounded on this contingency, will receive consideration in the sequel. This, then, is the appropriate region of human liberty."
"It comprises, first, the inward domain of consciousness; demanding liberty of conscience, in the most comprehensive sense; liberty of thought and feeling; absolute freedom of opinion and sentiment on all subjects, practical or speculative, scientific, moral, or theological."
"The liberty of expressing and publishing opinions may seem to fall under a different principle, since it belongs to that part of the conduct of an individual which concerns other people; but, being almost of as much importance as the liberty of thought itself, and resting in great part on the same reasons, is practically inseparable from it."
(Also note that your original citation was also related to judgements based on actions that concern only the judged person: "Though doing no wrong to anyone, a person may so act as to compel us to judge him, and feel to him, as a fool, or as a being of an inferior order")
For actions "which concerns others" Mill writes:
"The evil consequences of his acts do not then fall on himself, but on others; and society, as the protector of all its members, must retaliate on him; must inflict pain on him for the express purpose of punishment, and must take care that it be sufficiently severe. In the one case, he is an offender at our bar, and we are called on not only to sit in judgment on him, but, in one shape or another, to execute our own sentence: in the other case, it is not our part to inflict any suffering on him, except what may incidentally follow from our using the same liberty in the regulation of our own affairs, which we allow to him in his."