Signal or any surveillor surrounding their servers (with or without Signal's cooperation) almost certainly has enough timing/traffic-shape info to reconstruct who-to-who logs.

"Sealed sender" (and some of Signal's other tactics) just demonstrate: Signal's main & disclosed codepaths aren't stockpiling the canonical metadata via the same blatant & undenied mechanisms of other services. Sufficiently sophisticated outside attackers, or insider threats, can construct nearly-equivalent logs via other means. (And: Signal seems reluctant to make choices, like truly ditching phone numbers as account IDs, that could limit these 'shadow' leaks.)

> all they can see is an IP address

That is precisely the, ahem, signal metadata.

Yep, whoever gets hold of those records can cross-reference logs from the same time to narrow down or even outright identify Signal chat participants.