Redis provides an app called redis insight, works great for debugging such issues and it mostly supports elastic cache. https://github.com/redisinsight/redisinsight

Ah, yes - the usual fix: return true; things do indeed get faster when there's less to do.

They patched out a gross inefficiency that they created

I think the entire point of this post is just to share a war story and help others understand some Redis problems and debugging steps. It isn't some deeper optimization guide or anything. This exact problem is something I have caused and encountered multiple times in my career now. On my first day at Instacart the entire site was down in an extended outage due to a hot key issue exactly like this. I think stories like this are worth sharing with the community even if it seems trivial to yourself.

That's fair, there is value (to someone) in showing how the sausage is made.

That said, the diagnostics exhibited here are the exact things I'd do, as a person who visits Redis once maybe every two years.

They're not exactly groundbreaking, and I was hoping to find revelations here; not basics.

Looking at what the cluster as a whole is doing, and a specific node, and trying to move it. Amazing.

> They patched out a gross inefficiency that they created

You'd hope that code reviews would catch this kind of thing, especially on a small team.

This is team of 2 engineers that's building + shipping faster than most teams I've seen while having insane number DAUs. Moving fast (and patching problems even faster) is the name of the game here and these guys delivered.

I would also understand not everything is always perfect, things happen and people learn as they make mistakes

Fear of this is why I prefer a sorted set with ZREMRANGEBYSCORE to remove expired items