This would have been awesome to have back when I was heavy in to the UI test automation game. Our best option at that point was a spot instance EC2 fleet and analyzing commits to determine which tests would be the most valuable to run. It's awesome being able to easily run hundreds or thousands of tests in parallel, completely segmented, and pay only on demand. A fantastic use of AWS Lambda! It suddenly becomes reasonable to do full integration tests on every merge request, or even commit, and get feedback to the developer in seconds.
Do the math. If you're suggesting running all the tests, drastically increasing your compute volume, it's probably cheaper to use a dedicated instance running at max CPU utilization.
It absolutely depends on how burst driven your utilization pattern is, but don't forget the cost to actually manage the test cluster, and the engineering cycles that go into optimizing and sharding those tests across the cluster.
For sure, lots of considerations to take into account. If you're already running your own CI pipeline though, tests on top of it are trivial (in my experience running a Jenkins CI pipeline for a python monolith). Tweak your test runner, upsize the EC2 instance, and go grab a mojito.
It's of course based on what's more important for you: Running tests non-stop and perfectly utilizing a compute instance vs having the tests executed when you need the results as soon as possible. I might argue that in most cases the latter is what you'd actually want. Especially given that you're getting billed on a millisecond basis.
I would love to have something like this for my scala API tests. We have 3k tests running on 10 ec2 instances and the entire test-suite takes 10 minutes. If some infrastructure would allow me to pay a little more and run all tests in parallel, that would be a game changer for our team.
