> If you agree on that, now imagine that you added another project to the fixtures. Has the testing power of the tests changed just because fixtures have been changed?
No, _but_ (and this is a big _but_) you're not testing the contract of the method, which (presumably) is to return all and only active projects.
Testing that it returns _some_ of the active methods is useful, but there are cases where it won't point out an issue. For example, image
- Over time, more tests are added "elsewhere" that use the same fixtures
- More active projects are added to the fixture to support those tests
- The implementation in the method is changed to be faster, and an off-by-one error is introduced; so the last project in the list isn't returned
In that ^ case, testing that _some_ of the active projects are returned will still return true; the bug won't be noticed.
Not directly related to the above, but I'll note that I would also split 2/3 into different tests.
- Make sure all projects returned are active
- Make sure projects returned includes all active projects
I think that's more of a style thing, but I _try_ to stick to each test testing one and only one thing. I don't always do that, but it's a rule of thumb for me.
I'm with you on the one assertion per test. I bundled two assertions into the same test here because my whole point was to have them effectively together describe a single test, just in a more maintainable manner.
Regarding the fact that I'm not fully testing the contract of the method, you're absolutely correct. But also, no example based test suite is fully doing that. As long as the test suite is example based it is always possible to find a counter-case where the contract is violated but the test suite misses it.
These counter-cases will be more contrived and less likely the better the test suite. So all of us at some point decide that we've done enough and that more contrived cases are so unlikely and the cost of mistake is so small that it's not worth it to put in the extra testing effort. Some people don't explicitly think about it but that decision is still made one way or another.
This is a long way of saying that I both agree with you but that also, in most cases, I would still take the tradeoff and go for more maintainable tests.
No, _but_ (and this is a big _but_) you're not testing the contract of the method, which (presumably) is to return all and only active projects.
Testing that it returns _some_ of the active methods is useful, but there are cases where it won't point out an issue. For example, image
- Over time, more tests are added "elsewhere" that use the same fixtures
- More active projects are added to the fixture to support those tests
- The implementation in the method is changed to be faster, and an off-by-one error is introduced; so the last project in the list isn't returned
In that ^ case, testing that _some_ of the active projects are returned will still return true; the bug won't be noticed.
Not directly related to the above, but I'll note that I would also split 2/3 into different tests.
- Make sure all projects returned are active
- Make sure projects returned includes all active projects
I think that's more of a style thing, but I _try_ to stick to each test testing one and only one thing. I don't always do that, but it's a rule of thumb for me.