The specs of the baseboard read like OK, finally done right.
But then it depends on the price of the board, and the price and availability of the CM4 with 8GB, or that Nvidia thing with also 8GB RAM. And then you have them still interconnected over slow 1Gb/s LAN. Is that switch able to handle jumbo frames?
I'd guess you could blow it out of the water with the cheapest Mini-ITX board which can handle some CPU with integrated graphics and some real OOMPH, for less money, maybe slightly less power, equal or more and faster memory for all cores/threads at the same time, and less general hassle.
edit: Or slightly less compact, Micro-ATX, small case, even cheaper, because Mini-ITX seems to be always more expensive, even though less material is used :-) Which still leaves you with the choice of PSU, compact and expensive like Pico-PSU, or something larger, but cheaper? https://en.wikipedia.org/wiki/ATX#SFX
Or full size? Most cheap, and probably silent, because room for large and slow spinning fan, if at all!
