There are signing keys involved, so if someone puts up a new server but uses different keys then all sorts of federation trouble will await them.
That said it shouldn’t affect the general network, just that individual server (both the communities and the users of it)
Edit: As for switching domains on an existing server, that would be equally troublesome as ActivityPub kinda relies on domains for all sorts of IDs.
It uses WebTorrent for distribution between viewers watching at the same time which can temporarily help with the load on popular videos, but there still needs to be at least one source instance that’s sharing the video “regularly” (for unpopular or old stuff), which ends up having the same bandwidth issues you’d get with any other video platform.