If I understand correctly, the way it's going to work is similar to quote posts, something along the lines of
1) the server hosting the account making the reply sends a request to the server of the account being replied to asking whether it's okay to reply
2) the server with the account being replied to says "ok" or "nope"
Obviously this only works with cooperative software ... but that's also true with stuff like DMs and followers-only posts, so it comes with the territory here. Compatibility with older Mastodon versions, and other platforms that haven't implemented reply controls yet (or do it in a different way) is likely to be challenging -- as it is for quote posts. It's all a lot trickier than it sounds!
BTW, there was a similar transition back in 2016/7 when Mastodon unilaterally unlisted and followers-only posts got implemented. It was quite rocky for a while, but over time things moved forward. Stuff like this is a real challenge with interpoperable software in general, and decentralization only makes things more complex, but the other alterantivve is stagnation -- which is a lot worse. This is badly-needed functionality so it's great that it's happening.
@stefan @8124