You Need to Use Signal's Nickname Feature

Encryption can’t protect you from adding the wrong person to a group chat. But there is also a setting to make sure you don’t.