This can be easily done by syncing role permissions
Enable permission syncing
Dashboard -> Select your Discord server -> Select the correct Creator Channel -> Permissions Tab -> Temporary Channel Permission
Set permission in category
All permissions you set in the category for roles will be applied and synced to temporary channels. Deny View Channels or Connect permission to avoid that role to join temporary channels.
Permissions for @everyone
You cannot sync Connect, View Channel and Send Message permissions for @everyone, because these permissions are managed when using /voice privacy!