This functionality was never supported but before the new review queue it didn't have any errors. Now the combination of settings is prevented and existing sites with sso enabled will be migrated to remove invite only.
Robin Ward