When you sign in with Apple on OptiCull, Apple displays the Apple ID login dialog. If you have already signed in to your Mac with your Apple ID, it just needs to confirm if you want to continue with that account. For the confirmation, it either uses Face ID, Touch ID, or your device passcode (in case Touch ID is not available or not configured). Users often try their Apple ID password instead of the system passcode on this prompt, resulting in sign-in failure.
Once you click the Sign in with Apple button, what you see is Apple's own sign-in flow, and OptiCull doesn't receive any of your credentials. You can find more information about this in "Use Sign in with Apple" section of the Apple's sign in guide.