You are right. If X is 'no' then not-X wouldn't necessarily be "yes".

After looking around I see that Prolog recognizes some nuance around not: https://en.wikipedia.org/wiki/Prolog#Negation

And aldready deprecated one 'not' operator:

https://www.swi-prolog.org/pldoc/man?predicate=not/1

Look at how they are not using not, but rather "not provable".

I'm not sure if Prolog has straight up negation behaving in binary arithmetic way.