If you are right in front of his face, I believe it is becasue he is uncomfortable. Humans, as primates, love to be face to face and give hugs and such. Dogs, as canines, find this behavior uncomfortable whicc is why the ears fo back. I am sure there is more to it, but that is what I remember reading.