Or that contact allowing FB to scan their contact list. Remember, there are two endpoints here and if you did not allow it then is is more likely the other party did.