You could also give him the benefit of the doubt. Most uber drivers I've met have been very polite, and very concerned about bad ratings.

I had a driver who got lost once, and he just stopped the trip and took me to my destination -- because he would rather go a few miles out of pocket than risk a bad rating.