To answer part of your thinking... it depends what part of the summer it is as to when the hottest time of the day is. This time of the year we typically don't see the hottest part of the day past about 4 to 5 pm. However in early June or July we can see the most heat from 6 to even 7 or eight. Sun angle allows more heat to escape the atmosphere later in the day closer to dusk.
The reason it isn't right in the middle of the day is that it is a cummulative effect of the heat building throughout the day. Drier climates build that heat way faster than around here. Windier days can heat faster too. I'm not sure when the hottest part of the day was 11-2 or 11-3 unless it was the middle of January. Then it makes more sense.
Yeah, I'm putting the nerd in Herkeynerd with this one.