The idea that he won't be good at defense is pretty much based entirely on his pigmentation. We haven't seen enough of him playing healthy at a college level to know if that is true or not. He is long, he is fairly sturdy, and he seems to be a smart basketball player. From his success as a multi-sport athlete, I am guessing he has good hands. Those 4 things combined give you a pretty good head-start on being a solid defender.
Elite quickness certainly helps to be a good defender, but it is not a prerequisite, nor is it a guarantee. Using the NBA as an example, Kyrie Irving, Damian Lillard, and James Harden are all below-average defenders with good quickness. Jamal Crawford has been one of the most dangerous off-the-dribble players his entire career, but he has always been a bad defender.
On the flip side, Ricky Rubio has been one of the best perimeter defenders in the league his entire career because he is smart, has good hands, and is long. Marc Gasol won a defensive player of the year with middle-of-the-pack athleticism. Draymond Green's dominance on defense has little to do with quickness and much to do with strength, intelligence, and attitude.
Basically, that is a long-winded way to say I will wait and see if Connor is a good defender or not.