I think it's definitely a 'brain' thing. It either works one way or the other. There's a few real talented people who can do both. Algebra vs. geometry.
Part of my problem is, I never played the sports I tried to do color for. But, even then, I just wasn't comfortable. Not knowing when to crack the mic. Or when there was an opportunity I'd scramble for anything to say. You gotta know the sport pretty well, the culture, the team, the game....to do color. I don't know it at that level. Even then, I still sucked. The PBP guy I worked with on those two occasions was an amateur, but also one of the best talents I ever saw in PBP. I figured the less of me, the better.
But, I can tell a curve from a fastball....or describe the action on a hockey rink....or name the football play that's being run. And describe who's doing what. That's fairly binary for me. 0s and 1s. And, if it's radio, you can just sorta be wrong about a lot and nobody's any the wiser. Even then, I was really on serviceable. I didn't do it all that much. I suppose I could have gotten better. Color was just that. Color. And my palette is pretty bleak.
Yet, oddly....my experience in the studio on say morning shows and stuff where I was usually the 'straightman'....it was the same exact nature as color. But, we weren't necessarily talking about sports. It could be anything so I was OK there. I usually interacted with the hosts on air like I did IRL. Just shooting the sh#t. Or it was a pretty pre-planned bit.