I could also afford it and if I lived closer would probably buy tickets, at least while my 11 y.o. is at home. I cut out cable last year and that was $1,300 a year which takes care of a big share of it. I live 5 hours plus away from Iowa City, so there's a ton of gas, wear on my vehicle, meals, and a hotel room every time we go. My kid is diehard Hawkeye fan, so to compromise we go to the spring game every year, and catch for sure one B1G game, sometimes two. We are two hours closer to Lincoln than Iowa City, so if it's a year the Hawks play there we try to hit that one. Since we aren't making the drive every other weekend I justify splurging for really good seats at the games we do go to.
Aside from those games, it's just as much fun to get together with my buddy's family and watch games on TV. We make food, take a leak whenever we want, and get to see everything in HD while being warm.