For the human polls, it all depends on how everyone around you does. Oregon went from (2) to (1) without playing this past weekend, but from week 5 to week 6 Iowa stayed at (15) in the AP and ESPN polls, and was passed by Michigan State in both and Stanford in the AP due to impressive wins (to the extent beating USC is "impressive").
It DOES matter for the computer rankings--especially Billingsley, where a team accumulates zero points for a bye and always accumulates points for a win (even a bad win). So the fact that Iowa had its bye early will only help our ranking as long Iowa continues winning--particularly given the fact that Billingsley loves Michigan State, Ohio State, and Wisconsin.