Oh wow.. They don't have many students. Yet they've done very well over time
The first thing is they're parochial. They have nothing restricting the geographical area that they pull their kids in from, and they also have incentives such as tuition and other perqs that public schools have. My school has a couple mile radius and there are Catholic school kids at various schools driving 40-50 miles because of sports. This isn't just RSM. The last 30 years, 22 state 1A baseball champs were from 7 different private schools. Mostly Catholic except for Iowa Mennonite one time.
The other one is culture. RSM kids pretty much all go out for sports and they play all sports. On the boys side at least, almost every student plays baseball, football, basketball, and track.
Once you start wining and start a tradition, the rest falls into place. Like I mentioned, West Lyon has way fewer students than my school, but probably double the football players total. But WL has also produced LeVar Woods, Kyle Vanden Bosch, Daren Naatjes (Stanford FB and baseball), Brandon Snyder, Monte Pottebaum, and a bunch of others. And that's just D1 football. The Kramer family has multiple NAIA All-Americans from there in football and baseball (dual sporters in college), and one of them played baseball in the Big Ten (Indiana). They're well-coached, and for some reason most of the alumni sticks around to perpetuate when their kids come through. All of their coaches in baseball, football, and basketball excpet for a couple that I can think of are alumni. That's unheard of.