Nobody is upset at the idea of trying to make government work more efficiently. They are upset because this effort is being run by a small cadre of unelected individuals without the necessary expertise to understand how our large and complex government works. Further, they seem as interested in ideological issues as they do efficiency (see USAID, as an example). Their claims of savings have been shot full of holes, with errors and either misunderstandings or lies being frequently highlighted, drawing their competence into question. They have also made notable mistakes that they have tried to scramble to undo. Ultimately, it is pretty clear that they are working from a stance of: Government is terrible, we need to tear it down. Government has problems, but it also does a lot of good. They will end up harming a lot of people in their teardown effort. I do think there is a chance that much good comes from their efforts, as well, but I think this whole undertaking could have been more thoughtful and legal, and still been approached with similar vigor.
Also, Musk is an egomaniac closely aligned with far-right parties around the globe, so for all of his brilliance, I do not trust him to make reasoned decisions across all areas of a complex government. Being really good at engineering and entrepreneurism does not make one really good at redesigning government.