Here's my general views on everything said so far:
I believe that Redkard has a point. We, in the end, answer only to ourselves (and, trust me, we can be ver harsh on ourselves) and those above us who know where we've been and what we are doing now. Redkard has said it best. We are specifically told not to simply say, "I'm sorry, that call was wrong and I'm going to take it back." We can't. Later on, after a game, another referee may come over and talk to us about what we did right or wrong, or the assigner will tell us, but we can only answer to other referees. It's a matter of representation, appearance, and authority.
Heck, back when I first started, if I hadn't been put with the patient centers that I had, I don't think I'd even have gone past the first six months. Now, though, thanks to the kindness of other referees, reviewers, and friends who watched me to tell me how I was doing, that time is in the past and won't be returning (I should hope.) If I'd run into the parents that I see nowadays that the newbie refereeing community has to deal with, I definately wouldn't be here now. Why do you think we have such a high turnover rate each year?
I've worked with referees who I could never want to work with again, I've played under referees where I don't understand how they are certified, and I've watched referees that I have no clue what they saw. But that doesn't matter. They are still referees, they are wearing the badge, and they are taking the time to get out there and be on the field with the kids.
Over time, I think that many people come to see this and understand it. Yelling from the sidelines doesn't help anything. Telling us we were wrong doesn't change anything. Once we've made the call, we've made it and we simply can't take it back. So, one must live with what comes and move on. Talk to the referee assigner. I can tell you that, when I've done that before, things are done. People do go check, they make sure things are being done correctly, and they do respond. Maybe not directly to you, but it will be done.
In response to physical play, I do not completely believe that it is the referee's responsibility. I've played in games that are 140LB, 5'9" Freshman (Me) VS 220LB, 6'5" Senior (Them), and they never truly degenerated because the other team played the game, not the person. Admittedly, this was some time ago, but the point remains valid. There is some responsibility of the player to know how to play the game itself (footskills, getting around people, etc.), rather than play the game outside the game (shoving, elbows, going through people, etc.). If you're on the field, you have an understanding of the game and how it should or shouldn't be played. It's simply a coach and player decision whether to go outside those limits or not. Having said that, we are there to deal with you if you do decide to go outside those boundaries.
Well, I've said my (admittedly, rather large) piece on the topic. See it as you may.