How is it a judgment on his whole life?
It's more that Billy Madison was the first to put down the Sandler formula of lovable idiot faces challenge that attracts a beautiful woman way out of his league. It's more that basically every comedy film he's made since then has followed that same formula, which just makes them highly derivative of the stellar original.
Also, it's notable that Billy Madison featured Chris Farley, and the loss of Chris Farley to future Sandler films was honestly a huge blow. He could have potentially been in Happy Gilmore but wasn't, and by the time Sandler was making The Wedding Singer Farley had passed.
I am fairly sure it was also the only one that featured Norm MacDonald as well (but I'm not 100% sure on that, didn't follow the more recent ones as closely).
Steve Buscemi, on the other hand, has been a staple in Sandler film casts since the beginning, and his role on Billy Madison was one of the best.
So it's really more that Billy Madison built the formula, had the best cast overall, and was just lightning in a bottle that it felt like he spent a career trying to recreate the exact conditions that made Billy Madison so good.
EDIT: One final piece of why Billy Madison works where other Sandler films don't work as well (except maybe The Water Boy): Billy Madison's real villain isn't Eric Gordon, who wants to stop Billy so Eric can take over the company. The real villain is Madison's own loser past, where he has consistently shown himself to be a contemptuous, boorish fool. His own past behavior is why nobody has faith that he will succeed, and he has to overcome people's prior expectations of him to succeed. Some of the only people who have faith in him are the young children that he befriended while he was in elementary school. Happy on the other hand has that epic swing and all he has to do is practice the rest of golf. Shooter McGavin is also a more sympathetic villain because he's an actual professional being showed up by this jerk amateur who acts like he's king shit because he can hit long drives. Billy Madison just works better thematically, overall, because Billy is actually, genuinely awful at everything and has to start at the absolute bottom where Happy comes in already halfway good by pure chance.