Consumers just aren’t that interested in a product that’s visibly cheaper and worse than what everyone else is carrying. And that is what a smaller phone signals.
Phones are a status purchase; they all do basically the same things, but most people gravitate towards higher end phones because they offer all the fancy features. Flagship phones are all large, so that’s what you see in the marketing. Just like you’ll never see a car company put its cheapest base model on a car catalog cover.
A smaller phone tends to cut corners; it’s not just smaller, but also functionally worse. While the price might be appealing, the potential customer also knows that using said phone will mean a worse experience, and might even get them ridiculed because they got ‘the cheap one’.
So we can absolutely go back to small phones - we just don’t want to. Smaller, cheaper, worse products just don’t appeal to a status-conscious buyer. If phone manufacturers offered the same specs at different sizes, that might change. But any savvy tech buyer knows a smaller phone is worse than the bigger one.
Back in the pre-smartphone days, size was a thing companies could compete on since customers wanted small, light, distinctive designs in premium materials. Like the Motorola Razr V3. These days, that just doesn’t work.