The key thing here is that when it sucks for people in the First Age, it sucks much more than it does in the Second Age. Sure, maybe nobody dies of cholera. But in the Second Age, entire cities aren’t transformed into living crystal or squid-monster. As bad as the Perfect is, he is much nicer than the person who created his staff ever was, and that person had access to the orb which made it even worse.
And I’m not certain there are less slaves. I mean, the Dragonblooded don’t have entire races of slaves, after all.
The key to understanding the First Age is that it was not necessarily better than the Second Age. It was BIGGER and more extreme, in all directions.
The best parts of the First Age are better than the best parts of the Second Age, no doubt there.
The worst parts of the First Age are also far worse than the worst parts of the Second Age, also no doubt there.
The First Age was a place of precarious balance. It swung between peace and war with surprising ease. Heck, did you miss the part where the Solars engaged in a world-devastating civil war for longer than the Scarlet Empire has existed. Or the fact that the Solars managed to preside over the one and only time when Creation actually was destroyed?