The Golden Ages of audio have been only occasional blips on the screen because civilization must be paid for, and it is only occasionally that significant numbers of people with good taste and good sense have generous quantities of discretionary income. In the closing phases of renaissances, demonstrating both Sturgeon's Revelation and Gresham's Law, the people who want to make $30M annually running corporations view the people who will buy the 90% of everything that is crap as a neglected market, so the economies of mass production never apply to the really good stuff.
I think there's some truth to William Morris's assertion that the human being is happiest when he is making something useful with his hands. Since the contingencies of my experience more or less compelled me to acquire some modest skills an easy analysis quickly led me to think that if I could get the drivers and the information necessary I could build some speakers which might kiss about the boundaries of 'world class' at a cost very much below retail. It hasn't turned out to be that easy, but with some help from friends here and elsewhere it's been worthwhile.
As to how other people might feel about the money spent, I couldn't care less, especially when listening to Mozart or Clifford Brown or Joni Mitchell. I'm pretty certain the sound I'm getting couldn't have been acquired for less, so the amount is right.
The problem I have with the word "audiophile" is that it has been used by advertisers to valorize the focus on equipment, the slippery edge of our hobby, and to define an elite membership you can acquire by buying their stuff. It's not a new thing that language gets trounced with misuse and overuse, so I just ignore it. It's not that great a word anyway. The payoff for me is always in getting back to the music. And that is one reason why for me the system building stretches into the years. It's the long plateaus during which I'm listening to music that take the time. It's only when I've come to be distracted from the music by something in the sound that I start gearing up or tweaking. I really don't do a lot of critical listening otherwise.