This question was presented to R’ Meir Fishels (head of Prague Beis Din while the Noda B’Yehuda was Rav).
Answer: The wealthy envy the Talmidei Chachomim more because they will never get to be both, while Talmidei Chachomim have a chance at gaining wealth, as well.
Note: Rabbi Fishels himself was quite affluent, having Torah uGedula beMakom Echad.