The economist John Maynard Keynes once famously dismissed gold as a "barbarous relic." Notwithstanding the withering disdain of the most influential economist of the 20th century, gold has maintained its unique status as both a commodity and a store of monetary value. For investors, everything that is bearish for traditional financial assets — political instability, inflationary fears, a falling dollar — has been always bullish for the yellow metal. The factors mentioned above propelled gold prices to one of its greatest runs ever starting right around the dot-com bust. Between 2001 and 2011, gold rose from $250 to a high of $1,900 an ounce in 2011. That was massive 660% increase.That 12-year bull run made gold bugs seem like the smartest guys in the room. Alas, things have been a lot tougher since gold peaked in 2011, and it is highly questionable whether or not investors should continue to hold gold. Over the past five years, the price of gold has been flat while the S&P 500 is up over 85%. S&P 500 vesus GLD More recently, the price of gold has suffered one of its sharpest declines in years. Over the past three months alone, the price of the yellow metal has slid nearly 10%. The furious short-covering rally that helped gold recover somewhat in Friday's trading does little to alter its relentless downward trend. All of this has put the gold bugs on the defensive. A case in point is the example of the University of Texas endowment. In April 2011 — just a few months before gold peaked, local permabear Kyle Bass advised the country's third-largest endowment to take delivery of nearly $1 billion worth of gold bars. That decision cost Texas hundreds of millions of dollars. To its credit, Texas did reportedly cut its losses somewhat be selling $375 million worth of these bars less than two years later. A less-than-golden future Gold today is caught in a "perfect storm" likely to keep its price in check. First, gold thrives when the news is bad. And contrary to gold bugs' predictions, the U.S. economy has not imploded. Americans did not take to the streets with pitchforks. More recently, the U.S. economy expanded by a stronger-than-expected 3.5% in the third quarter, and unemployment now is below 6%. Second, the much-feared hyperinflation brought on by quantitative easing (QE) never came to pass. The U.S. dollar has yet to replace the collectible 100 trillion dollar Zimbabwe notes as an icon of inflation. Third, interest rates are now more likely to rise than fall. The Federal Reserve has ended its quantitative-easing program and suggested that strong economic growth and the improving labor market could mean higher interest rates as soon as 2015. The prospect of higher interest rates has also has fuelled the rally in the U.S. Dollar Index, which recently hit a four-year high. The collapse in gold prices has investors heading for the hills. Global holdings in exchange-traded funds backed by the yellow metal have dropped to their lowest level in five years. In 2013, when investors rushed for the golden exits, Asian bargain hunters in the physical-gold market flooded in and bought gold at what were then seemingly bargain prices. In 2104, Chinese buyers have been conspicuously absent. Retail gold consumption fell by one-fifth in the first nine months of the year compared with the same period last year, according to the China Gold Association. Having been burned last year, Chinese investors are understandably reluctant to catch the falling gold knife. The uncomfortable truth Gold is not the Platonic ideal of value. Its price is driven almost exclusively by sentiment.That makes it a speculative asset, in the purest sense of the word. Gold pays no interest. In fact, it costs you money to store it. And you can't really model the financial value of an asset that does not pay any income. That's why George Soros — as vociferous a critic of the global financial system as any gold bug — made billions by betting on gold. But Soros did so, not by buying and holding gold forever. He did so by realizing that gold was a speculative asset and exiting at much higher price levels than we have today. Meanwhile, committed gold bugs got caught without a place to sit in this speculative game of musical chairs. Yet, for me, the most compelling argument for not owning gold long term is more financial than speculative, and comes from investor Warren Buffett. On May 7, 2012, Buffett told CNBC: "When we took over Berkshire, it was selling at $15 a share and gold was selling at $20 an ounce. Gold is now $1,600 and Berkshire is $120,000." Since that statement, gold has fallen to as low as $1,131 and Berkshire shares have risen to $218,000. Not that a mere 200 times difference in returns would ever convince a gold bug. Gold bugs love to point out that you can still buy a nice suit with the gold coin, just as you could in Roman times. But you could also buy a townhouse in Sarasota, Florida, with a single share of Berkshire Hathaway BRK.B, -0.10% Of course, if a meteor is heading toward planet Earth, all bets are off. The guys in the Montana hills with the gold bars will have won the investment sweepstakes. But otherwise, if you are still blinded by the religious fervor of gold bugs, I urge you to repent, ditch your faith and face the truth — and make gold the barbaric relic part of your portfolio's past. Disclosure: Vardy owns Berkshire.