First, except in the special circumstances I discussed previously, speculative trading can increase volatility only if it loses money. As Milton Friedman noted in his seminal 1953 paper The Case For Flexible Exchange Rates
People who argue that speculation is generally destabilizing seldom realize that this is largely equivalent to saying that speculators lose money, since speculation can be destabilizing in general only if speculators on the average sell when the currency is low in price and buy when it is high.That does not mean that individual speculators cannot profit from activities that increase volatility, but they can do so only at the expense of other speculators. Consider a market in which trend-following speculators are active. An ingenious speculator might create an artificial trend by buying a stock in sufficient volume, causing the trend-followers to start buying into the trend, driving the stock higher. When the clever guy judges the trend-followers have filled their boots, he'll dump the stock at the higher price, locking in a profit. The stock will thereafter gradually revert to whatever it's really worth, and at some point the trend-followers will sell out, realising their losses.
But there is only so much money that unsuccessful speculators are willing to lose, so the capacity for speculators to increase volatility is limited. (Bankers have lost apocalyptic amounts of money in the last few years, but not on speculative trading of the sort that might be discouraged by an FTT.)
Nevertheless, the review paper I discussed previously cited no fewer than twelve papers attempting to predict the effect on volatility of a transaction tax. (The ante-penultimate link doesn't now work, and the paper, which is good, doesn't discuss volatility directly.) Each of them sets up an a model of a securities market, and predicts how it will operate with and without a transaction tax by means of theoretical analysis, or by computer simulation of trading strategies, or by having humans playing a trading game.
An essential component for a transaction tax to be able to reduce volatility in these models is the existence of what Fischer Black (of Black-Scholes) called "noise traders". These are traders who speculate in the market without having any information not already priced in, as distinct from "information traders". In Black's conception, traders often do not know which sort of trader they are, which creates uncertainty essential for the operation of a liquid market. The papers I list do not all follow the definition exactly, but they incorporate the concept in some form. Their results seem to depend on to the extent that the way the market is modelled tends to cause the transactions tax to discourage noise traders more than others.
None of the papers has a set-up which is much like actual equity markets: this is partly because they are analysing something like the Tobin tax proposed for FX markets and partly because it's often easier to analyse something other than reality. But it's the equity market that the proposed European FTT is mainly concerned with (the FX market is excluded). And none of the papers includes the full range of trading strategies that operate in actual markets.
A noise trader whose decisions are indistinguishable from random will add a small amount of volatility, and tend gradually to lose money. I am sceptical that there are many traders of this sort. Most speculative traders follow some sort of strategy, broadly the strategies are either trend-following or contrarian. (One of the papers listed explicitly includes both these strategies; others may do so implicitly by using human traders in their simulations.) It's important to include the trend-followers to give a transactions tax a fair chance to reduce volatility significantly
If I were to attempt something like this I would want to include at least the following:
- large trades being executed gradually. This would feed a series of trades in the same direction, with the broker varying the size and timing in an attempt to disguise what he's doing. These trades are profitable for trend-followers
- trades being done for exogenous reasons. These are not strictly noise trades, and will not be deterred by a small transactions tax, but their size and direction looks random.
- information trades (some authors call them fundamental trades). These are done by traders who have used private aptitude to deduce fundamental valuations from public information.
- hedge trades. These are trades done by option traders who are in aggregate either long or short gamma. If option traders are short gamma their hedging tends to increase volatility, and vice versa.
- insider trades. These are trades done using information that is not yet public, but is made public after some time delay. These trades are profitable for trend-followers.
- trend-following speculative trades.
- contrarian speculative trades.
- speculative trades attempting to profit at the expense of other speculative strategies
- a stochastic process for the fundamental value. All traders will be aware of the direction of large changes in fundamental value (corresponding to obviously important news).
That's a lot of things to put in, and a lot of parameters and relative weightings to vary. However, there are only three sorts of trades which tend to increase volatility - short gamma hedging, unsuccessful trend trades (successful trend trades don't increase volatility, they just bring the price change forward), and trades parasitic on trend trades. If things are set up so that trend trading is profitable despite being unsuccessful quite often then a transactions tax sufficient to make it unprofitable can decrease volatility significantly. Note that trend traders do need to trade quite often, because they need to unwind their position quickly if a trend they've traded on fails to continue.
My guess is that with some care it would be possible to create a set-up where this happens. More tentatively, I guess that the actual market doesn't match it.