Tom wrote the following comment:
Christian, great article! I was building an ATS myself but had to put it on the back burner. Do you dabble in neural nets and AI modeling as well? I predominately use YALE, an open source data mining, machine learning software to build currency, futures, and stock models.
This helps me identify emerging market trends and trade (discretionary) accordingly.
Funny question to ask, as a few days ago I had a few moments to talk with a trader to-be. Why him? Why not a trader? I wanted the insights of somebody who has fresh eye on things. Sometimes youth is not to be under-estimated. At the moment he works as a business analyst (finance) and he will be switching jobs to becoming a trader fulltime. I have known this person for a while and I trust his opinion as he is no-nonsense.
My Suspicion of Models
Here is the question I asked:
I have been looking at options for equities, and understand that the implied volatility is calculated by doing a reverse calculation of Black-Scholes equation. The resulting implied vol is a number that you need to compare with a model. Some models use historical vol, others use a more complicated model. Regardless of model you are always doing a correlation between the implied vol, and the vol of the model. Since the correlation is imperfect, even though many create extremely complicated models, is the trick in all of this the decision based on the data of the model and implied vol? Here is my thinking, if I have one model that is plain vanilla simple, then a more complicated sophisticated model is not going to help because you cannot do a prefect correlation and it is always up to the trader to make the right decisions.
Well it depends if you are talking structured products or plain vanilla products [Reader note: we are talking about plain vanilla products like options on equities]. If you are talking plain vanilla products yes the problem of models is that they don’t give you any extra insight. The trader will look at whatever model there is and tweak the numbers to see the effects. The trader is the individual who will make the money and not the model.
This is a suspicion I have been having for the past four months. Indicators and models are great, but a more complicated indicator is not going to give you any insight into the market. Putting it into a simpler context, there are all of these fancy stock indicators like Rabbit Q-Rank, but they are not going to help you. At the end of the day its a crap shot and its up to the trader to know what is right or wrong.
What I Really Think of Complicated Models
You might as well use a simple moving average because anything else will probably not help. So now comes the question, if these indicators are useless why are there so many? I think, because many feel they are better traders with their own concoction of special indicator even though to a large degree its all snake oil! I am not saying all indicators are bad. What I am saying is that if you read the book Technical Analysis of the Financial Markets you have all you need.
In this train of thought I include those who do long term investing using P/E, cashflow, and other indicators. All of these pieces of data are just that data that bear no relevance on the future performance of the company. If you don’t agree with me look up the philosophical “all swans are white” question. The indicators are not the important pieces, whereas the strategy based on the indicators is. Otherwise we would have more long-term Buffet wealthy investors. Instead we have many “The Only 3 Rules to Financial Success” books.
When I was talking to the trader-to-be he said the following.
The reason why people like these models, patterns, and indicators is that they sometimes work. The market is in-efficient often enough to give you a sense of confidence.
This comment was re-affirmed by a book I am reading called “Fooled by Randomness.” Indicators, patterns are interesting because they are amazing hind-sight tools. In hind sight you will always see why something went up or down. Yet when confronted with the moment you will not know if the process is going up or down. Indicators give you a warm and fuzzy feeling because they say the process will go in one direction or not in your moment of indecisiveness.
Though, now comes the question what is important when trading? My thinking is that the model is there to serve as a mechanism to spit out numbers. The model only needs to be reasonable thus Black-Scholes, or SMA indicators are good enough. What is more important is taking the data from the models and building a strategy. Of course many will say, “duh! tell me something I did not know.”
Well, yes right, tell you something that you did not know. The reason why I am being explicit is because when building automated trading systems, or trading systems in general there seems to be a pre-occupation of using algorithms to build models. For example the book Volatility and Correlation while interesting is a heavy read. Ok let me be frank I read the book got some interesting insights, but did not understand half of the math. The focus of the book is to make sense of implied vol and your model’s vol and trying to make a connection between the two as per the quote from the beginning of the book.
I do give a lot of importance to the congruence between the outputs of the model and the available empirical evidence.
Yet reading one of the Amazon comments you have to wonder if this is a more complicated snake oil.
Although the author warns the reader in the Preface, that because he ran out of pages (come on it is more than 800!) he omitted dealing with Copulas, it is still a pity that a book about correlation does not present at least a small chapter on this new (state-of-the-art) area.
Everything else is very good, solid material with a good balance between maths surrounding the topic, explanations and worked out examples.
I highlighted that comment because it implied a book of 800 pages published 3 years ago is out of date because he did not deal with Copulas (1,2,3)! Oh gee bite me in the butt for being “behind the times.” Where is that Copulas book that we all need to make us better traders! Oooh no snake oil being sold here folks (sarcasm)!
Ok, so I am slamming and dissing people with PH’D’s that happen to get Nobel prizes, and traders that are obviously richer than I. Since I mentioned that I could not understand a book is it me or the book? And who am I? A nobody, a spec of dust!
I am saying this because when I investigate a field I always try to learn from first principles. It is not that I will do everything from first principles, I just want to know what I am dealing with. I adhere to the rule in life that things need not be simple, but you should not overcomplicate things. It seems to me that by trying to model the market many keep developing more complicated models to get that last correlation.
The Problem Of Probable Statistics
My real suspicion is that if more and more models are modeling the market the market will become less modelable because the interaction of the models have not been accounted for.
Let’s say I have this big vat of red and black balls. If I pick two balls from the vat where one is red, and one black my initial distribution analysis of red and black balls would be fifty-fifty. Yet my confidence would not be high because I only picked two balls. To become more confident I would need to pick more balls. And as time continued even though I did not know the true distribution or red and black balls I can get a reasonable impression without having to count all balls.
This sort of statistics works well, and is applied in many many places with manufacturing being the prime example. Yet you cannot apply this sort of statistics to a market because the market is not a process. In Fooled By Randomness the author describes the problem of the mischievous child.
Again consider the big vat of red and black balls. As you are picking out the balls and creating a probability the child adds red or black balls as the child sees fit. This means if you think there are 50-50 red and black balls a mischievous child might double the number of red balls completely blowing your probability numbers.
This behavior is used by some to account why the market has more “rare”, and “random” events than real life. You need to ask the question that if a child is mischievous how can you model the market?
What I Think is Important
The problem is that the market is a stochastic process and thus mathematics based on continuous math are not going to help you. Look at the last earnings of Amazon and Microsoft. Did anybody see that happening? Or how about the collapse of Yahoo? Did anybody see that happening? I think to build a successful trading system you need to focus on the strategy, again a no brainer moment.
Though since strategy is a no-brainer why are we thinking about Neural Nets?
In wikipedia it says the following about Neural Nets:
The tasks to which artificial neural networks are applied tend to fall within the following broad categories:
Neural nets are great for patterns, and prediction, but yet the markets are stochastic, which implies the following:
A stochastic process, or sometimes random process, is the counterpart of a deterministic process (or deterministic system) considered in probability theory. Instead of dealing only with one possible ‘reality’ of how the process might evolve under time (as it is the case for solutions of an ordinary differential equation, just as an example), in a random process there is some indeterminacy in its future evolution described by probability distributions. This means that even if the intial condition (or starting point) is known, there are more possibilities the process might go to, but some paths are more probable and others less.
You can’t have it both ways. You can’t use technologies that is geared towards patterns on a process that is not based on patterns. I think the simplest correlation is trying to use Neural Nets to predict the outcome of the past show called Sliders. To be able to deal with multiple realities you need to be able to reason based on a situation. Yes you can tweak and twiddle a neural net to be able to deduce, but I ask why fit a square peg into a round hole?
I think people are using Neural Nets because they give us a warm and fuzzy feeling. It is the thinking that if I can train a net on the past then magically it will figure out similar, but different situations in future.
Though, why reasoning? Reasoning is what humans do best. Humans and traders are able to take to dissimilar topics and make sense of it. Since the magic of the markets are the traders which happen to be human the issue is deduction, even if half of the traders are automated. In the book Evidence Based Technical Analysis which moves technical analysis from snake oil to craftsmanship there is an all too short discussion of combined rules. It seemed the book was handwaving, even though saying combined rules have shown to be effective. Why the handwaving? I think not enough has been done bridging the model world with the reasoning and decision making world.
And that is where I am bridging the model world with the reasoning and decision making world. Will it work? I don’t really know, but I feel it is moving me a step closer to my goal of a trading system.