In an uncertain world assumptions are inevitable, but survival also depends on avoiding bad assumptions. Uncertainty and fear are generated when fundamental assumptions are proved unsound by events, or discoveries.
When dealing with constant or slowly changing processes there is a strong incentive to patch over the unknown by declaring it to be random, because truly random processes have clear mathematical properties.
To say that something is random is actually to state something quite definite, namely that these properties apply. To many this will come as quite a shock, because that is not how it is commonly understood.
Casino owners use these very properties to part the unwise from their wealth. But these properties only apply to truly random processes, and disappear if the coin is a double header, or if the dice are loaded, because that would produce patterns. The Casino owners actually have an interest in keeping their game clean, because patterns are detectable, and therefore valuable .
Finding patterns, in Casinos or elsewhere is not of itself a difficult thing, doing it quickly is what matters, because it means that wider searches of greater complexity can be performed.
Our original brief was to find meaningful stimuli for price rises in any one of thirty seven currency exchange rates, working minute to minute, and only with the actual prices.
Markets are meant to follow random processes, meaning there should be no stimulus patterns detectable at meaningful levels, say one in ten. The more daunting aspect was that just thirty seven pairs, can generate a staggering 137,438,953,472 different combinations.
The ParaCudaR System
'Para' alongside, 'Cuda', 'R'easoning
Our software is a fully operational Linux CUDA shared library for binary pattern finding; astonishing acceleration has been achieved by using parallel hardware from NVIDIA. Because patterns can only be parallel it's simply the right tool for the job.
More importantly, we found stimuli patterns wherever we looked, spiking where it made sense for there to be spikes. The results are laid out in some detail at the following link.
Once a problem is reduced to a formula it loses its spooky nature, it is admitted as an equal to the body of accepted wisdom, and it becomes usable as a base assumption in further constructions.
Classifying a process as random when it could equally be described as "not yet known" can therefore have consequences far beyond the obvious. Nowhere is this more clearly seen than in the derivatives sector of the finance industry, whose central guiding equation postulated that returns resemble flipping coins, and that there was a riskless rate of return, whose proxy on earth was the yield on U.S. Government debt.
This toxic pair of false assumptions made risk appear as certainty, enabling cheap and plentiful credit, for every size of borrower, from government to citizen.
The false certainty of the penny flipper evaporated on August 9th 2007 when France's BNP Paribas announced that it could not value the assets held by three of its hedge funds; the riskless rate assumption was exposed in 2011 when Standard & Poor's downgraded US sovereign debt to AA+, while both Moody's and Fitch put it on negative outlook.
The CUDA hardware is explicitly parallel, with the result that the ParaCudaR prototype running on this server completes a standard IBM generated test more than 100x faster than the industry leader, RapidMiner , as shown on the left here.
This hardware is the lowest end of the CUDA range, inter-linked CUDA hardware presents enormous opportunities for those able to harness the power.
NVIDIA is a leader in the computer games market, which demands hardware that updates pixels fast, and at the same time. It has also sought to expand the market for their Graphical Processor Units ( GPU ) into other areas, such as video editing, scientific simulation, and cryptanalysis, where the parallel nature of the task is constrained by sequential execution.
Data-mining is just such an area, but to date few GPU enhanced commercial applications have appeared. Problems have to be completely re-formulated in order to benefit from parallel execution, but the speed improvement can be substantial.
STOP PRESS May 16 2012 Chip maker Nvidia has revealed details of a new graphics processing unit (GPU) which it says will create the world's most powerful computer. Thousands of the firm's Tesla K20 modules will be fitted to an existing supercomputer at Oak Ridge National Laboratory in Tennessee, US.
In 1986 I started to show the results of a program that identified risk free trades on the London Traded Options Market, the London Stock Exchange's equity options market. This was a crowd pleaser at the time because nobody understood Black Scholes, and all hated using it, but it did also pose an alternative to the statistical approach emanating from Chicago, namely reasoning.
The program seemed to suggest that markets were far from efficient. I was asked to do a stream of consciousness for International Banking Review 1986, which now seems strangely prophetic . I'm the last article, under the title "The reason for machines", and the first was by Karl Otto Pohl, the Chairman of the Bundesbank, his title was "Global co-operation in monetary policy"!
The techniques advanced to solving problems considered beyond solution, such as hedging and margining. In fact they were just an evolution of ideas formed in Denmark about how to combine disparate instruments to advantage, but they were effective nevertheless. I was asked to present them at a conference, and a transcription of my commentary survives
My chapter ends with words to the effect that the statistical approach is a dead end. Everyone is proved right in the end! In my case it was twenty years, during which the number magicians held sway.
The alternative approach didn't stand still over that period, the game changer turned out to come from an unlikely source, video cards. This enabled parallel execution on a massive scale. So now we can check the combinations in parallel.