• What's the
    problem?
  • Why is that
    so important?
  • What about
    solutions?
  • Can you show
    me an example?
  • About the
    Author



Q:
Where is the wisdom we have lost in knowledge?
Where is the knowledge we have lost in information?


Choruses from the Rock 1934



































































Identifying patterns in group behaviour

In an uncertain world assumptions are inevitable, but survival also depends on avoiding bad assumptions. Uncertainty and fear are generated when fundamental assumptions are proved unsound by events, or discoveries.

When dealing with constant or slowly changing processes there is a strong incentive to patch over the unknown by declaring it to be random, because truly random processes have clear mathematical properties.

To say that something is random is actually to state something quite definite, namely that these properties apply. To many this will come as quite a shock, because that is not how it is commonly understood.

Casino owners use these very properties to part the unwise from their wealth. But these properties only apply to truly random processes, and disappear if the coin is a double header, or if the dice are loaded, because that would produce patterns. The Casino owners actually have an interest in keeping their game clean, because patterns are detectable, and therefore valuable .

Finding patterns, in Casinos or elsewhere is not of itself a difficult thing, doing it quickly is what matters, because it means that wider searches of greater complexity can be performed.

Our original brief was to find meaningful stimuli for price rises in any one of thirty seven currency exchange rates, working minute to minute, and only with the actual prices.

Markets are meant to follow random processes, meaning there should be no stimulus patterns detectable at meaningful levels, say one in ten. The more daunting aspect was that just thirty seven pairs, can generate a staggering 137,438,953,472 different combinations.

The ParaCudaR System

'Para' alongside, 'Cuda', 'R'easoning

Our software is a fully operational Linux CUDA shared library for binary pattern finding; astonishing acceleration has been achieved by using parallel hardware from NVIDIA. Because patterns can only be parallel it's simply the right tool for the job.

More importantly, we found stimuli patterns wherever we looked, spiking where it made sense for there to be spikes. The results are laid out in some detail at the following link.

Done and dusted - myth busted


In May 2002, brokerage firm
Charles Schwab Corporation ran a
television advertisement pointing
out Wall Street brokerage firms'
conflicts of interest by showing
an unidentified sales manager
telling his salesmen,

"Let's put some lipstick on this pig!"

May 25 2012, Madrid. Bankia revised
its earnings statement for 2011,
stating that instead of a
profit of 309 million euros,
it had in fact lost 4.3 billion euros.

" All that glisters is not gold "

The Merchant of Venice II.vii

Once a problem is reduced to a formula it loses its spooky nature, it is admitted as an equal to the body of accepted wisdom, and it becomes usable as a base assumption in further constructions.

Classifying a process as random when it could equally be described as "not yet known" can therefore have consequences far beyond the obvious. Nowhere is this more clearly seen than in the derivatives sector of the finance industry, whose central guiding equation postulated that returns resemble flipping coins, and that there was a riskless rate of return, whose proxy on earth was the yield on U.S. Government debt.

This toxic pair of false assumptions made risk appear as certainty, enabling cheap and plentiful credit, for every size of borrower, from government to citizen.

The false certainty of the penny flipper evaporated on August 9th 2007 when France's BNP Paribas announced that it could not value the assets held by three of its hedge funds; the riskless rate assumption was exposed in 2011 when Standard & Poor's downgraded US sovereign debt to AA+, while both Moody's and Fitch put it on negative outlook.

ParaCudaR vs. RapidMiner
IBM T40I10D100k test data
The greater the range to cover, the greater the acceleration achieved.


























Of course, all patterns are parallel, and all parallels are patterns!

The CUDA hardware is explicitly parallel, with the result that the ParaCudaR prototype running on this server completes a standard IBM generated test more than 100x faster than the industry leader, RapidMiner , as shown on the left here.

<figcaption >

The NVIDIA GTX Graphics card, retailing for less than $150.

This hardware is the lowest end of the CUDA range, inter-linked CUDA hardware presents enormous opportunities for those able to harness the power.

NVIDIA is a leader in the computer games market, which demands hardware that updates pixels fast, and at the same time. It has also sought to expand the market for their Graphical Processor Units ( GPU ) into other areas, such as video editing, scientific simulation, and cryptanalysis, where the parallel nature of the task is constrained by sequential execution.

Data-mining is just such an area, but to date few GPU enhanced commercial applications have appeared. Problems have to be completely re-formulated in order to benefit from parallel execution, but the speed improvement can be substantial.

STOP PRESS May 16 2012 Chip maker Nvidia has revealed details of a new graphics processing unit (GPU) which it says will create the world's most powerful computer. Thousands of the firm's Tesla K20 modules will be fitted to an existing supercomputer at Oak Ridge National Laboratory in Tennessee, US.

The real world that economics
should model involves at least
the exchange of goods, the filling
and emptying of the shopping cart.



The goods need not be physical,
they could be digital, just as
the currency with which they
are settled.


The recording of the transaction
provides the most detailed
source of information for
understanding buyer behaviour.



Supermarkets have long used this
data to target products, by physical
product placement within the store,
or by promotional discounts in
their web stores.

The technique of Association Rules induction
is well understood, but not commercially
available via web services until now.
A web service also offering performance
comparable to the ParaCudaR remains a threat
against which only further investment can protect.

Naturally supermarkets keep their customer
profiling information rather close to their
chests.

This makes it difficult to generate example
data that has the smack of reality about it,
exactly the reality you hope to capture.
However, there is a market that produces,
minute by minute, cart loads of clean ,
timestamped, data. It's called the Capital
Market.

In the Capital Market there is one shopping
trolley that can act as a proxy for the market
as a whole, whose contents are divined thus..

"If the price is up they are buying it".
"If they're buying it then it's on the trolley"

We lined up trolleys for the currency market, one
for each minute back to Lehmans, and checked it out.

You can explore the results here!



Chris Painter is a
freelance researcher
in the field of
artificial intelligence
and finance.

1972-1976 Exeter College Oxford
M.A Oxon. Classics, majoring
in Ancient History and Philosophy

1976-1977 Chemical Bank London
Corporate Bank Trainee,
Completed formal credit analysis
course, but by 1977 UK PLC was
in lockdown, and corporate
lending stopped.
1977-1979 Manufacturers Hanover Ltd.
Assistant to Director of Eurodollar
Syndication for COMECON states
and Scandinavian multinationals

1979-1981 Burmeister and Wain
Copenhagen. Headhunted as deputy
treasurer responsible for non-Kroner
liabilities. Devised an arbitrage
between state bonds and ship
industry support loans which
reduced the cost to the purchaser
by one third. This enabled the sale
of six Panamax bulk carriers to
Wheelock Marden, which in turn
saved Copenhagen's
largest employer.

1981-1983 Realising that
the future of such deals was
mechanical, I visited the
States to check out the
new marvel, the IBM PC.

On return to the UK I
got my hands on an Apple II
and a fantastic wife, and
settled down to program
the guts out of the Black
Scholes option equation.
Started Enigma Systems to produce
and market financial software

1983-1985 Hoare Govett London
Assistant to Geoffrey
Chamberlain, Options
Director. HG bought the
Enigma realtime valuation
and analysis package,
and my services fulltime.
That was fine until I wanted
to try non-standard techniques
that did not involve the
Black Scholes premises.
....read on above.

In 1986 I started to show the results of a program that identified risk free trades on the London Traded Options Market, the London Stock Exchange's equity options market. This was a crowd pleaser at the time because nobody understood Black Scholes, and all hated using it, but it did also pose an alternative to the statistical approach emanating from Chicago, namely reasoning.

The program seemed to suggest that markets were far from efficient. I was asked to do a stream of consciousness for International Banking Review 1986, which now seems strangely prophetic . I'm the last article, under the title "The reason for machines", and the first was by Karl Otto Pohl, the Chairman of the Bundesbank, his title was "Global co-operation in monetary policy"!

The techniques advanced to solving problems considered beyond solution, such as hedging and margining. In fact they were just an evolution of ideas formed in Denmark about how to combine disparate instruments to advantage, but they were effective nevertheless. I was asked to present them at a conference, and a transcription of my commentary survives

My chapter ends with words to the effect that the statistical approach is a dead end. Everyone is proved right in the end! In my case it was twenty years, during which the number magicians held sway.

The alternative approach didn't stand still over that period, the game changer turned out to come from an unlikely source, video cards. This enabled parallel execution on a massive scale. So now we can check the combinations in parallel.