News Stay informed about the latest enterprise technology news and product updates.

Data mining techniques for financial markets

This article, the third in a series, examines random and predictable events in financial data mining efforts and reviews the author's experience with data mining in 30-year U.S. Treasury Bond futures.

This article originally appeared on the BeyeNETWORK.

Random Data

Until recently, many people argued that the movements of prices in the financial markets are random. Many researchers now confirm what practitioners have known for years. Price movements are not random. Few would argue against the existence of trends and trading ranges in the financial markets. The occurrence of these characteristics is sufficient to demonstrate that the markets are not a series of random price movements.

The question then reverts to whether the market is predictable. The logical answer is that predictability depends upon the talent of the predictor. The financial markets have the potential to be predictable. When market change is predicted successfully, the trader can reap great rewards. That is why so many people are pursuing data mining efforts.

Based on the characteristics of behavioral systems and the constraints of data mining technology, it is fair to say that the markets are partially predictable. In many cases, there is insufficient information to make a reliable prediction. Where predictive capability exists, there is no certainty. Only the probabilistic reaction to a set of conditions is attainable.

If we accept the tenets of Chaos theory that are associated with a dependence upon initial conditions, we must accept that after some finite number of steps into the future, the divergence of complex systems with even small initial differences makes it impossible to predict accurately. The ability to predict, with a high level of precision, what the price of an instrument is going to be becomes more difficult the further out in time we move.

That does not imply that the Efficient Market Hypothesis and Portfolio Theory are correct. It also does not support the Random Walk Theory of market activity. With increasing regularity, diverse markets are shown to exhibit behaviors that fail to support these self-serving proposals. The price activity of almost all instruments is not random and is not normally distributed. The simple existence of trends in prices is sufficient to disprove a normality or randomness assumption. The impact of this reality is beyond the scope of this article, but needs to be understood by the successful trader.

This realistic view of our knowledge is mandatory if we are to trade successfully. Understanding what we can rely on, and what is unreliable, is critical. Having the discipline to act on the reliable, rather than on emotion, is essential to financial survival.

Random Events

The existence of random events that impact the financial markets is unquestioned. No one can account for the occurrence of events such as the assassination of a president, an Oklahoma City bombing or weather catastrophes.

When these types of events occur, one or more markets may move dramatically. Participants in those markets often experience huge swings in their financial position. Little can be done to enhance or protect oneself from the events.

Predictable Events

In contrast, many events are incorrectly portrayed as random. Shocks due to news may or may not be predictable. In a large number of cases, the content of the news is unknown, but the timing of the occurrence is known well in advance.

No market participant should complain about getting “caught” by the release of a scheduled government report. These, and many other events, are scheduled well in advance and offer the market participant the opportunity to establish an appropriate position.

The characteristics of these events can be modeled. Whether this modeling is a part of the construction of an objective function, captured in a rule-based system or modeled as a series of components in a more comprehensive system, the data mining practitioner should incorporate them effectively.

Common Knowledge

Knowing what everyone else knows offers little advantage anywhere. This is especially true in the financial markets. Our profitable trading opportunities come from possessing rare information. Conversely, not knowing what everyone else knows can be very painful.

Knowing that a particular government report, released four hours ago, had a particular value might help you explain recent price moves. That knowledge would do little to help you determine the movements for the coming hour.

Identifying and analyzing data in a manner that gives you special insights has a great deal of value. The entire data mining process in the financial markets is driven by that reality.

In most cases, insight is simply a matter of clearly recognizing an opportunity for what it is and then reacting in an appropriate manner. One relatively low-tech example of that concept involves a report that becomes available at a fixed time each week. The report is available by fax and is widely followed. A few insightful traders realized that a particular piece of fax software showed the report on their computer monitor as it was being received. The traditional fax machine receives the entire page before printing it out. Utilizing that software provided a few seconds of unique knowledge that more than paid for the computer and software in the first week of use.

The Application of Advanced Technology

The application of advanced technology to data mining in the financial markets offers many exciting opportunities. It should be remembered, however, that there are no “silver bullets.” As with any tool, they can be misused. And, generally, the more advanced the tools, the more likely they are to be used incorrectly.

Many market participants, in their rush to take profits, refuse to do the preparatory work necessary. There are many necessary and profitable activities using “remedial” technologies. Only when the basics are fully utilized should the advance techniques be considered. It is hard to imagine a trader successfully entering any market unaware of whether the market trades in thirty-seconds or sixty-fourths. It happens. And other critical aspects of understanding are just as likely to be considered unimportant in the rush to take advantage of the newest technique to capture the attention of the market.

Neural networks, genetic algorithms, chaos theory, fuzzy logic and expert systems all have their strengths. Many of the basic technologies like technical analysis and linear statistics also have something significant to offer. The key is in using each piece effectively. Some will have more value than others for your particular purposes. All have value in developing a clearer understanding of the financial markets. This is why the interest in data mining continues to develop.

Build an inventory of valid techniques that fit the way you trade. Understand your business. Trade your information, not the noise. By using the solid pieces of knowledge, gained over time, your chances of success are greatly enhanced. Your research should lead you to expand the knowledge you have accumulated. This may include adding new financial instruments or new techniques to your library of mined knowledge.

Staking a Claim in Bond Futures

Over a period of six years, the author has developed an obsession with predicting behavior in the 30-year U.S. Treasury Bond. I have had the pleasure of the insight of some exceptional people along the way. They have contributed to the development of my understanding of data mining in the financial markets generally and specifically to the profitable trading of bonds.

The general structure for my data mining efforts has been relatively stable the past two years. However, it took four-plus years of full-time effort to comfortably define what I was trying to do with the bond market.

My stated goal is to average four ticks profit per day. I look at this average quarterly. A tick in the Bond futures is one-thirty second of a point. Each point is worth $1,000. Assuming 20 trading days monthly, that equates to approximately 240 ticks profit per quarter, or about $7,500 profit per quarter, after commissions. These assumptions are based on trading one contract. Trading one contract normally requires $2,500 in a trading account. This represents an annual return of $30,000 on capital of $2,500.

Achieving that profit level is done with a number of constraints. I have restricted my trading to Bond futures and utilize only the day session. I do not use options for this program. I am willing to trade on multiple time frames: intra-day, daily, weekly. I use price targets, profit targets and stops. I am out of the market for major scheduled news events.

I initially began trying to predict price. The reality was that even very good accuracy could not be traded. I quickly began to look for directional moves in the market: up, down, sideways. More recently, I have devoted significant efforts to predicting price ranges. Each type of predictive capability has its own uses and strengths.

My data mining efforts use neural networks and chaos theory extensively. After some initial success on paper, I began the brutal experience of live trading. I was fortunate to have an advisor with real money management experience. He stressed that the single most important priority in trading was restricting your activity so that you could continue to trade tomorrow. In contrast to conventional wisdom, there will be new opportunities tomorrow.

Trading a small number of contracts relative to what a trading account is capable of handling is a guaranteed way of being able to trade tomorrow. By never putting more than 5% at risk on a single trade, and never having more than 10% at risk at one time, I am comfortable with being able to continue trading indefinitely.

Each trader has a risk tolerance level. Some people will argue that I could make more profit with higher levels. Others will suggest that I have too much at risk, or that I am too arbitrary. As pointed out earlier, traders develop their own standards. At these levels, I am comfortably profitable, and I can sleep at night.

My initial experience left a lot to be desired. Interpret that as, “I lost money.” It quickly became apparent that knowledge was the key to success. Not just the superficial knowledge that the noise of the market and historical data mining offer. Rather, the successful trader needs real knowledge of the market that comes from in-depth analysis and testing. I reverted to the basics. I started spending my time searching for variables that added information content or transformations that made the information content easier to extract from the data. In the process, I became an information trader.

My understanding developed over time. In the process, I began to build a library of situations of predictable behaviors and expectations of the participants in the Bond market.

Like many data mining practitioners, I had concluded that Fridays are significantly different than other trading days. This superficial knowledge is rediscovered and confirmed regularly. By studying the market, I gained an understanding of why the behavior of the market participants was different on Friday. Trading Bond futures is, in essence, trading interest rates. What has an effect on interest rates also has an effect on Bond prices. The simple fact is that many scheduled government reports that affect interest rates are released on Friday. These are known events. However, if a trader blindly focuses on the data, he sees only the impact of the event and cannot anticipate it.

The basics, like knowing the average daily price range for the past month, the past quarter or the past five years, significantly enhance your opportunity for profit. Attempting to extract a one point profit (32 ticks) from a position using an intra-day trade is noteworthy. Knowing that non-news days for the past year have an average daily range of 18 ticks makes an information trader question the expected probability of success.

Over time, I have reintroduced the advanced technologies, drawing on their strengths. The principle of strange attractors, from chaos theory, has contributed to my range predictions, to my trend classification capabilities and to my ability to consider multiple time frames. Neural computing has helped me identify variables that consistently exhibit strong information content, examine the interaction among variables and consider the non-linear aspects of the data. I have utilized the capabilities of expert systems, genetic algorithms and fuzzy logic as well.

None of these technologies are a panacea. The data mining practitioner has a box of tools and must understand how to use each appropriately and effectively. This involves taking what has value to the trader and discarding the rest. In mining the data, the practitioner is better served by using traditional techniques well than by using advanced technology poorly.


Successful mining of data in the financial market requires extensive effort and a diverse set of skills. The potential rewards are significant. This article has examined a number of key issues. Inappropriate handling of these issues has hindered the data mining performance of many traders. Trading information, rather than noise, involves developing a well-defined set of skills that meet the needs of the individual trader. A clear understanding of the objective is critical. Enhancing the information content of your data adds value. Applying diverse techniques, new and old, enhances the data miner’s library.

As the number of techniques in your library grows, so do the number of opportunities you have to trade. You do not have to trade. You should be driven by the need to reduce your trading activity to high quality opportunities, not by the urgency to take a position to justify the time you have spent watching the market. There is no need to become one of the casualties of the markets.

Dig Deeper on Business intelligence data mining

Start the conversation

Send me notifications when other members comment.

Please create a username to comment.