How to "uncluster" a set of financial data?

External question — Quantitative Finance Stack Exchange Author: Mike Furlender Original post: https://quant.stackexchange.com/questions/1947 License: CC BY-SA 3.0 — https://creativecommons.org/licenses/by-sa/3.0/ Adaptation: HTML converted to plain text; contact email addresses removed. I am attempting to evaluate and compare the profit factor of different "test runs" of a FOREX trading strategy. My problem is that, despite an average time between orders of 2hr+, some of these runs can have 20+ orders in a row, every 5 minutes, in the same direction. I need some way to normalize these clusters that occur without just throwing the data out. I want to treat the cluster as 1 data point by averaging the gain/loss of each trade within the cluster. I was thinking of doing it with a moving time window in the following manner: For each order: Weight = 1/(N+1) N = count of consecutive orders within 30 minutes of the current order. But I am not sure if that is correct.

External answer — Quantitative Finance Stack Exchange Author: Tal Fishman Original post: https://quant.stackexchange.com/a/1953 License: CC BY-SA 3.0 — https://creativecommons.org/licenses/by-sa/3.0/ Adaptation: HTML converted to plain text; contact email addresses removed. That is definitely not correct. Your test results will suffer from look-ahead bias. At the point in time that you will be entering your order, you will not know yet how many orders will come in the next 30 minutes, thus a proper backtest should not use that information to size the trade. There are many factors to consider when running a proper backtest (see wikipedia ), but the key factor behind all of them is to replicate as closely as possible what is realizable in live trading. In your case, that means that you will have to predict how many trades your strategy is about to enter and size all subsequent trades appropriately. Once you reach your "limit," at which point in real trading you would have run out of capital, your backtest is effectively prevented from entering new trades. If your strategy isn't able to predict whether trades are about to cluster, you could try an incremental system, whereby the first trade is assigned a proportion $\delta$ of total capital, the second gets $\delta(1-\delta)$, etc.

External answer — Quantitative Finance Stack Exchange Author: icequations Original post: https://quant.stackexchange.com/a/2225 License: CC BY-SA 3.0 — https://creativecommons.org/licenses/by-sa/3.0/ Adaptation: HTML converted to plain text; contact email addresses removed. you could perhaps cluster the information in a candle stick manner and bin the price data into high, low and closing, instead of throwing some data out and keeping only its closing price. Using the high, low and closing prices you are able to make 3 separate estimates with regards to Profit and lost. Obviously with a long position, profit calculated by entering at a high price while using a low price for its exit would be the pessimistic estimate of profit.