Rare Events in Normal Multivariate distributions
Rare Events in Normal Multivariate distributions
Loading saved threads...
Alex · External communityPost link
External question — Quantitative Finance Stack Exchange
Author: Alex
Original post: https://quant.stackexchange.com/questions/44955
License: CC BY-SA 4.0 — https://creativecommons.org/licenses/by-sa/4.0/
Adaptation: HTML converted to plain text; contact email addresses removed.
I don't work in finance, but I've stumbled upon a problem that you guys may have to deal with in your jobs.
My problem is a random walk in high dim spaces ( > 100), in which I'm looking for vectors that explore a limited portion of that hyperspace. An event is labelled a posteriori, with no indications allowing to attribute such event to a subset of the vector components.
Examination of a set of rare events show that 'some' vector components slightly deviate from a normal distribution (higher central peak, side peaks), and/or that their distribution has changed shape. But it's extremely difficult to pull a dominant factor, let alone find a subset of components that would allow to limit the scope of investigation, or even create an event fingerprint.
Put it simply: given a normal multivariate distribution, what algorithm, or investigation approach, would you recommend to classify (future) rare events (say occurence of 1/10), provided one has an history of all previous events, and in that history, all rare events are labelled.
In your day to day work, such rare events could be those preceding or indicative of an imminent stock-price drop.
Note:
KMeans clusterisation, Random Forest model, Bayes where tried, but did not allow to make any breakthrough
The assumption is that the distribution is normal multivariate. It looks it does, all vector components have a Gaussian distribution and are uncorrelated. When rare events occur, some vectors do deviate from a Gaussian though.
EDIT: My question didn't have much success, so I'll try to reformulate in a different way. Say there's a trading desk, with 1000 brokers, and their operation are well approximated using a multivariate normal distribution. In that desk, there are 5 brokers who on top of their normal operations, decide to do something else:
each guy picks a random number, p, if p <= p0, he places a different-order, following a normal distribution different from the one he normally has. If p > p0, he uses the originally assigned distribution.
Someone at the risk dept. classifies each batch of a 100 orders coming from the trading desk, using an algorithm of his own.
If p0 is small (otherwise, that's easy), How would you detect who the 5 rogue-guys are, and how would you identify the abnormal orders, if you had all the time series of the trading desk orders, labelled as good vs. suspicious orders?
Quote
Report
Dave · External communityPost link
External answer — Quantitative Finance Stack Exchange
Author: Dave
Original post: https://quant.stackexchange.com/a/69309
License: CC BY-SA 4.0 — https://creativecommons.org/licenses/by-sa/4.0/
Adaptation: HTML converted to plain text; contact email addresses removed.
If you’re comfortable making the assumption of multivariate normality (I’m not sure that you are), then this seems like a perfect place to use Mahalanobis distance.
One of the first facts that statistics students learn is the “empirical rule”: that
$\sim 68\%$
of the density is within one standard deviation of the mean and
$\sim 95\%$
of the density is within two standard deviations of the mean.
In high dimensions, it gets more complicated, but we can do something similar. Instead of measuring how many standard deviations we are from the mean, we measure how many units of Mahalanobis distance we are from the mean. In some sense, standard deviation is the univariate special case of Mahalanobis distance.
For better or for worse, the number of units of Mahalanobis distance containing a proportion of the density depends on the dimension
$d$
. We can calculate the exact number from quantiles of a
$\chi^2_d$
distribution and then taking the square root.
If you want to contain the central
$95\%$
in one dimension, find the
$0.95$
quantile of
$\chi^2_1$
and take the square root. In R software:
sqrt(qchisq(0.95, 1)) # I get the expected ~1.96
You mentioned that rare events for your purposes happen
$1$
out of
$10$
times. If you want to know how many units of Mahalanobis distance you need to be to have such a rare event, determine your dimension, d, and consult a
$\chi^2_d$
distribution:
sqrt(qchisq(0.9, d))
As with the univariate empirical rule, this can fail miserably when the (multivariate, not just marginal) Gaussian assumption is violated, so be careful, and don’t let
a common misinterpretation of the central limit theorem mislead you into thinking that a large sample size saves you.
Mahalanobis Distance
$$D_{M}({\vec {x}})={\sqrt {({\vec {x}}-{\vec {\mu }})^{\mathsf {T}}\mathbf {S} ^{-1}({\vec {x}}-{\vec {\mu }})}}$$
Quote
Report
Post Reply
Quoted from Forex.com.bd-Editorial External question — Quantitative Finance Stack Exchange Author: Alex Source score (net votes, not local likes): 3 Original post: https://quant.stackexchange.com/questions/44955 License: CC BY-SA 4.0 — https://creativecommons.org/licenses/by-sa/4.0/ Adaptation: HTML converted to plain text; contact email addresses removed. I don't work in finance, but I've stumbled upon a problem that you guys may have to deal with in your jobs. My problem is a random walk in high dim spaces ( > 100), in which I'm looking for vectors that explore a limited portion of that hyperspace. An event is labelled a posteriori, with no indications allowing to attribute such event to a subset of the vector components. Examination of a set of rare events show that 'some' vector components slightly deviate from a normal distribution (higher central peak, side peaks), and/or that their distribution has changed shape. But it's extremely difficult to pull a dominant factor, let alone find a subset of components that would allow to limit the scope of investigation, or even create an event fingerprint. Put it simply: given a normal multivariate distribution, what algorithm, or investigation approach, would you recommend to classify (future) rare events (say occurence of 1/10), provided one has an history of all previous events, and in that history, all rare events are labelled. In your day to day work, such rare events could be those preceding or indicative of an imminent stock-price drop. Note: KMeans clusterisation, Random Forest model, Bayes where tried, but did not allow to make any breakthrough The assumption is that the distribution is normal multivariate. It looks it does, all vector components have a Gaussian distribution and are uncorrelated. When rare events occur, some vectors do deviate from a Gaussian though. EDIT: My question didn't have much success, so I'll try to reformulate in a different way. Say there's a trading desk, with 1000 brokers, and their operation are well approximated using a multivariate normal distribution. In that desk, there are 5 brokers who on top of their normal operations, decide to do something else: each guy picks a random number, p, if p <= p0, he places a different-order, following a normal distribution different from the one he normally has. If p > p0, he uses the originally assigned distribution. Someone at the risk dept. classifies each batch of a 100 orders coming from the trading desk, using an algorithm of his own. If p0 is small (otherwise, that's easy), How would you detect who the 5 rogue-guys are, and how would you identify the abnormal orders, if you had all the time series of the trading desk orders, labelled as good vs. suspicious orders?
Checking account access…