Change feature importance in a trained model
Change feature importance in a trained model
Loading saved threads...
Ricky · External communityPost link
External question — Data Science Stack Exchange
Author: Ricky
Original post: https://datascience.stackexchange.com/questions/96265
License: CC BY-SA 4.0 — https://creativecommons.org/licenses/by-sa/4.0/
Adaptation: HTML converted to plain text; contact email addresses removed.
I am giving a toy example for describing a real world business problem. Let's say I am a publisher and I have some book stores to visit. By visiting those stores I will check whether they have sufficient stock of my books, they are visible on shelf etc. Now, I am training a model, to recommend me the stores to visit. I have, say, 20 features and some historical data which has a target variable which represents whether a store was visited or not (1/0). I trained a
RandomForestClassifier
model on that data and this is the feature importance I got.
feature_14, feature_2 are more important than feature_11. Now, assume that feature_11 is a feature with high business importance. As an example, let's say feature_11 is the
number of books on shelf
and I want to put more importance on it in deciding whether to visit a store. Is there any way to put more importance on this feature_11 than feature_14 or feature_2? From historical data, model has learned that it is third most importance variable but I want to make it a key deciding factor. Is there any way of doing it?
Quote
Report
NaiveBayesian · External communityPost link
External answer — Data Science Stack Exchange
Author: NaiveBayesian
Original post: https://datascience.stackexchange.com/a/96303
License: CC BY-SA 4.0 — https://creativecommons.org/licenses/by-sa/4.0/
Adaptation: HTML converted to plain text; contact email addresses removed.
If I am understanding the real-world aspect of this correctly, it sounds like you are trying to decide on a set of sites to visit. Overall model "variable importances" are generally not a great approach here, and you want to try to find something from the predictive model itself related to the decision-of-interest.
You may want to consider using something different from the previous round of site-visits as the target variable. Instead, the modeling target should be an expected benefit that depends on a site visit. Then, after a model is built, you can use the results to generate a ranked list site-visits predicted to provide the largest benefit.
One approach would be:
Define some profitability or other benefit metric, that was available both before and after the site visits in your data were made.
If your system can model on a continuous variable, use the before-after difference directly as the target. Otherwise, bin the differences into t-shirt size classes (S/M/L/XL if you can run a multi-class classifer; S/L if you are limited to binary modeling)
Using site-visit as an input variable, build your random-forest model on your benefit-difference target.
Now run your cases through the predicted model, once with visited=No and once with visited=Yes.
Sort the results in order of biggest predicted difference between "visited" predictions. The top N sites that provide the biggest positive difference are the first N sites to visit.
If feature_11 is of business importance, i.e. it is related to profitability or some other success metric, it should fall naturally out of the analysis. However, a word of caution: For any kind of prediction based on past examples method to work, there needs to be some random variation between the variables. For example, if every case where feature_11 is large was also visited last year, then the model doesn't have anything to go on when feature_11=large/visited=No are suspect, and predictions assuming those conditions are suspect.
Quote
Report
Post Reply
Quoted from Forex.com.bd-Editorial External answer — Data Science Stack Exchange Author: NaiveBayesian Source score (net votes, not local likes): 1 Original post: https://datascience.stackexchange.com/a/96303 License: CC BY-SA 4.0 — https://creativecommons.org/licenses/by-sa/4.0/ Adaptation: HTML converted to plain text; contact email addresses removed. If I am understanding the real-world aspect of this correctly, it sounds like you are trying to decide on a set of sites to visit. Overall model "variable importances" are generally not a great approach here, and you want to try to find something from the predictive model itself related to the decision-of-interest. You may want to consider using something different from the previous round of site-visits as the target variable. Instead, the modeling target should be an expected benefit that depends on a site visit. Then, after a model is built, you can use the results to generate a ranked list site-visits predicted to provide the largest benefit. One approach would be: Define some profitability or other benefit metric, that was available both before and after the site visits in your data were made. If your system can model on a continuous variable, use the before-after difference directly as the target. Otherwise, bin the differences into t-shirt size classes (S/M/L/XL if you can run a multi-class classifer; S/L if you are limited to binary modeling) Using site-visit as an input variable, build your random-forest model on your benefit-difference target. Now run your cases through the predicted model, once with visited=No and once with visited=Yes. Sort the results in order of biggest predicted difference between "visited" predictions. The top N sites that provide the biggest positive difference are the first N sites to visit. If feature_11 is of business importance, i.e. it is related to profitability or some other success metric, it should fall naturally out of the analysis. However, a word of caution: For any kind of prediction based on past examples method to work, there needs to be some random variation between the variables. For example, if every case where feature_11 is large was also visited last year, then the model doesn't have anything to go on when feature_11=large/visited=No are suspect, and predictions assuming those conditions are suspect.
Checking account access…