How Do You Balance Feature Search Strategy and HP Optimization Cost?
How Do You Balance Feature Search Strategy and HP Optimization Cost?
Loading saved threads...
Ten · External communityPost link
External question — Data Science Stack Exchange
Author: Ten
Original post: https://datascience.stackexchange.com/questions/134539
License: CC BY-SA 4.0 — https://creativecommons.org/licenses/by-sa/4.0/
Adaptation: HTML converted to plain text; contact email addresses removed.
What I’m trying to figure out
I'm working on a machine learning project and would love to hear your thoughts on two things:
A. How to prioritize feature exploration
B. Whether to fix hyperparameters (HP) during feature/model evaluation
A. Feature exploration strategy
When testing new features, how do you decide the order of exploration and evaluation?
I’m considering two approaches:
1. Exhaustive approach
Generate as many feature candidates as possible and test them all.
2. Stepwise approach
First, group features by theme and test only the representative ones from each theme.
For example, in stock price prediction, themes might include:
Price ratios over time
Time-based features
Peer company data
A representative feature might be the stock price trend of a peer company.
Then, if a representative feature from a theme performs well, explore similar features within that theme.
For instance, if peer company stock prices are useful, I might try peer company revenue trends next.
I feel the stepwise approach is more efficient, but I’m worried it might miss cases where the representative feature performs poorly, while a similar feature could have worked well.
B. Fixing hyperparameters during evaluation
When comparing different features or models, do you fix hyperparameters or re-optimize them each time?
Assuming cross-validation is used, re-running HP optimization (e.g., with Optuna) for every new feature is computationally expensive.
On the other hand, fixing HPs based on earlier tuning might cause unfair evaluations—some features might appear ineffective simply due to poor compatibility with the fixed HPs.
How do you balance
computational cost
and
fair evaluation
in practice?
Quote
Report
MuhammedYunus · External communityPost link
External answer — Data Science Stack Exchange
Author: MuhammedYunus
Original post: https://datascience.stackexchange.com/a/134542
License: CC BY-SA 4.0 — https://creativecommons.org/licenses/by-sa/4.0/
Adaptation: HTML converted to plain text; contact email addresses removed.
I prefer to build features in progressively, starting from a small core and adding more if necessary. It is somewhat inseparable from EDA, since I am guided by understanding (as well as experimentation).
I have a preference towards fewer features that perform robustly and well, rather than many features that perform a bit better.
My rationale for this:
Fewer features help mitigate overfitting, and also mean more robust deployment (fewer failure points, and could also be more efficient)
Better for developing an understanding of the data/model, and explaining it to others
I find that once a few key features are identified, the additional features tend to not improve things by much (whether the x% extra is worth it is context-dependent)
I wouldn't re-optimise each time, since if a new feature is genuinely good it would usually show with whatever hyperparameters you already had. I aim for tuning at the start and also at the end. I would find decent hyperparameters at the start and broadly stick to them.
Quote
Report
Post Reply
Quoted from Forex.com.bd-Editorial External question — Data Science Stack Exchange Author: Ten Source score (net votes, not local likes): 6 Original post: https://datascience.stackexchange.com/questions/134539 License: CC BY-SA 4.0 — https://creativecommons.org/licenses/by-sa/4.0/ Adaptation: HTML converted to plain text; contact email addresses removed. What I’m trying to figure out I'm working on a machine learning project and would love to hear your thoughts on two things: A. How to prioritize feature exploration B. Whether to fix hyperparameters (HP) during feature/model evaluation A. Feature exploration strategy When testing new features, how do you decide the order of exploration and evaluation? I’m considering two approaches: 1. Exhaustive approach Generate as many feature candidates as possible and test them all. 2. Stepwise approach First, group features by theme and test only the representative ones from each theme. For example, in stock price prediction, themes might include: Price ratios over time Time-based features Peer company data A representative feature might be the stock price trend of a peer company. Then, if a representative feature from a theme performs well, explore similar features within that theme. For instance, if peer company stock prices are useful, I might try peer company revenue trends next. I feel the stepwise approach is more efficient, but I’m worried it might miss cases where the representative feature performs poorly, while a similar feature could have worked well. B. Fixing hyperparameters during evaluation When comparing different features or models, do you fix hyperparameters or re-optimize them each time? Assuming cross-validation is used, re-running HP optimization (e.g., with Optuna) for every new feature is computationally expensive. On the other hand, fixing HPs based on earlier tuning might cause unfair evaluations—some features might appear ineffective simply due to poor compatibility with the fixed HPs. How do you balance computational cost and fair evaluation in practice?
Checking account access…