Predicting student exam outcome based upon study patterns

External question — Data Science Stack Exchange Author: brianlmerritt Original post: https://datascience.stackexchange.com/questions/117935 License: CC BY-SA 4.0 — https://creativecommons.org/licenses/by-sa/4.0/ Adaptation: HTML converted to plain text; contact email addresses removed. I have a few years of data now for HE students participation in their course(es) including exam results. If I just compare formative exam results with summative results there is good correlation, and linear regression models deal well with that. For each new academic year however I would like to predict future results based upon study patterns. Data would be in the format Actual date | academic year | academic month | academic week | activity id | activity name | activity type | time spent | student identifier This results in 1000s of rows per individual student. According to https://machinelearningmastery.com/convert-time-series-supervised-learning-problem-python/ I can convert time series into a date pair (date yesterday | current date) but assuming each student has hundreds of activities this would result in a vast number of categorical columns comparing each activity. To reduce the number of rows which apply to one feature (the student) I can combine individual activities into a first accessed date, last accessed date, number of visits, total time spent, academic week, academic month, academic year etc which will reduce the number of rows but if then categorise individual activities (study_a, study_b, study_c) I end up with 1000s of features. If I then combine all the rows pertaining to one student into one row with 10,000 features I have a thousand rows for some cohorts with tens of thousands of features, for smaller cohorts I get only 100 rows but still have all those features. Y can be categorical (fail, low pass, medium pass, hi pass) or numerical I don't see a lot of material handling time events in this way. Everything time series based is like a daily stock position. I would appreciate references to better ways to handle time events all pertaining to the same feature e.g. student (or location or product etc) If xgboost or similar with one hot encoding with one very wide row per student per cohort and to do feature pruning iterations happens to be one of the best methods I will go with that, but I feel some form of memory and go with 1000s of rows must have some methodology.

External answer — Data Science Stack Exchange Author: brianlmerritt Original post: https://datascience.stackexchange.com/a/117971 License: CC BY-SA 4.0 — https://creativecommons.org/licenses/by-sa/4.0/ Adaptation: HTML converted to plain text; contact email addresses removed. According to https://stackoverflow.com/questions/71023822/lstm-multi-variate-multi-feature-in-pytorch I should feed each subject related set of rows as an array, one event per array element. np.array([[[feature1, feature2, feature3, feature4], [feature1, feature2, feature3, feature4], [feature1, feature2, feature3, feature4], [feature1, feature2, feature3, feature4]]]) so np.array([[[Actual date | academic year | academic month | academic week | activity id | activity name | activity type | time spent | student identifier],[Actual date | academic year | academic month | academic week | activity id | activity name | activity type | time spent | student identifier],[Actual date | academic year | academic month | academic week | activity id | activity name | activity type | time spent | student identifier],[Actual date | academic year | academic month | academic week | activity id | activity name | activity type | time spent | student identifier]]]) where each feature between the pipe symbols is either a value or a hot encoded category set of features. The Y then being a categorical exam range or an actual exam result.