What holding out participants tests AI
A train–validation–test split separates parameter fitting, model selection and final evaluation. When many records come from one person, splitting records randomly can allow person-specific patterns into both training and evaluation. Grouped splitting keeps each person’s data in one partition; see the scikit-learn grouped-validation discussion.
The activity pilot uses 17 training participants, four validation participants and the official nine test participants. Normalization is fitted only on training signals. Epoch 15 is selected using validation macro F1, then evaluated on test windows. Accuracy is 96.9% on validation and 90.5% on test: selection performance does not fully transfer to these other people.
Accuracy counts correct windows. Macro F1 averages each activity’s F1 score, giving classes equal weight rather than letting the most frequent activity dominate. Adjacent windows overlap, so 1,399 validation windows are not 1,399 independent people.
The pet report evaluates a different task on 740 validation images and does not report an official test result. Its accuracy should not be ranked directly against activity accuracy.