Given data on 30 early problems, predict student performance ("struggle") on each of 20 later problems.

Juan D. Pinto | CSEDM Workshop @ LAK23

• CS1 course from the 2nd CSEDM Data Challenge
  • 2 phases, both with same test data (123 students)
    • Cross-semester: training data = 247 students
    • Within-semester: training data = 740 students

Juan D. Pinto | CSEDM Workshop @ LAK23

Example of weighting application.

Juan D. Pinto | CSEDM Workshop @ LAK23

1. Source code similarity
   • code2vec (Alon et al., 2019)
2. Problem prompt similarity
   • doc2vec (Le and Mikolov, 2014)
3. Struggling similarity
   • Performance correlation
4. Problem order

+ combinations of the above

Juan D. Pinto | CSEDM Workshop @ LAK23

Test AUC of various weighting schemes.

Juan D. Pinto | CSEDM Workshop @ LAK23

• 3 DLKT baselines:
  • LSTM implementation of DKT (Piech et al., 2015)
  • SAKT (Pandey and Karypis, 2019)
  • SAINT (Choi et al., 2020)
• IRT model:
  • Two-parameter logistic model (2PL; Embretson and Reise, 2000)

The test AUC of models with just problem correctness and ID sequences.

Juan D. Pinto | CSEDM Workshop @ LAK23

1. Source code similarity, struggling similarity, and problem order weighting improved prediction performance consistently.
2. Prompt similarity weighting did not improve prediction performance.
3. Combining weighting schemes by taking their average did not improve prediction performance.
4. Struggling similarity weighting in a lasso logistic regression model had performance comparable to the SOTA deep learning models.

Juan D. Pinto | CSEDM Workshop @ LAK23

• Problem relations and temporal order can be exploited through weighting schemes, leveraging some of the same information that deep learning models take advantage of.
• Can reach the accuracy of complex SOTA models while retaining better interpretability.

Juan D. Pinto | CSEDM Workshop @ LAK23

• Cold start problem remains since struggling and source code similarity rely on existing data about students' solutions or performance.
• The struggling weighting scheme assumes no or little change in students' knowledge states between related problems.
• Some added complexity to the process. Though the final models are relatively simple, deciding on weighting properties, how to quantify these properties, and similarity measures can be a complex process.

Juan D. Pinto | CSEDM Workshop @ LAK23