Since writing my project proposal, there have been some potentially major developments concerning the future goals for 15150’s autograding. In particular, there has been talk among the 15150 course staff, Umut Acar, and Bob Harper to advance a more unified system for autograding SML programs among the various classes at CMU that teach SML. It is not yet clear exactly how this will develop, but my advisor, Umut Acar, has stated that if this does move forward, he would like to have Zeus (my project) involved in it.
My primary goal prior for the end of the semester was to receive and evaluate test results from the 15150 course staff in order to assess Zeus’ current performance. Unfortunately, due to the various talks and uncertainty concerning the future of 15150’s autograding (by talks and uncertainty, I refer to the possibility of developing, a wider, more unified SML autograding infrastructure among the parties I mention above), the 15150 course staff hasn’t had the opportunity to test Zeus out on this semester’s data yet. Additionally, there was nothing in particular I could do on my end because the 15150 instructors did not wish to provide me the data necessary to run the tests myself because I am not a 15150 TA (even if the data were to be anonymized using a script that I would provide).
As stated, above, I was not able to reach my stated milestone of collecting experimental data concerning Zeus’ current performance. Unfortunately, because I have not been allowed access to the data necessary to proceed, there was nothing that I could do to reach my milestone individually. Having said that, based on conversations that I have pursued with the 15150 course staff as well as my advisor, I expect to have the data necessary to run these tests by next semester, either from 15150 as originally intended, or from code submitted to Diderot that my advisor has access to.
The efforts to develop a unified SML autograding infrastructure across multiple classes came as a surprise to me, but a welcome one, because as of present, it appears that there is an opportunity to incorporate Zeus in these efforts, which would mean that Zeus would have an opportunity to be more useful than originally expected.
The information I have learned since writing my milestones does not have a significant impact on the milestones themselves. I still expect to either work primarily on improving Zeus’ power, or work with others on incorporating Zeus into a wider infrastructure, based on Zeus’ performance with student data. The effort to unify multiple classes’ autograding may have an impact on what that second option would look like, but it does not change the general goal (and I do not need to revise any specific milestones for that second option because I already knew that if that second option were to be pursued, the specifics of how it would be pursued would depend drastically on others’ intentions).
Aside from the experimental data that would be useful for running my benchmarking tests, which I expect to acquire either from 15150 or my advisor by next semester, there are no resources that I lack to complete my 15-400 project.