HPCS Guide for Faculty / Technicians

The purpose of this document is to present the steps that should be followed to set up an in-class experiment. Links to examples and templates are included with each step.

Each of the steps of the checklist will be of one of the following types and will be tagged appropriately:

• (Do once!): indicates a step that should be performed only once for the whole semester for one class.
• (Prepare once; do for each assignment!): indicates a step that should be prepared once, but should be performed once for each assignment.
• (Do for each assignment!):  indicates a step that should be performed for each assignment.

Some of the steps will be tagged with the text (Technician). These indicate steps that we suggest to be performed by a technician assisting the professor, for example, a TA or system administrator.

Content:

1. Before class starts
   1.a Complete Human Subjects Forms and submit to your University Review Board (IRB) for approval
   1.b Register experiment machine and create class in Experiment Manager
   1.c Specify assignments for the students
   1.d Set up any additional artifacts for assignments
2. As soon as class starts
   2.a Ensure that the data collection mechanisms are set up
   2.b Have students sign consent forms and register in the HPCS repository
   2.c Have the students complete the Background Questionnaire
3. When the assignments begin
   3.a Explain the data to be collected and give out the homework assignment(s)
4. As soon as all assignments are done
   4.a Enable students to fill in the End-of-study Questionnaire
   4.b Make sure all experimental data and questionnaires are sent to UMD

1 Before the class starts

1.a Complete Human Subjects Forms and submit to your University Review Board (IRB) for approval (Do once!)

Federal regulations require that all experiments, regardless of the type, that involve human subjects require such approval. This process is neither complicated nor difficult, but depending upon the university, this may require several months to accomplish. Start this early if this is your first time through the IRB process. Generally the IRB requires the experimenter to fill in forms that explain the goals and reasons for running the experiment, describe the tasks that are to be done by the subjects, and predict any harm that could come to the subjects as a result of participating in the study. We have had no trouble getting these forms approved in previous studies. An example of a completed form, based on the requirements at the University of Maryland, can be found here along with the attached answers to the required questions on the form. The consent form (see step 6) should also be approved by the University.  An example of a consent form can be found here. Contact us for help if you are novice at this process.

1.b Register experiment machine and create class in Experiment Manager (Do once!)

The goal of this step is to create a record to help you as well as the data collection and analysis team keep track of the relevant factors in the experiment.

Access the Experiment Manager system.

• If you are not registered in the system yet, you will need to contact the Administrator of the system and ask for an account. There is a link in the main page of the Experiment Manager to send an e-mail requesting an account. Once an account is created you will receive a user id and password.
• Log in using the account information that you received by e-mail.

Create an experiment machine.

• You need to register the machine (HPC hardware) that will be used in the experiment. To do so, use the option "Create experiment machine" from the main page of Experiment Manager. The screen will show a list of machines already registered.
  • If you have used the machine in previous experiments, you should see it in the list. Click on "HOME" to go back to main page.
  • If you don't see the machine in the list. Scroll down and fill in the form at the bottom of the page to register the machine. When you are finished click on "save".

Create a class for the experiment.

• From the main page of Experiment Manager, select the option "create and edit classes". You will see the experiments page.
• Click on "Create new experiment".
  • If you haven't registered the class previously, enter the information about the class. This information can be modified later, except for the class identifier that will be used by the students to sign up for the system. When you finish entering all information click on "save".

Make sure you inform the students of the class identifier. They will need it to register in Experiment Manager.

1.c Specify assignments for the students (Do for each assignment!)

In this step an assignment specification needs to be created for each of the different combinations of computational models and problems.

Register assignment in Experiment Manager

• First, register your assignment with the HPCS data collection team in the Experiment Manager. From the main page, select "Create and edit classes". You will see a list of one or more classes previously registered. Click on the icon for Assignments in the row of class you want to create an assignment for. This will take you to the Assignments Page.
• To create a new assignment, click on "Create New Assignments". Fill in the required information for the assignment. This information can be edited, so most of this information can be filled in later if it is not available yet. When finished entering all information, click on "save".

Create assignment description to be given out to the students

You need to create an assignment description that can be given out to the students. By reusing descriptions from previous classes and staying within a template, you can help us ensure that student performance on the same problem will be comparable across classes.

Below are a series of assignment specification templates with example problem descriptions that you can reuse. In those documents, text in black represents the most common way of describing the problem – these should be kept “as is” to allow the comparison of results between classes. If you feel that a change is necessary, please let the Maryland team know at hpcs-issues@cs.umd.edu. Fields in green should be filled out according to the local context. Text in brackets provides hints as to what type of information should be included.

• Buffon-Laplace

• Game of Life

• Game of Life 2D

• Matrix Multiplication

• Parallel Sum

• Sharks & Fishes

• One file with all assignment specification templates 

Select which students will do each assignment

You need to select which students will do each assignment depending on the experimental design chosen for your experiment. For more details on experimental designs please see these slides: Experimental Designs. Please contact us at hpcs-issues@cs.umd.edu if you need any help desiging your experiment.

Once you decided on the experimental designs and created the assignments for the class, you need to indicate in Experiment Manager which students will do each assignment. From the main page, select "Create and edit classes". You will see a list of one or more classes previously registered. Click on the icon for Assignments in the row of class you want to give assignments to students. This will take you to the Assignments Page. For each assignment, click on the icon in the column "Students" and make sure that only the students that will be doing that assignment are selected. By default, all students will be checked. Uncheck the students who should nott be doing that assignment. When you are finished click on "save". 

1.d (Optional) Set up any additional artifacts for assignments (Do for each assignment!)

In this step we suggest you create artifacts that could be handed out to the students together with the assignment specification in order to clarify the assignment and/or to ease the grading of the assignments. Examples of artifacts that could be handled out are test cases, examples of input files, etc.

Below is a list of examples of artifacts used in classes where the experiments were conducted that can be reused. We are starting to create this list, so if you have artifacts that you think could be useful for other faculty conducting experiments and would like to make them available from this page, please send an e-mail to hpcs-issues@cs.umd.edu.

Assignments and artifacts:

• Dense matrix and vector multiplication: Problem[1], Harness[1], Sequential Matlab code[1]
• Game of Life: Problem[1], Harness[1], Sequential Matlab code[1]
• Sparse matrix and dense vector multiplication: Problem[1], Harness[1], Sequential Matlab code[1]

[1] UCSB - CS240A: Applied Parallel Computing - Winter 2006 - Prof. John R. Gilbert; Assistant Viral Shah (http://www.cs.ucsb.edu/~cs240a/)

2 As soon as class starts

2.a (Technician) Ensure that the data collection mechanisms are set up (Prepare once; do for each assignment!)

Ensure that you have the appropriate authorizations to modify the operating environment of the machines to be used and to send the student data to the University of Maryland. As with the IRB process, the process is not complicated, but does require planning. Do this BEFORE the class begins to avoid problems later.

It is necessary to ensure that these automated data collection mechanisms are functioning properly and that all data collection forms to be filled by the students are specified prior to commencement of the experiment. This step should be performed as soon as the instructor knows the accounts that will be used for the class.

Go to the download page to get the instrumentation package that contains all the necessary files. The instructions on how to install and run the instrumentation is found here. The instructor or the TA should install the instrumentation package. Each subject will need to run a script to make sure their accounts are configured.  Before they start implementing a solution to their problems, they should notify the experimenter that the instrumentation package is properly configured.  The experimenter should check if the data is being collected and notify the subjects to proceed with the implementation once that is verified before each assignment.

Please make sure to discuss with the system administrator of your organization of the experiment and the types of resources that the students will be using during the semester, to prevent technical problems and issues.

2.b Have students sign consent forms and register in the HPCS repository.  (Do once!)

In the step, the experimenter should ask the students to sign the consent forms. An example of a consent form can be found here.

Once the forms are signed, the experimenter should instruct the students participating in the experiment to enroll in the Experiment Manager. Instructions on hor to enroll in Experiment Manager are provided in the HPCS Guide for Students. They will need the class ID registered in the repository, so make sure that that have that information before they register.

 

2.c Have the students complete the "Background Questionnaire" (Do once!)  

Prior to beginning work on any of the assignments related to the experiment, each subject should log in into Experiment Manager and complete the Background Questionnaire. This form provides the researchers with important information about the background and experiences of the subjects in a variety of tasks that are related to software development for high performance computers. The data provided by the subjects will be used to analyze their final results and look for relationships between types of experience and outcomes of the experiment.

Before you hand out any assignments, make sure every subject has completed their background questionnaire.

3 When the assignments begin

3.a Explain the data to be collected and give out the homework assignment(s) (Prepare once; do for each assignment!)

Make sure that the assignment description is complete and distribute it to the students.

In addition to any explanation of the programming models and the problems to be implemented, the instructors also need to explain to the subjects the effort and defect data that are being collected during this study. It is important that the subjects understand how to record their effort. The experimenters must be careful at this point not to inform the subjects of the reasons for collecting the data as this information may bias their working habits. The students should use the reporting options in Experiment Manager .

4 As soon as all assignments are done

4.a Enable students to fill in the "End-of-study Questionnaire" (Do once!)

At the completion of all of the assignments that are part of the study, the subjects will each complete the "Post-Experiment Questionnaire." This questionnaire consists of a series of questions to help the researchers better understand what actually occurred during the study. The responses to these questions will also help the researchers more accurately analyze the data and interpret the results. The questionnaire is available in Experiment Manager, but it will only be accessible by the students when the professor enables it. To enable the questionnaire for the students:

• Log into Experiment Manager
• From the main page chose the option "Create and edit classes"
• Click on the red icon for "end of study". The icon changes to indicate that this feature has been enabled.
• The students will see a new option in their menu called "Complete end of study questionnaire".

4.b Make sure all experimental data and questionnaires are sent to UMD (Do once!)

(Technician) E-mail hpcs-issues@cs.umd.edu all the log files from the instrumentation. Alternatively you can use the upload scripts function in Experiment Manager to send us the data.