The ODIN Researcher Website enables social and behavioral science researchers to design their study protocols and manage the continuous stream contextual data as it is being collected from study participants. Study protocols require the researcher to design a set of questions and a set of associated rules. A rule is formal specifications of the contexts in which a question should be asked of a study participant. Within ODIN, rules are compound logical expressions which may refer to data such as:
- Calendar date and time of day
- Engagement with particular locations (GPS, RFID, Bluetooth beacon)
- Physical proximity to other study participants (Bluetooth)
- Self-reports triggered by within-app interaction
- Biosensor data (stress, heart rate using Empatica E4; cognitive engagement via Emotiv Insight).
- Physical activity (via Google’s activity recognition API)
- Electronic contact (SMS, phone)
- Answers to previous questions
This webpage is intended for researchers who wish to use the ODIN system to conduct a scientific study. The document serves to provide a description of how to set up a study using the ODIN Research Web UI. A description of how ODIN is experienced by study participants is available here.
To start, researchers must log in to the ODIN website. Presently, to obtain an account on the ODIN website, researchers must send an email to the ODIN team. A web-based account application form will be available in the near future.
Upon successful login, the researcher will be presented with the screen below. In it, all the studies that the researcher has created will be listed on the left under “Leading Studies“. Studies that were created by other researchers, but to which this researcher has been added as a participant, will be listed on the right side under “Participating Studies“.
Each study is in exactly one of 3 states:
- Preparing (Green): Researchers are defining the study (its questions, contextual rules, parameters, and forms). In this state, the study is at its most malleable (existing questions and rules may be deleted, new questions and rules may be added). However, no study participant is allowed to register to the study using their ODIN App.
- In Progress (Blue): In this state, participants are able to register to the study using their ODIN App. Researchers may recruit participants are and determine their eligibility. They may issue a coupon to each eligible participant, who can then download the ODIN App and enters the issued coupon, as described here. The study is partly malleable (researchers may disable but not delete existing questions and rules, and new questions and rules may be added).
- Completed (Orange): In this state, participants may no longer register to the study using their ODIN App. The study is no longer malleable (researchers may not disable, delete, or add questions and rules.
A researcher can create a new study, by pressing the Create New button located above the left list. Doing this will bring the researcher to the screen below, where they must name the study, provide a description, and select which sensors are going to be used in the study. Each sensor selected may require the researcher to specify further parameters on the fidelity and frequency at which the sensor will be engaged by the ODIN App when it runs on the phones of study participants.
For example, the image above shows the researcher entering information for a hypothetical study on Sexual Abuse in which the researcher intends to engage the GPS sensor. Because GPS has been selected from the list of checkboxes, the researcher must further indicate: (1) the frequency with which GPS measurements should be made (in this case, they have specified 5 minutes), and (2) the minimum amount of movement that will be considered significant enough to be recorded (by specifying 0 meters here, they are indicating that measurements should always be recorded every 5 minutes, even if no movement occurs).
After filling in the fields in the screen above and pressing the Create New button, a new study is made and the researcher is taken to a study dashboard screen for where details of the new study can be further specified.
Alternately, a researcher can also manipulate an existing study by clicking on the pencil symbol next to the study name (in either the Leading Studies or Participating Studies Lists), provided the study is in either the Preparing (Green) or In-Progress (Blue) state. By clicking on the pencil icon, the researcher indicates that they wish to edit the specifications of the study to the extent that editing is permitted. So, for example, if the researcher clicked on the pencil symbol next to the Alcohol Craving Study, they would be shown the study dashboard screen below:
As can be seen in the image above, a study dashboard always contains 6 panels:
- About: Provides a high level overview of the study and its state
- Survey: Allows one to define the questions and the contextual rules under which they should be asked
- Coupons: Allows the management and monitoring of study coupons that have been (or will be) given to eligible participants
- Participants: Allows monitoring of data collected from study participants while the study is in progress
- Consent Form: Allows management of the consent form fields that will be presented to study participants within the ODIN App
- Download: Allows the researcher to download the study participants’ data so far
Each of the panels is described below:
1. The About Panel: This panel allows the researcher to change the study name and description. It also allows the lead researcher to add other researchers as participants. The panel allows the lead researcher to change the state of the study, advancing it from Preparing to In-Progress, and then from In-Progress to Completed. Note that a study cannot revert backwards to an earlier state. A study that is In-Progress cannot be moved to Preparing; once in Completed state, the study’s state cannot be further altered.
2. The Consent Panel of the Study Dashboard: This panel allows the researcher to specify the content of the consent form that will be presented to the study participant once they have registered their ODIN App using the coupon provided. A description of how consent is given by study participants is provided here.
The Consent panel (above) contains 9 distinct sub-forms, indicated by by the column of titles on the left. These are:
- Data Gathering
- Data Use
- Time Commitment
- Study Tasks
- Study Survey
- Review Content
Each of these 9 subforms allows for the specification of both short (summary) and long (description) versions of the informational text . These will be shown to participant in this study, when they register via their ODIN App (see here for more details). The design of this process follows the ResearchStack framework (http://researchstack.org/), which is the Android adaptation of Apple’s ResearchKit (https://www.apple.com/researchkit/). Each of the 9 screens presents a short version of informational text, and allows the participant to elicit a longer explanation if desired. Both the have been specified by the researcher(s) in the process of designing their study (see here for more details).
3. The Coupons Panel of the Study Dashboard: This dashboard allows the researcher to create new coupons for use in the study, and to track their progress within the study. Coupons are created by entering the desired number and pressing the Generate button. Once coupons have been created, they appear in the (leftmost) Inactive Participants list. Over time, the researcher(s) may issue these coupons to eligible participants, who may subsequently download the ODIN App and use their personal researcher-issued coupon to register (as described here). When a participant registers their coupon using the ODIN App, the coupon is transferred from the Inactive Participants list to the Consent Pending list (second from the left). The participant must then go through the consent process using the ODIN App (as described here). At the end of this, when the server receives evidence of the participants consent, the coupon is once again transferred from Consent Pending to the Active Participants list (third from the right). At this point, the participant will be actively generating data for the study. If at any point while the study is in In-Progress state, the participant decides to withdraw consent, their coupon is transferred from the Active Participants column to the Consent Withdrawn column. Once a participant is in the Consent Withdrawn phase, the ODIN App generates no new data. Data that was collected from these participants prior to their withdrawal of consent continues to be accessible to researchers.
4. The Participants Panel of the Study Dashboard: This dashboard allows the researcher to monitor the data generated by specific participants as the study is in the In-Progress state (as well as after it has been moved to Completed state). The panel shows a list of coupons which are Active or Once Active. In it we see coupons that are presently in Active or Consent Withdrawn state (note that for Consent Withdrawn coupons, only data that was collected prior to withdrawl of consent is recorded and available). The researcher may select one of these coupons, and then use the dropdown menu to select the type of data to be viewed. Data types supported are: sensor data, and survey data (each sensor has its own dropdown entry). The screen below is what would be shown if the use chose to view the GPS sensor data for ‘gwwuuzlyer’.
Each row in the table above contains the date and time, as well as the latitude and longitude of participant ‘gwwuuzlyer’ at that time. More generally, every sensor data table (of which GPS is one example) always has 3 columns: the coupon number, the timestamp, and the sensor reading at that time.
It is also possible for the researcher to look at the survey data by selecting survey from the dropdown menu. The screen below is what would be shown if the use chose to view the GPS sensor data for ‘ztrrcwbeue’.
Data for each question can now be examined separately. The researcher identifies the question of interest by clicking on the question’s tag in the first column. Upon doing this, the data for that question appears in columns 2-8. Each row in the table above contains the coupon number, the rule tag, the time the rule fired, the time the question was shown, the time the question was answered, the answer (to the question that the rule firing resulted in), and the time the answer was uploaded to the server. Because more than one rule can cause a question to be shown (e.g. if there are multiple contexts in which the same question should be posed), multiple rules can appear in the table for a single question.
5. The Survey Panel of the Study Dashboard: This is where the researcher makes and edits questions and rules for the study. To create a new question the researcher clicks on the +Add Question button. The following popup will appear, allowing the researcher to select the question type. Presently two question types are supported:
- Multiple choice (single answer) questions
- Fill (free) text
As seen above, in addition to providing the question text, researcher specifies a tag for the question. The tag is not shown to the user, but rather serves as a column label for data that is collected on the basis of the question being defined. The researcher also provides a description; this is available to researchers reviewing the codebook for the study (automatically generated). Once a question has been added in this way, choices can be added by pressing the +Add Response button and typing in the new response option. Questions may be edited by pressing the Edit button, upon which the following sort of popup might appear.
In this popup (above), the researcher may change the tag, description, question text, and choice options for each of question. Note that the question type may not be changed once a question is created.
Having created a question, the researcher may now add rules to the question by pressing the Add Rule button. A rule is simply a formal description of the context in which the question must be shown to the participant. When that context occurs, the rule is said to be have been triggered or fired. Each question may have multiple rules associated to it. In this case when every any of the rules’ conditions are satisfied, the question will be shown. Rules come in two rule flavors:
- Phone-side rules are evaluated within the ODIN App itself. Because they are closest to the participant, the lag between when the rule’s contextual conditions are met and the question is asked, is minimized. However, because phones are unstable (e.g. can be turned off), the rule may also fail to be triggered at all because of technical interference. Phone-side rules are maximally responsive, but pay a cost in terms of reliability.
- Server-side rules are evaluated with the ODIN (Back End) Server. Because they are only evaluated after the Server receives data from the phone, there can be a significant delay between when the rule’s the contextual conditions are met and the question is asked. On the other hand, the server is extremely stable and so it is fairly certain that the if the rule was supposed to be triggered, it will definitely be triggered, albeit with some delay. Server-side rules are maximally reliable, but pay a cost in terms of responsiveness.
Beyond flavors, several different types of rules are supported. These can be seen in the dropdown menu that appears in the popup window when the Add Rule button is pressed. Currently supported types include:
- Time-specific: this rule triggers when a specific time occurs, or periodically, e.g. at given sets of times on given days.
- On arrival: this rule triggers when a participant arrives at a location
- On departure: this rule triggers when a participant departs from a location
- While at: this rule triggers while a participant is at a location
- While not at: this rule triggers while a participant is not at a location
- Self-reported: this rule triggers while a participant presses a custom button in the ODIN App
- Chain: this rule triggers when a participant answers a specified question with a specified answer
Each rule type has several different parameters which need to be provided for the contextual criteria to be fully specified. For a Time-Specific rule for example, the researcher must specify the periodicity, the days, and the time. Below is an example of a Time-Specific rule that would cause a question to be shown once a week.
Below is an example of a Time-Specific rule that would cause a question to be shown once a day.
In comparison, the Chain type rule has different parameters which need to be provided for the contextual criteria to be fully specified. For a Chain, the researcher must specify the tag of the question, and the response that must be chosen by the participant for the rule to get triggered. Below is an example of a Chain rule that would be triggered 1 second after the participant answered CH_Yes to question Crave2b.
As seen below, on the screen, rules always appear to the right of the questions to which they are associated. Note that a question may have multiple rules associated with it. Questions themselves may be sorted by question number or question tag. Question numbers are assigned sequentially when the researcher adds questions; tags, however, are customizable by the researcher.
While a study is in the Preparing state (see above), questions and rules can be edited, added, and deleted. Once a study is moved to In-Progress state (see below), questions and rules can no longer be deleted; they can only be disabled. It may take some time for a participant’s ODIN App to be notified that the rule/question was disabled, but once this notification is received, the context-driven question will not longer be asked.
6. The Download Panel of the Study Dashboard: This is used to download all the data for the study as a zipped directory of interlinked comma-separated variable format files.