Jack S.
|
|
Group: Forum Members
Posts: 8,
Visits: 99
|
Hello all,
I'm attempting to run an experiment in Inquisit Web that involves a binary between-subjects variable and two binary variables (presentation orders) that are being counterbalanced. Thus, in my script, I have 8 expt elements, each with groupassignment = groupnumber and subjects = (x of 8), where x ranges from 1 to 8. Running the script in Inquisit lab, I am able to assign subjects to the appropriate condition by entering the appropriate group number.
However, when I run the experiment on Inquisit Web, problems arise. I've set participant IDs to be generated randomly without replacement, so that they can serve as confirmation codes for MTurk participants, and set group IDs to be generated sequentially, in order to make sure that participants are assigned evenly to all 8 conditions. However, in my data, the group IDs appear as large, non-consecutive numbers (740797481, 740797483, 740797485, 740797487, 740797489...), and thus only half of my conditions are being run.
Is there a way to ensure than participants are assigned evenly to conditions (i.e., the 1st, 9th, 17th, etc. participant to condition 1, the 2nd, 10th, 18th, etc. participant to condition 2...)? Or am I better off removing the groupassignment attribute and just trusting that with enough participants, the conditions will receive more or less the same number of participants? More generally, any reason why the group IDs take the form they do, instead of {1, 2, 3, 4, ...}? Lastly, if you could direct me to information about how exactly the groupassignment and subjects attributes interact in Inquisit Web, I'd greatly appreciate it!
Thanks in advance, and please let me know if there's any additional info I can provide. I'm more than happy to upload my script if you'd like!
|
|
|
Dave
|
|
Group: Administrators
Posts: 13K,
Visits: 105K
|
+xHello all, I'm attempting to run an experiment in Inquisit Web that involves a binary between-subjects variable and two binary variables (presentation orders) that are being counterbalanced. Thus, in my script, I have 8 expt elements, each with groupassignment = groupnumber and subjects = (x of 8), where x ranges from 1 to 8. Running the script in Inquisit lab, I am able to assign subjects to the appropriate condition by entering the appropriate group number.
However, when I run the experiment on Inquisit Web, problems arise. I've set participant IDs to be generated randomly without replacement, so that they can serve as confirmation codes for MTurk participants, and set group IDs to be generated sequentially, in order to make sure that participants are assigned evenly to all 8 conditions. However, in my data, the group IDs appear as large, non-consecutive numbers (740797481, 740797483, 740797485, 740797487, 740797489...), and thus only half of my conditions are being run.
Is there a way to ensure than participants are assigned evenly to conditions (i.e., the 1st, 9th, 17th, etc. participant to condition 1, the 2nd, 10th, 18th, etc. participant to condition 2...)? Or am I better off removing the groupassignment attribute and just trusting that with enough participants, the conditions will receive more or less the same number of participants? More generally, any reason why the group IDs take the form they do, instead of {1, 2, 3, 4, ...}? Lastly, if you could direct me to information about how exactly the groupassignment and subjects attributes interact in Inquisit Web, I'd greatly appreciate it!
Thanks in advance, and please let me know if there's any additional info I can provide. I'm more than happy to upload my script if you'd like! > However, in my data, the group IDs appear as large, non-consecutive numbers (740797481, 740797483, 740797485, 740797487, 740797489...).
That doesn't sound right. Could you please provide the link to the experiment's launch page? Also, did you have group id generation set to random _prior_ to switching to sequential?
> Is there a way to ensure than participants are assigned evenly to conditions (i.e., the 1st, 9th, 17th, etc. participant to condition 1, the 2nd, 10th, 18th, etc. > participant to condition 2...)?
If you want full control over the assignment, you can pass the group number in to the launch page via a URL query parameter. I.e., you'd give 1/8th of your participants a link that runs the 1st condition, 1/8th a link that runs the 2nd condition, 1/8th a link that runs the 3rd condition, and so forth.
See e.g. https://www.millisecond.com/forums/FindPost19411.aspx > Or am I better off removing the groupassignment attribute and just trusting that with enough participants, the conditions will receive > more or less the same number of participants?
I don't see how removing the /groupassignment would lead to that result. /groupassignment merely determines on what *basis* condition assignment is performed, i.e. based on a numerical subject id vs based on a numerical group id. You can use random group ids instead of sequential, and that then should indeed lead to approximately equal distribution of participants across conditions over the long run (i.e. with an increasing number of participants, things should even out more and more).
> Lastly, if you could direct me to information about how exactly the groupassignment and subjects attributes interact in Inquisit Web [...]
They work the same in Inquisit Web as in Inquisit Lab. See https://www.millisecond.com/forums/Topic13856.aspx for the math behind condition assignment and how to figure out which condition was run based on a given subject or group number.
|
|
|
Jack S.
|
|
Group: Forum Members
Posts: 8,
Visits: 99
|
Thank you for the prompt response! The link for my study's launch page is http://research.millisecond.com/hopper/facemorphexp071517.web. I don't believe I had group ID generation set to random initially... I recall selecting sequential group ID generation when I set up the web script.
|
|
|
Dave
|
|
Group: Administrators
Posts: 13K,
Visits: 105K
|
Thanks for the link -- not sure about the high group numbers, first having random ones and then switching to sequential would have explained it. What you can try to "reset" things is to take down the experiment completely (i.e. "unregister" the web experiment) and the re-register / re-upload the experiment under a different name (i.e. change the script file's name slightly before re-registering it). Ultimately, though, whether the sequential group numbers start at 1 or at 700000 should not matter in terms of equal condition assignment. As far as condition assignment goes, I seem to be unable to reproduce the " 740797481, 740797483, 740797485, 740797487, 740797489,..." -- i.e. odd numbers only -- phenomenon. When conducting two consecutive test runs, the group number for the first one was 740797491, and the group number for the one directly following was 740797492, i.e. as expected / as it should be. I'm attaching two summary data files from those runs so you can see / verify for yourself.
I'm not sure how exactly you have conducted your test runs, but if you manually refreshed the launch page between various test runs on the same device, try conducting your test runs _without_ forcing a refresh and let me know if you're still only seeing odd group numbers.
Hope this helps.
|
|
|
Jack S.
|
|
Group: Forum Members
Posts: 8,
Visits: 99
|
I just did a handful of test runs without refreshing the launch page (i.e., kept coming back to the page and clicking the green "Start" button), and everything seemed to work as intended: I got group IDs "740797497, 740797498, 740797499, 740797500, 740797501, 740797502", which (conveniently) corresponded to conditions 1 through 6 of my experiment, in order. So far, so good!
When I distribute the URL for the launch page on MTurk, will it continue to behave this way? Or will it randomize the group IDs again, since each participant is "refreshing" (loading) the launch page seperately?
Again, thank you so much your help!
|
|
|
Dave
|
|
Group: Administrators
Posts: 13K,
Visits: 105K
|
+xI just did a handful of test runs without refreshing the launch page (i.e., kept coming back to the page and clicking the green "Start" button), and everything seemed to work as intended: I got group IDs "740797497, 740797498, 740797499, 740797500, 740797501, 740797502", which (conveniently) corresponded to conditions 1 through 6 of my experiment, in order. So far, so good! When I distribute the URL for the launch page on MTurk, will it continue to behave this way? Or will it randomize the group IDs again, since each participant is "refreshing" (loading) the launch page seperately? Again, thank you so much your help! How you distribute the link should have no effect, and group numbers should continue to be generated / given out as expected, i.e. sequentially (not randomly). An id would be given out when a participant first visits the launch page, notably regardless of whether the participant then decides to participate or simply leaves. In other words, you are likely to encounter some gaps / non-consecutive numbers under real world conditions due to participants dropping out before ever launching the experiment, but that should even out with enough participants. Hope this helps.
|
|
|