By AnthonyFCollinsSussex - 5/12/2021
Hello. I have a remaining outstanding issue that I need to resolve before data collection, and I have been searching for the solution and would just like to double clarify if I am correct before I go ahead with it.
There are two conditions (Positive IBT training or Negative IBT training). In each of these conditions, there is a pre- and post- cognitive bias assessment that assesses cognitive bias before and after the IBT training type - RTA or RTB (Recognition Task). RTA and RTB are both the same test, only differing in material set used (e.g. set A or B). The order with which RTA and RTB are presented pre- and post- training are counterbalanced as can be seen below:
<batch>
/ subjects = (1 of 2)
/ groupassignment = groupnumber
/ file = "RTA.iqx"
/ file = "Positive IBT.iqx"
/ file = "RTB.iqx
</batch>
<batch>
/ subjects = (2 of 2)
/ groupassignment = groupnumber
/ file = "RTB.iqx"
/ file = "Positive IBT.iqx"
/ file = "RTA.iqx
</batch>
HOWEVER, a random 50% of each training condition need to get RTA followed by RTB, and the other 50% the other way around. Hence material order is orthogonal to condition. In my data output, I would need information about which material set (e.g. RTA or RTB) were encountered first for each participant. I will eventually create in excel columns denoting something like 'material order' (0 = RTA first, 1 = RTB first), 'first RT score' (regardless of material type) 'second RT score' (regardless of material type). I can do these last parts independently, but the way I was going to approach this design set-up was to simply do the following:
// order RTA -> Positive IBT -> RTB
<batch>
/ subjects = (1 of 4)
/ groupassignment = groupnumber
/ file = "RTA.iqx"
/ file = "Positive IBT.iqx"
/ file = "RTB.iqx
</batch>
// order RTA -> Control IBT -> RTB
<batch>
/ subjects = (2 of 4)
/ groupassignment = groupnumber
/ file = "RTA.iqx"
/ file = "Positive IBT.iqx"
/ file = "RTB.iqx
</batch>
// order RTB -> Positive IBT -> RTA
<batch>
/ subjects = (3 of 4)
/ groupassignment = groupnumber
/ file = "RTB.iqx"
/ file = "Positive IBT.iqx"
/ file = "RTA.iqx
</batch>
// order RTB -> Control IBT -> RTA
<batch>
/ subjects = (4 of 4)
/ groupassignment = groupnumber
/ file = "RTB.iqx"
/ file = "Positive IBT.iqx"
/ file = "RTA.iqx
</batch>
As mentioned though, a random 50% of each training condition must get RTA followed by RTB, and the other 50% the other way around. I just need to check if this strategy is correct.
Yours Sincerely
Anthony
|
By Dave - 5/12/2021
+xHello. I have a remaining outstanding issue that I need to resolve before data collection, and I have been searching for the solution and would just like to double clarify if I am correct before I go ahead with it. There are two conditions (Positive IBT training or Negative IBT training). In each of these conditions, there is a pre- and post- cognitive bias assessment that assesses cognitive bias before and after the IBT training type - RTA or RTB (Recognition Task). RTA and RTB are both the same test, only differing in material set used (e.g. set A or B). The order with which RTA and RTB are presented pre- and post- training are counterbalanced as can be seen below: <batch> / subjects = (1 of 2) / groupassignment = groupnumber / file = "RTA.iqx" / file = "Positive IBT.iqx" / file = "RTB.iqx </batch> <batch> / subjects = (2 of 2) / groupassignment = groupnumber / file = "RTB.iqx" / file = "Positive IBT.iqx" / file = "RTA.iqx </batch> HOWEVER, a random 50% of each training condition need to get RTA followed by RTB, and the other 50% the other way around. Hence material order is orthogonal to condition. In my data output, I would need information about which material set (e.g. RTA or RTB) were encountered first for each participant. I will eventually create in excel columns denoting something like 'material order' (0 = RTA first, 1 = RTB first), 'first RT score' (regardless of material type) 'second RT score' (regardless of material type). I can do these last parts independently, but the way I was going to approach this design set-up was to simply do the following: // order RTA -> Positive IBT -> RTB <batch> / subjects = (1 of 4) / groupassignment = groupnumber / file = "RTA.iqx" / file = "Positive IBT.iqx" / file = "RTB.iqx </batch> // order RTA -> Control IBT -> RTB <batch> / subjects = (2 of 4) / groupassignment = groupnumber / file = "RTA.iqx" / file = "Positive IBT.iqx" / file = "RTB.iqx </batch> // order RTB -> Positive IBT -> RTA <batch> / subjects = (3 of 4) / groupassignment = groupnumber / file = "RTB.iqx" / file = "Positive IBT.iqx" / file = "RTA.iqx </batch> // order RTB -> Control IBT -> RTA <batch> / subjects = (4 of 4) / groupassignment = groupnumber / file = "RTB.iqx" / file = "Positive IBT.iqx" / file = "RTA.iqx </batch> As mentioned though, a random 50% of each training condition must get RTA followed by RTB, and the other 50% the other way around. I just need to check if this strategy is correct. Yours Sincerely Anthony
What's the difference between these two?
// order RTB -> Positive IBT -> RTA
<batch>
/ subjects = (3 of 4)
/ groupassignment = groupnumber
/ file = "RTB.iqx"
/ file = "Positive IBT.iqx"
/ file = "RTA.iqx
</batch>
// order RTB -> Control IBT -> RTA
<batch>
/ subjects = (4 of 4)
/ groupassignment = groupnumber
/ file = "RTB.iqx"
/ file = "Positive IBT.iqx"
/ file = "RTA.iqx
</batch>
I can see that it says "Control IBT" for group 4, but it actually runs the same "Positive IBT.iqx" script as in group 3, i.e. they appear 100% identical.
(Same for groups 1 and 2).
|
By AnthonyFCollinsSussex - 5/12/2021
One of those days, my bad :
// order RTA -> Positive IBT -> RTB
/ subjects = (1 of 4)
/ groupassignment = groupnumber
/ file = "RTA.iqx"
/ file = "Positive IBT.iqx"
/ file = "RTB.iqx
// order RTA -> Control IBT -> RTB
/ subjects = (2 of 4)
/ groupassignment = groupnumber
/ file = "RTA.iqx"
/ file = "Control IBT.iqx"
/ file = "RTB.iqx
// order RTB -> Positive IBT -> RTA
/ subjects = (3 of 4)
/ groupassignment = groupnumber
/ file = "RTB.iqx"
/ file = "Positive IBT.iqx"
/ file = "RTA.iqx
// order RTB -> Control IBT -> RTA
/ subjects = (4 of 4)
/ groupassignment = groupnumber
/ file = "RTB.iqx"
/ file = "Control IBT.iqx"
/ file = "RTA.iqx
This is to ensure that we can identify that positive IBT does exert an effect that is not confounded by materials Set presentation (to rule out a material presentation effect, in this case A and B, we very much don’t want this , hence the above
|
By Dave - 5/12/2021
+xOne of those days, my bad :// order RTA -> Positive IBT -> RTB / subjects = (1 of 4)/ groupassignment = groupnumber / file = "RTA.iqx"/ file = "Positive IBT.iqx"/ file = "RTB.iqx// order RTA -> Control IBT -> RTB / subjects = (2 of 4)/ groupassignment = groupnumber/ file = "RTA.iqx"/ file = "Control IBT.iqx"/ file = "RTB.iqx// order RTB -> Positive IBT -> RTA / subjects = (3 of 4)/ groupassignment = groupnumber/ file = "RTB.iqx"/ file = "Positive IBT.iqx"/ file = "RTA.iqx// order RTB -> Control IBT -> RTA / subjects = (4 of 4)/ groupassignment = groupnumber/ file = "RTB.iqx"/ file = "Control IBT.iqx"/ file = "RTA.iqxThis is to ensure that we can identify that positive IBT does exert an effect that is not confounded by materials Set presentation (to rule out a material presentation effect, in this case A and B, we very much don’t want this , hence the above
Okay, that looks correct to me now. Group ID will be logged in the data files (unless you deliberatey excluded it), and you can derive order (RTA first vs RTB first) from that.
As for how condition assignment works generally, see https://www.millisecond.com/forums/Topic13856.aspx
|
By AnthonyFCollinsSussex - 5/12/2021
Ok this is what I thought , 50% are randomly allocated to each respective condition (or rather random 50% of each training condition get RTA followed by RTB, and the other 50% the other way around) .
I’ll will give this link you include a good read through just to be sure. But thank you again for that clarity. I will only reply if it is absolutely urgent!
Very much appreciated thank you soooo much
|
By AnthonyFCollinsSussex - 5/14/2021
+x+xOne of those days, my bad :// order RTA -> Positive IBT -> RTB / subjects = (1 of 4)/ groupassignment = groupnumber / file = "RTA.iqx"/ file = "Positive IBT.iqx"/ file = "RTB.iqx// order RTA -> Control IBT -> RTB / subjects = (2 of 4)/ groupassignment = groupnumber/ file = "RTA.iqx"/ file = "Control IBT.iqx"/ file = "RTB.iqx// order RTB -> Positive IBT -> RTA / subjects = (3 of 4)/ groupassignment = groupnumber/ file = "RTB.iqx"/ file = "Positive IBT.iqx"/ file = "RTA.iqx// order RTB -> Control IBT -> RTA / subjects = (4 of 4)/ groupassignment = groupnumber/ file = "RTB.iqx"/ file = "Control IBT.iqx"/ file = "RTA.iqxThis is to ensure that we can identify that positive IBT does exert an effect that is not confounded by materials Set presentation (to rule out a material presentation effect, in this case A and B, we very much don’t want this , hence the above Okay, that looks correct to me now. Group ID will be logged in the data files (unless you deliberatey excluded it), and you can derive order (RTA first vs RTB first) from that. As for how condition assignment works generally, see https://www.millisecond.com/forums/Topic13856.aspx
Hello Dave
Hope all is well. So it turns out this set-up is not quite right and may still impose an element of order effect (as it is sequential). I need to check whether the following is implementable within Inquisit, or whether this needs to be done in Qualtrics . Given there are 4 'conditions' in the above, we need to create the following array, where the length is determined by maximum number of participants (we hope this to be 150). I have gone for 20 below for sake of ease:
Condition = [ 1 3 2 4 4 3 1 2 2 3 4 1 1 2 3 4 4 2 1 3]
The condition for each participant would be based on the order in which they are run, such that the first participant would be allocated to condition 1 (the first number in the condition array), the second person would be allocate to condition 3 (the second number in the condition array) etc. the array is constructed such that the order of every 4 participants is random but includes one in each condition i.e. above you have 1 3 2 4 followed by 4 3 1 2 etc. In this way you know that provided you run a multiple of 4 participants you will always have an equal number in each of the 4 conditions. Is there a way I can achieve this?
|
By Dave - 5/14/2021
+x+x+xOne of those days, my bad :// order RTA -> Positive IBT -> RTB / subjects = (1 of 4)/ groupassignment = groupnumber / file = "RTA.iqx"/ file = "Positive IBT.iqx"/ file = "RTB.iqx// order RTA -> Control IBT -> RTB / subjects = (2 of 4)/ groupassignment = groupnumber/ file = "RTA.iqx"/ file = "Control IBT.iqx"/ file = "RTB.iqx// order RTB -> Positive IBT -> RTA / subjects = (3 of 4)/ groupassignment = groupnumber/ file = "RTB.iqx"/ file = "Positive IBT.iqx"/ file = "RTA.iqx// order RTB -> Control IBT -> RTA / subjects = (4 of 4)/ groupassignment = groupnumber/ file = "RTB.iqx"/ file = "Control IBT.iqx"/ file = "RTA.iqxThis is to ensure that we can identify that positive IBT does exert an effect that is not confounded by materials Set presentation (to rule out a material presentation effect, in this case A and B, we very much don’t want this , hence the above Okay, that looks correct to me now. Group ID will be logged in the data files (unless you deliberatey excluded it), and you can derive order (RTA first vs RTB first) from that. As for how condition assignment works generally, see https://www.millisecond.com/forums/Topic13856.aspx Hello Dave Hope all is well. So it turns out this set-up is not quite right and may still impose an element of order effect (as it is sequential). I need to check whether the following is implementable within Inquisit, or whether this needs to be done in Qualtrics . Given there are 4 'conditions' in the above, we need to create the following array, where the length is determined by maximum number of participants (we hope this to be 150). I have gone for 20 below for sake of ease: Condition = [ 1 3 2 4 4 3 1 2 2 3 4 1 1 2 3 4 4 2 1 3] The condition for each participant would be based on the order in which they are run, such that the first participant would be allocated to condition 1 (the first number in the condition array), the second person would be allocate to condition 3 (the second number in the condition array) etc. the array is constructed such that the order of every 4 participants is random but includes one in each condition i.e. above you have 1 3 2 4 followed by 4 3 1 2 etc. In this way you know that provided you run a multiple of 4 participants you will always have an equal number in each of the 4 conditions. Is there a way I can achieve this?
If you select random (without replacement) group number generation in the Inquisit Web experiment's settings and indicate the number of groups as 4, then for each set of four participants that visit the start page, one will be randomly assigned to condition 1, one will be randomly assigned to condition 2, one to condition 3, one to condition 4.
If this is not to your liking, you can have Qualtrics perform the condition assignment and have it send group ID over via URL parameter, just like subject ID.
|
By AnthonyFCollinsSussex - 5/14/2021
+x+x+x+xOne of those days, my bad :// order RTA -> Positive IBT -> RTB / subjects = (1 of 4)/ groupassignment = groupnumber / file = "RTA.iqx"/ file = "Positive IBT.iqx"/ file = "RTB.iqx// order RTA -> Control IBT -> RTB / subjects = (2 of 4)/ groupassignment = groupnumber/ file = "RTA.iqx"/ file = "Control IBT.iqx"/ file = "RTB.iqx// order RTB -> Positive IBT -> RTA / subjects = (3 of 4)/ groupassignment = groupnumber/ file = "RTB.iqx"/ file = "Positive IBT.iqx"/ file = "RTA.iqx// order RTB -> Control IBT -> RTA / subjects = (4 of 4)/ groupassignment = groupnumber/ file = "RTB.iqx"/ file = "Control IBT.iqx"/ file = "RTA.iqxThis is to ensure that we can identify that positive IBT does exert an effect that is not confounded by materials Set presentation (to rule out a material presentation effect, in this case A and B, we very much don’t want this , hence the above Okay, that looks correct to me now. Group ID will be logged in the data files (unless you deliberatey excluded it), and you can derive order (RTA first vs RTB first) from that. As for how condition assignment works generally, see https://www.millisecond.com/forums/Topic13856.aspx Hello Dave Hope all is well. So it turns out this set-up is not quite right and may still impose an element of order effect (as it is sequential). I need to check whether the following is implementable within Inquisit, or whether this needs to be done in Qualtrics . Given there are 4 'conditions' in the above, we need to create the following array, where the length is determined by maximum number of participants (we hope this to be 150). I have gone for 20 below for sake of ease: Condition = [ 1 3 2 4 4 3 1 2 2 3 4 1 1 2 3 4 4 2 1 3] The condition for each participant would be based on the order in which they are run, such that the first participant would be allocated to condition 1 (the first number in the condition array), the second person would be allocate to condition 3 (the second number in the condition array) etc. the array is constructed such that the order of every 4 participants is random but includes one in each condition i.e. above you have 1 3 2 4 followed by 4 3 1 2 etc. In this way you know that provided you run a multiple of 4 participants you will always have an equal number in each of the 4 conditions. Is there a way I can achieve this? If you select random (without replacement) group number generation in the Inquisit Web experiment's settings and indicate the number of groups as 4, then for each set of four participants that visit the start page, one will be randomly assigned to condition 1, one will be randomly assigned to condition 2, one to condition 3, one to condition 4. If this is not to your liking, you can have Qualtrics perform the condition assignment and have it send group ID over via URL parameter, just like subject ID.
Sounds like this set-up achieves what we need it to, in other words participant 1 could be assigned to any one of the 4, let's say at random, 3rd condition, the second participant gets assigned to the 1st condition, and then the final two participants will be randomly allocated to one of the final remaining conditions (e.g. 2nd and 4th condition in this case). Then this randomizing process restarts for the next 4 participants (with a different randomised order/allocation) and so on. If I have understood this correctly, then we should not need Qualtrics, and this achieves the array above (random 50% of each training condition get RTA followed by RTB, and the other 50% the other way around)
Thank you Dave again, this is very much appreciated
|
By Dave - 5/14/2021
+x+x+x+x+xOne of those days, my bad :// order RTA -> Positive IBT -> RTB / subjects = (1 of 4)/ groupassignment = groupnumber / file = "RTA.iqx"/ file = "Positive IBT.iqx"/ file = "RTB.iqx// order RTA -> Control IBT -> RTB / subjects = (2 of 4)/ groupassignment = groupnumber/ file = "RTA.iqx"/ file = "Control IBT.iqx"/ file = "RTB.iqx// order RTB -> Positive IBT -> RTA / subjects = (3 of 4)/ groupassignment = groupnumber/ file = "RTB.iqx"/ file = "Positive IBT.iqx"/ file = "RTA.iqx// order RTB -> Control IBT -> RTA / subjects = (4 of 4)/ groupassignment = groupnumber/ file = "RTB.iqx"/ file = "Control IBT.iqx"/ file = "RTA.iqxThis is to ensure that we can identify that positive IBT does exert an effect that is not confounded by materials Set presentation (to rule out a material presentation effect, in this case A and B, we very much don’t want this , hence the above Okay, that looks correct to me now. Group ID will be logged in the data files (unless you deliberatey excluded it), and you can derive order (RTA first vs RTB first) from that. As for how condition assignment works generally, see https://www.millisecond.com/forums/Topic13856.aspx Hello Dave Hope all is well. So it turns out this set-up is not quite right and may still impose an element of order effect (as it is sequential). I need to check whether the following is implementable within Inquisit, or whether this needs to be done in Qualtrics . Given there are 4 'conditions' in the above, we need to create the following array, where the length is determined by maximum number of participants (we hope this to be 150). I have gone for 20 below for sake of ease: Condition = [ 1 3 2 4 4 3 1 2 2 3 4 1 1 2 3 4 4 2 1 3] The condition for each participant would be based on the order in which they are run, such that the first participant would be allocated to condition 1 (the first number in the condition array), the second person would be allocate to condition 3 (the second number in the condition array) etc. the array is constructed such that the order of every 4 participants is random but includes one in each condition i.e. above you have 1 3 2 4 followed by 4 3 1 2 etc. In this way you know that provided you run a multiple of 4 participants you will always have an equal number in each of the 4 conditions. Is there a way I can achieve this? If you select random (without replacement) group number generation in the Inquisit Web experiment's settings and indicate the number of groups as 4, then for each set of four participants that visit the start page, one will be randomly assigned to condition 1, one will be randomly assigned to condition 2, one to condition 3, one to condition 4. If this is not to your liking, you can have Qualtrics perform the condition assignment and have it send group ID over via URL parameter, just like subject ID. Sounds like this set-up achieves what we need it to, in other words participant 1 could be assigned to any one of the 4, let's say at random, 3rd condition, the second participant gets assigned to the 1st condition, and then the final two participants will be randomly allocated to one of the final remaining conditions (e.g. 2nd and 4th condition in this case). Then this randomizing process restarts for the next 4 participants (with a different randomised order/allocation) and so on. If I have understood this correctly, then we should not need Qualtrics, and this achieves the array above (random 50% of each training condition get RTA followed by RTB, and the other 50% the other way around) Thank you Dave again, this is very much appreciated
> in other words participant 1 could be assigned to any one of the 4, let's say at random, 3rd condition, the second participant gets assigned to the 1st condition, and then the final two participants will be randomly allocated to one of the final remaining conditions (e.g. 2nd and 4th condition in this case)
Yes, that's how it works. Note, though, that group ID generation happens when the participant visits the start page, regardless of whether that particpant ends up actually launching the study at that point or not. So you will see gaps due to dropouts, that's just a reality of online research
|
By AnthonyFCollinsSussex - 5/14/2021
+x+x+x+x+x+xOne of those days, my bad :// order RTA -> Positive IBT -> RTB / subjects = (1 of 4)/ groupassignment = groupnumber / file = "RTA.iqx"/ file = "Positive IBT.iqx"/ file = "RTB.iqx// order RTA -> Control IBT -> RTB / subjects = (2 of 4)/ groupassignment = groupnumber/ file = "RTA.iqx"/ file = "Control IBT.iqx"/ file = "RTB.iqx// order RTB -> Positive IBT -> RTA / subjects = (3 of 4)/ groupassignment = groupnumber/ file = "RTB.iqx"/ file = "Positive IBT.iqx"/ file = "RTA.iqx// order RTB -> Control IBT -> RTA / subjects = (4 of 4)/ groupassignment = groupnumber/ file = "RTB.iqx"/ file = "Control IBT.iqx"/ file = "RTA.iqxThis is to ensure that we can identify that positive IBT does exert an effect that is not confounded by materials Set presentation (to rule out a material presentation effect, in this case A and B, we very much don’t want this , hence the above Okay, that looks correct to me now. Group ID will be logged in the data files (unless you deliberatey excluded it), and you can derive order (RTA first vs RTB first) from that. As for how condition assignment works generally, see https://www.millisecond.com/forums/Topic13856.aspx Hello Dave Hope all is well. So it turns out this set-up is not quite right and may still impose an element of order effect (as it is sequential). I need to check whether the following is implementable within Inquisit, or whether this needs to be done in Qualtrics . Given there are 4 'conditions' in the above, we need to create the following array, where the length is determined by maximum number of participants (we hope this to be 150). I have gone for 20 below for sake of ease: Condition = [ 1 3 2 4 4 3 1 2 2 3 4 1 1 2 3 4 4 2 1 3] The condition for each participant would be based on the order in which they are run, such that the first participant would be allocated to condition 1 (the first number in the condition array), the second person would be allocate to condition 3 (the second number in the condition array) etc. the array is constructed such that the order of every 4 participants is random but includes one in each condition i.e. above you have 1 3 2 4 followed by 4 3 1 2 etc. In this way you know that provided you run a multiple of 4 participants you will always have an equal number in each of the 4 conditions. Is there a way I can achieve this? If you select random (without replacement) group number generation in the Inquisit Web experiment's settings and indicate the number of groups as 4, then for each set of four participants that visit the start page, one will be randomly assigned to condition 1, one will be randomly assigned to condition 2, one to condition 3, one to condition 4. If this is not to your liking, you can have Qualtrics perform the condition assignment and have it send group ID over via URL parameter, just like subject ID. Sounds like this set-up achieves what we need it to, in other words participant 1 could be assigned to any one of the 4, let's say at random, 3rd condition, the second participant gets assigned to the 1st condition, and then the final two participants will be randomly allocated to one of the final remaining conditions (e.g. 2nd and 4th condition in this case). Then this randomizing process restarts for the next 4 participants (with a different randomised order/allocation) and so on. If I have understood this correctly, then we should not need Qualtrics, and this achieves the array above (random 50% of each training condition get RTA followed by RTB, and the other 50% the other way around) Thank you Dave again, this is very much appreciated > in other words participant 1 could be assigned to any one of the 4, let's say at random, 3rd condition, the second participant gets assigned to the 1st condition, and then the final two participants will be randomly allocated to one of the final remaining conditions (e.g. 2nd and 4th condition in this case) Yes, that's how it works. Note, though, that group ID generation happens when the participant visits the start page, regardless of whether that particpant ends up actually launching the study at that point or not. So you will see gaps due to dropouts, that's just a reality of online research
This is very useful to know thank you (in fact vital to know), I will pass this on to my team and then we can decide whether to take this risk, but as you indeed say, it is one such inevitability
Thank you so much!!!
|
By AnthonyFCollinsSussex - 5/21/2021
+x+x+x+x+x+xOne of those days, my bad :// order RTA -> Positive IBT -> RTB / subjects = (1 of 4)/ groupassignment = groupnumber / file = "RTA.iqx"/ file = "Positive IBT.iqx"/ file = "RTB.iqx// order RTA -> Control IBT -> RTB / subjects = (2 of 4)/ groupassignment = groupnumber/ file = "RTA.iqx"/ file = "Control IBT.iqx"/ file = "RTB.iqx// order RTB -> Positive IBT -> RTA / subjects = (3 of 4)/ groupassignment = groupnumber/ file = "RTB.iqx"/ file = "Positive IBT.iqx"/ file = "RTA.iqx// order RTB -> Control IBT -> RTA / subjects = (4 of 4)/ groupassignment = groupnumber/ file = "RTB.iqx"/ file = "Control IBT.iqx"/ file = "RTA.iqxThis is to ensure that we can identify that positive IBT does exert an effect that is not confounded by materials Set presentation (to rule out a material presentation effect, in this case A and B, we very much don’t want this , hence the above Okay, that looks correct to me now. Group ID will be logged in the data files (unless you deliberatey excluded it), and you can derive order (RTA first vs RTB first) from that. As for how condition assignment works generally, see https://www.millisecond.com/forums/Topic13856.aspx Hello Dave Hope all is well. So it turns out this set-up is not quite right and may still impose an element of order effect (as it is sequential). I need to check whether the following is implementable within Inquisit, or whether this needs to be done in Qualtrics . Given there are 4 'conditions' in the above, we need to create the following array, where the length is determined by maximum number of participants (we hope this to be 150). I have gone for 20 below for sake of ease: Condition = [ 1 3 2 4 4 3 1 2 2 3 4 1 1 2 3 4 4 2 1 3] The condition for each participant would be based on the order in which they are run, such that the first participant would be allocated to condition 1 (the first number in the condition array), the second person would be allocate to condition 3 (the second number in the condition array) etc. the array is constructed such that the order of every 4 participants is random but includes one in each condition i.e. above you have 1 3 2 4 followed by 4 3 1 2 etc. In this way you know that provided you run a multiple of 4 participants you will always have an equal number in each of the 4 conditions. Is there a way I can achieve this? If you select random (without replacement) group number generation in the Inquisit Web experiment's settings and indicate the number of groups as 4, then for each set of four participants that visit the start page, one will be randomly assigned to condition 1, one will be randomly assigned to condition 2, one to condition 3, one to condition 4. If this is not to your liking, you can have Qualtrics perform the condition assignment and have it send group ID over via URL parameter, just like subject ID. Sounds like this set-up achieves what we need it to, in other words participant 1 could be assigned to any one of the 4, let's say at random, 3rd condition, the second participant gets assigned to the 1st condition, and then the final two participants will be randomly allocated to one of the final remaining conditions (e.g. 2nd and 4th condition in this case). Then this randomizing process restarts for the next 4 participants (with a different randomised order/allocation) and so on. If I have understood this correctly, then we should not need Qualtrics, and this achieves the array above (random 50% of each training condition get RTA followed by RTB, and the other 50% the other way around) Thank you Dave again, this is very much appreciated > in other words participant 1 could be assigned to any one of the 4, let's say at random, 3rd condition, the second participant gets assigned to the 1st condition, and then the final two participants will be randomly allocated to one of the final remaining conditions (e.g. 2nd and 4th condition in this case) Yes, that's how it works. Note, though, that group ID generation happens when the participant visits the start page, regardless of whether that particpant ends up actually launching the study at that point or not. So you will see gaps due to dropouts, that's just a reality of online research
Hello Dave
I hope you are well and that you have had a pleasant week so far. I do need to check the algorithm concerning random without replacement on the Inquisit web-player. Just to re-cap:
My batch is the following:
<batch>
/ subjects = (1 of 4)
/ groupassignment = groupnumber
/ file = RTA.iqx
/ file = Positive IBT.iqx
/ file = RTB.iqx
</batch>
<batch>
/ subjects = (2 of 4)
/ groupassignment = groupnumber
/ file = RTB.iqx
/ file = Control IBT.iqx
/ file = RTA.iqx
</batch>
/ subjects = (3 of 4)
/ groupassignment = groupnumber
/ file = RTB.iqx
/ file = Positive IBT.iqx
/ file = RTA.iqx
</batch>
/ subjects = (4 of 4)
/ groupassignment = groupnumber
/ file = RTA.iqx
/ file = Control IBT.iqx
/ file = RTB.iqx
</batch>
<defaults>
/ screencolor = white
/ txcolor = black
/ txbgcolor = white
/ fontstyle = ("Arial", 2.8%)
</defaults>
<data>
/separatefiles = true
</data>
Thus, the order of every 4 participants is random but includes one in each condition, and therefore each condition ought only be summoned once in every set of 4 participants (array). To do so, accordingly as instructed above, I ensured that I selected random (without replacement) group number generation, as well as for subject ID generation, in the Inquisit Web experiment's settings, and indicated the number of groups as 4.
I conducted some runs to ensure the allocation worked correctly. This simply entails me ctrl+Q'ing my through my experiment each time (for speed) and checking one of my script's group id variable in the data output. I have so far conducted many runs and found some weird results. This is the first lot when I first uploaded the experiment.
1, 3, 2, 4
2, 2, 2, 2, <-- I now realise that this was because I stupidly used the same start-up screen for those 4 consecutive trials.
3, 1, 4, 1
3, 2, 2, 4
2, 1, 3, 4
2, 1, 3, 4
3, 2, 1, 4
2, 3, 1, 4
2, 3, 1, 4
1, 3, 2, 4
1, 2, 3, 4
2, 3, 1, 4
1, 2, 3, 4
2, 1, 3, 4
2, 3, 1, 4
After unregistering/deleting the exp, and merely re-uploading the exact same one again, I obtained the following:
2, 1, 3, 4
2, 1, 3, 4
3, 2, 1, 4
3, 2, 1, 4
3, 1, 2, 4
1, 2, 3, 4
2, 1, 3, 4
3, 2, 4, 1
I proceeded to upload the same experiment again, and finally got the following:
3, 1, 2, 4
3, 2, 1, 4
3, 2, 1, 4
1, 3, 2, 4
3, 1, 4, 3 <-- On the last turn here, I took a longer pause from the previous launch (4th condition), and weirdly...it used the same condition as at the start, 3
2, 1, 4, 2 <-- And again the same as above, merely for a slightly longer time interval between run 3 and run 4.
Finally, I notice from all my test-runs, not one of them starts with the 4th condition. Is that coincidental? Is there something in the algorithm that is leading to this? There is nothing in my code to suggest why this is happening, and I am genuinely very stumped! I just wondered if these 'random' sequences looked normal.
Thank you very much in advance...
Anthony
|
By Dave - 5/21/2021
+x+x+x+x+x+x+xOne of those days, my bad :// order RTA -> Positive IBT -> RTB / subjects = (1 of 4)/ groupassignment = groupnumber / file = "RTA.iqx"/ file = "Positive IBT.iqx"/ file = "RTB.iqx// order RTA -> Control IBT -> RTB / subjects = (2 of 4)/ groupassignment = groupnumber/ file = "RTA.iqx"/ file = "Control IBT.iqx"/ file = "RTB.iqx// order RTB -> Positive IBT -> RTA / subjects = (3 of 4)/ groupassignment = groupnumber/ file = "RTB.iqx"/ file = "Positive IBT.iqx"/ file = "RTA.iqx// order RTB -> Control IBT -> RTA / subjects = (4 of 4)/ groupassignment = groupnumber/ file = "RTB.iqx"/ file = "Control IBT.iqx"/ file = "RTA.iqxThis is to ensure that we can identify that positive IBT does exert an effect that is not confounded by materials Set presentation (to rule out a material presentation effect, in this case A and B, we very much don’t want this , hence the above Okay, that looks correct to me now. Group ID will be logged in the data files (unless you deliberatey excluded it), and you can derive order (RTA first vs RTB first) from that. As for how condition assignment works generally, see https://www.millisecond.com/forums/Topic13856.aspx Hello Dave Hope all is well. So it turns out this set-up is not quite right and may still impose an element of order effect (as it is sequential). I need to check whether the following is implementable within Inquisit, or whether this needs to be done in Qualtrics . Given there are 4 'conditions' in the above, we need to create the following array, where the length is determined by maximum number of participants (we hope this to be 150). I have gone for 20 below for sake of ease: Condition = [ 1 3 2 4 4 3 1 2 2 3 4 1 1 2 3 4 4 2 1 3] The condition for each participant would be based on the order in which they are run, such that the first participant would be allocated to condition 1 (the first number in the condition array), the second person would be allocate to condition 3 (the second number in the condition array) etc. the array is constructed such that the order of every 4 participants is random but includes one in each condition i.e. above you have 1 3 2 4 followed by 4 3 1 2 etc. In this way you know that provided you run a multiple of 4 participants you will always have an equal number in each of the 4 conditions. Is there a way I can achieve this? If you select random (without replacement) group number generation in the Inquisit Web experiment's settings and indicate the number of groups as 4, then for each set of four participants that visit the start page, one will be randomly assigned to condition 1, one will be randomly assigned to condition 2, one to condition 3, one to condition 4. If this is not to your liking, you can have Qualtrics perform the condition assignment and have it send group ID over via URL parameter, just like subject ID. Sounds like this set-up achieves what we need it to, in other words participant 1 could be assigned to any one of the 4, let's say at random, 3rd condition, the second participant gets assigned to the 1st condition, and then the final two participants will be randomly allocated to one of the final remaining conditions (e.g. 2nd and 4th condition in this case). Then this randomizing process restarts for the next 4 participants (with a different randomised order/allocation) and so on. If I have understood this correctly, then we should not need Qualtrics, and this achieves the array above (random 50% of each training condition get RTA followed by RTB, and the other 50% the other way around) Thank you Dave again, this is very much appreciated > in other words participant 1 could be assigned to any one of the 4, let's say at random, 3rd condition, the second participant gets assigned to the 1st condition, and then the final two participants will be randomly allocated to one of the final remaining conditions (e.g. 2nd and 4th condition in this case) Yes, that's how it works. Note, though, that group ID generation happens when the participant visits the start page, regardless of whether that particpant ends up actually launching the study at that point or not. So you will see gaps due to dropouts, that's just a reality of online research Hello Dave I hope you are well and that you have had a pleasant week so far. I do need to check the algorithm concerning random without replacement on the Inquisit web-player. Just to re-cap: My batch is the following: <batch> / subjects = (1 of 4) / groupassignment = groupnumber / file = RTA.iqx / file = Positive IBT.iqx / file = RTB.iqx </batch> <batch> / subjects = (2 of 4) / groupassignment = groupnumber / file = RTB.iqx / file = Control IBT.iqx / file = RTA.iqx </batch> / subjects = (3 of 4) / groupassignment = groupnumber / file = RTB.iqx / file = Positive IBT.iqx / file = RTA.iqx </batch> / subjects = (4 of 4) / groupassignment = groupnumber / file = RTA.iqx / file = Control IBT.iqx / file = RTB.iqx </batch> <defaults> / screencolor = white / txcolor = black / txbgcolor = white / fontstyle = ("Arial", 2.8%) </defaults> <data> /separatefiles = true </data> Thus, the order of every 4 participants is random but includes one in each condition, and therefore each condition ought only be summoned once in every set of 4 participants (array). To do so, accordingly as instructed above, I ensured that I selected random (without replacement) group number generation, as well as for subject ID generation, in the Inquisit Web experiment's settings, and indicated the number of groups as 4. I conducted some runs to ensure the allocation worked correctly. This simply entails me ctrl+Q'ing my through my experiment each time (for speed) and checking one of my script's group id variable in the data output. I have so far conducted many runs and found some weird results. This is the first lot when I first uploaded the experiment. 1, 3, 2, 4 2, 2, 2, 2, <-- I now realise that this was because I stupidly used the same start-up screen for those 4 consecutive trials. 3, 1, 4, 1 3, 2, 2, 4 2, 1, 3, 4 2, 1, 3, 4 3, 2, 1, 4 2, 3, 1, 4 2, 3, 1, 4 1, 3, 2, 4 1, 2, 3, 4 2, 3, 1, 4 1, 2, 3, 4 2, 1, 3, 4 2, 3, 1, 4 After unregistering/deleting the exp, and merely re-uploading the exact same one again, I obtained the following: 2, 1, 3, 4 2, 1, 3, 4 3, 2, 1, 4 3, 2, 1, 4 3, 1, 2, 4 1, 2, 3, 4 2, 1, 3, 4 3, 2, 4, 1 I proceeded to upload the same experiment again, and finally got the following: 3, 1, 2, 4 3, 2, 1, 4 3, 2, 1, 4 1, 3, 2, 4 3, 1, 4, 3 <-- On the last turn here, I took a longer pause from the previous launch (4th condition), and weirdly...it used the same condition as at the start, 3 2, 1, 4, 2 <-- And again the same as above, merely for a slightly longer time interval between run 3 and run 4. Finally, I notice from all my test-runs, not one of them starts with the 4th condition. Is that coincidental? Is there something in the algorithm that is leading to this? There is nothing in my code to suggest why this is happening, and I am genuinely very stumped! I just wondered if these 'random' sequences looked normal. Thank you very much in advance... Anthony
I fail to see any issue here. As explained previously, any visit to the start page, by yourself or somebody else, will trigger generation of a group number, as will any extraneous refresh of the page. In the given set of 4, that ID then is out of the picture and will only re-occur in the next set of 4.
|
By AnthonyFCollinsSussex - 5/21/2021
+x+x+x+x+x+x+x+xOne of those days, my bad :// order RTA -> Positive IBT -> RTB / subjects = (1 of 4)/ groupassignment = groupnumber / file = "RTA.iqx"/ file = "Positive IBT.iqx"/ file = "RTB.iqx// order RTA -> Control IBT -> RTB / subjects = (2 of 4)/ groupassignment = groupnumber/ file = "RTA.iqx"/ file = "Control IBT.iqx"/ file = "RTB.iqx// order RTB -> Positive IBT -> RTA / subjects = (3 of 4)/ groupassignment = groupnumber/ file = "RTB.iqx"/ file = "Positive IBT.iqx"/ file = "RTA.iqx// order RTB -> Control IBT -> RTA / subjects = (4 of 4)/ groupassignment = groupnumber/ file = "RTB.iqx"/ file = "Control IBT.iqx"/ file = "RTA.iqxThis is to ensure that we can identify that positive IBT does exert an effect that is not confounded by materials Set presentation (to rule out a material presentation effect, in this case A and B, we very much don’t want this , hence the above Okay, that looks correct to me now. Group ID will be logged in the data files (unless you deliberatey excluded it), and you can derive order (RTA first vs RTB first) from that. As for how condition assignment works generally, see https://www.millisecond.com/forums/Topic13856.aspx Hello Dave Hope all is well. So it turns out this set-up is not quite right and may still impose an element of order effect (as it is sequential). I need to check whether the following is implementable within Inquisit, or whether this needs to be done in Qualtrics . Given there are 4 'conditions' in the above, we need to create the following array, where the length is determined by maximum number of participants (we hope this to be 150). I have gone for 20 below for sake of ease: Condition = [ 1 3 2 4 4 3 1 2 2 3 4 1 1 2 3 4 4 2 1 3] The condition for each participant would be based on the order in which they are run, such that the first participant would be allocated to condition 1 (the first number in the condition array), the second person would be allocate to condition 3 (the second number in the condition array) etc. the array is constructed such that the order of every 4 participants is random but includes one in each condition i.e. above you have 1 3 2 4 followed by 4 3 1 2 etc. In this way you know that provided you run a multiple of 4 participants you will always have an equal number in each of the 4 conditions. Is there a way I can achieve this? If you select random (without replacement) group number generation in the Inquisit Web experiment's settings and indicate the number of groups as 4, then for each set of four participants that visit the start page, one will be randomly assigned to condition 1, one will be randomly assigned to condition 2, one to condition 3, one to condition 4. If this is not to your liking, you can have Qualtrics perform the condition assignment and have it send group ID over via URL parameter, just like subject ID. Sounds like this set-up achieves what we need it to, in other words participant 1 could be assigned to any one of the 4, let's say at random, 3rd condition, the second participant gets assigned to the 1st condition, and then the final two participants will be randomly allocated to one of the final remaining conditions (e.g. 2nd and 4th condition in this case). Then this randomizing process restarts for the next 4 participants (with a different randomised order/allocation) and so on. If I have understood this correctly, then we should not need Qualtrics, and this achieves the array above (random 50% of each training condition get RTA followed by RTB, and the other 50% the other way around) Thank you Dave again, this is very much appreciated > in other words participant 1 could be assigned to any one of the 4, let's say at random, 3rd condition, the second participant gets assigned to the 1st condition, and then the final two participants will be randomly allocated to one of the final remaining conditions (e.g. 2nd and 4th condition in this case) Yes, that's how it works. Note, though, that group ID generation happens when the participant visits the start page, regardless of whether that particpant ends up actually launching the study at that point or not. So you will see gaps due to dropouts, that's just a reality of online research Hello Dave I hope you are well and that you have had a pleasant week so far. I do need to check the algorithm concerning random without replacement on the Inquisit web-player. Just to re-cap: My batch is the following: <batch> / subjects = (1 of 4) / groupassignment = groupnumber / file = RTA.iqx / file = Positive IBT.iqx / file = RTB.iqx </batch> <batch> / subjects = (2 of 4) / groupassignment = groupnumber / file = RTB.iqx / file = Control IBT.iqx / file = RTA.iqx </batch> / subjects = (3 of 4) / groupassignment = groupnumber / file = RTB.iqx / file = Positive IBT.iqx / file = RTA.iqx </batch> / subjects = (4 of 4) / groupassignment = groupnumber / file = RTA.iqx / file = Control IBT.iqx / file = RTB.iqx </batch> <defaults> / screencolor = white / txcolor = black / txbgcolor = white / fontstyle = ("Arial", 2.8%) </defaults> <data> /separatefiles = true </data> Thus, the order of every 4 participants is random but includes one in each condition, and therefore each condition ought only be summoned once in every set of 4 participants (array). To do so, accordingly as instructed above, I ensured that I selected random (without replacement) group number generation, as well as for subject ID generation, in the Inquisit Web experiment's settings, and indicated the number of groups as 4. I conducted some runs to ensure the allocation worked correctly. This simply entails me ctrl+Q'ing my through my experiment each time (for speed) and checking one of my script's group id variable in the data output. I have so far conducted many runs and found some weird results. This is the first lot when I first uploaded the experiment. 1, 3, 2, 4 2, 2, 2, 2, <-- I now realise that this was because I stupidly used the same start-up screen for those 4 consecutive trials. 3, 1, 4, 1 3, 2, 2, 4 2, 1, 3, 4 2, 1, 3, 4 3, 2, 1, 4 2, 3, 1, 4 2, 3, 1, 4 1, 3, 2, 4 1, 2, 3, 4 2, 3, 1, 4 1, 2, 3, 4 2, 1, 3, 4 2, 3, 1, 4 After unregistering/deleting the exp, and merely re-uploading the exact same one again, I obtained the following: 2, 1, 3, 4 2, 1, 3, 4 3, 2, 1, 4 3, 2, 1, 4 3, 1, 2, 4 1, 2, 3, 4 2, 1, 3, 4 3, 2, 4, 1 I proceeded to upload the same experiment again, and finally got the following: 3, 1, 2, 4 3, 2, 1, 4 3, 2, 1, 4 1, 3, 2, 4 3, 1, 4, 3 <-- On the last turn here, I took a longer pause from the previous launch (4th condition), and weirdly...it used the same condition as at the start, 3 2, 1, 4, 2 <-- And again the same as above, merely for a slightly longer time interval between run 3 and run 4. Finally, I notice from all my test-runs, not one of them starts with the 4th condition. Is that coincidental? Is there something in the algorithm that is leading to this? There is nothing in my code to suggest why this is happening, and I am genuinely very stumped! I just wondered if these 'random' sequences looked normal. Thank you very much in advance... Anthony I fail to see any issue here. As explained previously, any visit to the start page, by yourself or somebody else, will trigger generation of a group number, as will any extraneous refresh of the page. In the given set of 4, that ID then is out of the picture and will only re-occur in the next set of 4.
It just looks as though for example condition 4 is always being landed at the end of most sets, and I was wondering if the algorithm was truly random, should this happen, as it looks so systematic
|
|