Millisecond Forums » Millisecond Forums » Inquisit 6 » Issues with counterbalancing and sequential groupid

Issues with counterbalancing and sequential groupid

annievu94 — Mon, 23 Nov 2020 20:15:56 GMT

I linked my study with SONA so I set my participants ID as URL parameter and groupid as sequential, hoping that students will be counterbalanced into 1 of the 8 conditions per the sequence that they click on the link (so each of my condition will be counterbalanced and has even number of participants). However, the order is somewhat random? Please see attached and advise how I should proceed!

RE: Issues with counterbalancing and sequential groupid

annievu94 — Mon, 23 Nov 2020 20:15:56 GMT

Thank you so much Dave!!! I really appreciate all the help!!!

RE: Issues with counterbalancing and sequential groupid

Dave — Mon, 23 Nov 2020 19:45:54 GMT

+x[quote]

[b]annievu94 - 11/23/2020[/b]
+x[quote]

[b]Dave - 11/23/2020[/b]
+x[quote]

[b]annievu94 - 11/23/2020[/b]
Thank you Dave for the clarification!
So does set the group setting as either Randomization With Replacement or Randomization Without Replacement will solve this problem? I just want equal sample sizes for each condition!
[/quote]

Neither will solve the problem. You're going to have dropouts or people who take a peak at the start page, but don't start the experiment right then and there and instead return later. That will lead to imbalances no matter which group number generation method you choose.
[/quote]

Thank you Dave!

Last question, since this is my first time running online experiment on Inquisit and I couldn't find the answer on the forum, what is the difference between "Random Assignment With Replacement" and "Random Assignment without Replacement"?
[/quote]

Same thing as drawing from a deck of cards (conditions) with or without putting the drawn card back into the deck after the draw.

Suppose you have 4 groups. Then, for every set of 4 participants that hit the start page, random assignment without replacement will assign one of those participants to group 1, one to group 2, one to group 3, and one to group 4.

With random assignment with replacement, every participant that hits the start page has en equal chance of being assigned to any of the 4 groups, essentially as if flipping a coin.

RE: Issues with counterbalancing and sequential groupid

annievu94 — Mon, 23 Nov 2020 19:31:27 GMT

+x[quote]

[b]Dave - 11/23/2020[/b]
+x[quote]

[b]annievu94 - 11/23/2020[/b]
Thank you Dave for the clarification!
So does set the group setting as either Randomization With Replacement or Randomization Without Replacement will solve this problem? I just want equal sample sizes for each condition!
[/quote]

Neither will solve the problem. You're going to have dropouts or people who take a peak at the start page, but don't start the experiment right then and there and instead return later. That will lead to imbalances no matter which group number generation method you choose.
[/quote]

Thank you Dave!

Last question, since this is my first time running online experiment on Inquisit and I couldn't find the answer on the forum, what is the difference between "Random Assignment With Replacement" and "Random Assignment without Replacement"?

RE: Issues with counterbalancing and sequential groupid

Dave — Mon, 23 Nov 2020 19:13:02 GMT

+x[quote]

[b]annievu94 - 11/23/2020[/b]
Thank you Dave for the clarification!
So does set the group setting as either Randomization With Replacement or Randomization Without Replacement will solve this problem? I just want equal sample sizes for each condition!
[/quote]

Neither will solve the problem. You're going to have dropouts or people who take a peak at the start page, but don't start the experiment right then and there and instead return later. That will lead to imbalances no matter which group number generation method you choose.

RE: Issues with counterbalancing and sequential groupid

annievu94 — Mon, 23 Nov 2020 18:36:56 GMT

Thank you Dave for the clarification!
So does set the group setting as either Randomization With Replacement or Randomization Without Replacement will solve this problem? I just want equal sample sizes for each condition!

RE: Issues with counterbalancing and sequential groupid

Dave — Mon, 23 Nov 2020 12:50:34 GMT

+x[quote]

[b]annievu94 - 11/21/2020[/b]
I linked my study with SONA so I set my participants ID as URL parameter and groupid as sequential, hoping that students will be counterbalanced into 1 of the 8 conditions per the sequence that they click on the link (so each of my condition will be counterbalanced and has even number of participants). However, the order is somewhat random? Please see attached and advise how I should proceed!
[/quote]

Sort your logs ascending by Event Time and it won't look so random:

Note that a group number will be generated whenever someone visits the study's start page, regardless of whether that participant then starts and completes the study at that point or not. That is why you will see gaps between the sequential IDs (e.g. 3 and 4 missing between 2 and 5 in the above).