Millisecond Forums » Millisecond Forums » Inquisit 6 » D scores not calculating as expected

RE: D scores not calculating as expected

Dave — Thu, 15 Oct 2020 23:15:56 GMT

+x[quote]

[b]crstevenson - 10/14/2020[/b]

Hi Dave,

Thanks for all this! Even after I do an inclusive SD like you described, I still don't get the same answer. I've attached the updated spreadsheet here. This is a screenshot of the values and the result I'm getting. Here are the steps I go through - maybe there is something wrong in the general steps I'm doing?

1. calculate SD for practice blocks and real blocks
2. calculate mean for incompatible and compatible practice blocks
3. find the mean difference for practice blocks ((incompatible mean - compatible mean)/2)
4. D_Pract = mean diff practice/SD for practice blocks
5. calculate mean for incompatible and compatible real blocks
6. find the mean difference for real blocks ((incompatible mean - compatible mean)/2)
7. D_Real = mean diff real/SD for real blocks
8. D = (D_Prac + D_Real)/2

Thank you so much for your help!

Cassandra
[/quote]

You have mistakes in your script that lead to the exclusion of certain trials from the D-scores which should not be excluded.

You uses the picture IAT template as the basis for your script. https://www.millisecond.com/download/library/v6/iat/iattemplates/pictureiat/pictureiat/pictureiat.iqzip

Look at this:

<block compatibletest1>
/ bgstim = (targetAleftmixed, orleft, attributeAleft, targetBrightmixed, orright, attributeBright)
/ trials = [
    1=instructions;
    3,5,7,9,11,13,15,17,19,21= random(targetAleft, targetBright);
    2,4,6,8,10,12,14,16,18,20 = random(attributeA, attributeB)]
/ errormessage = true(error,200)
/ responsemode = correct
/ ontrialend = [
    if(block.compatibletest1.latency <= 10000 && block.compatibletest1.currenttrialnumber != 1 ) {
        values.sum1a += block.compatibletest1.latency;
        values.n1a += 1;
        values.ss1a += (block.compatibletest1.latency * block.compatibletest1.latency);
        values.n_correct += block.compatibletest1.correct;
    };

    if(block.compatibletest1.latency < 300) {
        list.RT300.appenditem(1);
    } else {
        list.RT300.appenditem(0);
    };
]
</block>

The latency for the first trial in the "practice" blocks is not included in the calculations. Why? Because here the first trial is an instructions trial.

Now look at your various "practice" blocks:

<block compatiblepractice_WMMath_AFArts>
/ bgstim = (faceAleftmixed, wordAleft, faceBrightmixed, wordBright)
/ trials = [
    1,3,5,7,9,11,13,15,17,19= random(faceAleft, faceBright);
    2,4,6,8,10,12,14,16,18,20 = random(wordAL, wordBR)]
/ errormessage = true(error,200)
/ responsemode = correct
/ onblockbegin = [
parameters.showtoofast = true;
]
/ ontrialend = [values.n1a += 1;
    if(block.compatiblepractice_WMMath_AFArts.latency <= 10000 && block.compatiblepractice_WMMath_AFArts.currenttrialnumber != 1 && script.currenttrial != "toofast") {
        values.sum1a += block.compatiblepractice_WMMath_AFArts.latency;
        values.ss1a += (block.compatiblepractice_WMMath_AFArts.latency * block.compatiblepractice_WMMath_AFArts.latency);
        values.n_correct += block.compatiblepractice_WMMath_AFArts.correct;
    };
    if(block.compatiblepractice_WMMath_AFArts.latency < 300 && script.currenttrial != "toofast") {values.toofast +=1;

    } else {values.toofast = 0;

    };
    if(block.compatiblepractice_WMMath_AFArts.latency < 300 && script.currenttrial != "toofast") {
        list.RT300.appenditem(1);
    } else {
        list.RT300.appenditem(0);
    };

]
</block>

Here, too, you are excluding anything that happens in the first trial from the calculations. But your first trial isn't an instructions trial -- it's an experimental trial.

Then you have a serious mistake in your various incompatible blocks:

<block incompatiblepractice_AFMath_WMArts>
/ bgstim = (faceBleftmixed, wordAleft, faceArightmixed, wordBright)
/ trials = [
    1,3,5,7,9,11,13,15,17,19 = random(faceBleft, faceAright);
    2,4,6,8,10,12,14,16,18,20 = random(wordAL, wordBR)]
/ errormessage = true(error,200)
/ responsemode = correct
/ onblockbegin = [
parameters.showtoofast = true;
]
/ ontrialend = [values.n2a += 1;
    if(block.incompatiblepractice_AFMath_WMArts.latency <= 10000 && block.incompatiblepractice_AFMath_WMArts.currenttrialnumber != 1 && script.currenttrial != "toofast") {
        values.sum2a += block.incompatiblepractice_AFMath_WMArts.latency;
        values.n2a += 1;
        values.ss2a += (block.incompatiblepractice_AFMath_WMArts.latency * block.incompatiblepractice_AFMath_WMArts.latency);
        values.n_correct += block.incompatiblepractice_AFMath_WMArts.correct;
    };

    if(block.incompatiblepractice_AFMath_WMArts.latency < 300 && script.currenttrial != "toofast") {values.toofast +=1;

    } else {values.toofast = 0;

    };

    if(block.incompatiblepractice_AFMath_WMArts.latency < 300 && script.currenttrial != "toofast") {
        list.RT300.appenditem(1);
    } else {
        list.RT300.appenditem(0);
    };
]
</block>

<block incompatiblepractice_WMArts_AFMath>
/ bgstim = (faceAleftmixed, wordBleft, faceBrightmixed, wordAright)
/ trials = [
    1,3,5,7,9,11,13,15,17,19 = random(faceAleft, faceBright);
    2,4,6,8,10,12,14,16,18,20 = random(wordBL, wordAR)]
/ errormessage = true(error,200)
/ responsemode = correct
/ onblockbegin = [
parameters.showtoofast = true;
]
/ ontrialend = [values.n2a += 1;
    if(block.incompatiblepractice_WMArts_AFMath.latency <= 10000 && block.incompatiblepractice_WMArts_AFMath.currenttrialnumber != 1 && script.currenttrial != "toofast") {
        values.sum2a += block.incompatiblepractice_WMArts_AFMath.latency;
        values.n2a += 1;
        values.ss2a += (block.incompatiblepractice_WMArts_AFMath.latency * block.incompatiblepractice_WMArts_AFMath.latency);
        values.n_correct += block.incompatiblepractice_WMArts_AFMath.correct;
    };

    if(block.incompatiblepractice_WMArts_AFMath.latency < 300 && script.currenttrial != "toofast") {values.toofast +=1;

    } else {values.toofast = 0;

    };

    if(block.incompatiblepractice_WMArts_AFMath.latency < 300 && script.currenttrial != "toofast") {
        list.RT300.appenditem(1);
    } else {
        list.RT300.appenditem(0);
    };
]
</block>

<block incompatibletest_AFMath_WMArts>
/ bgstim = (faceBleftmixed, wordAleft, faceArightmixed, wordBright)
/ trials = [
2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40 = random(faceBleft, faceAright);
1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39 = random(wordAL, wordBR)]
/ errormessage = true(error,200)
/ responsemode = correct
/ onblockbegin = [
parameters.showtoofast = true;
]
/ ontrialend = [values.n2b += 1;
    if(block.incompatibletest_AFMath_WMArts.latency <= 10000 && script.currenttrial != "toofast") {
        values.sum2b += block.incompatibletest_AFMath_WMArts.latency;
        values.n2b += 1;
        values.ss2b += (block.incompatibletest_AFMath_WMArts.latency * block.incompatibletest_AFMath_WMArts.latency);
        values.n_correct += block.incompatibletest_AFMath_WMArts.correct;
    };

    if(block.incompatibletest_AFMath_WMArts.latency < 300 && script.currenttrial != "toofast") {values.toofast +=1;

    } else {values.toofast = 0;

    };


    if(block.incompatibletest_AFMath_WMArts.latency < 300 && script.currenttrial != "toofast") {
        list.RT300.appenditem(1);
    } else {
        list.RT300.appenditem(0);
    };
]
</block>

<block incompatibletest_WMArts_AFMath>
/ bgstim = (faceAleftmixed, wordBleft, faceBrightmixed, wordAright)
/ trials = [
2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40 = random(faceAleft, faceBright);
1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39 = random(wordBL, wordAR)]
/ errormessage = true(error,200)
/ responsemode = correct
/ onblockbegin = [
parameters.showtoofast = true;
]
/ ontrialend = [values.n2b += 1;
    if(block.incompatibletest_WMArts_AFMath.latency <= 10000 && script.currenttrial != "toofast") {
        values.sum2b += block.incompatibletest_WMArts_AFMath.latency;
        values.n2b += 1;
        values.ss2b += (block.incompatibletest_WMArts_AFMath.latency * block.incompatibletest_WMArts_AFMath.latency);
        values.n_correct += block.incompatibletest_WMArts_AFMath.correct;
    };

    if(block.incompatibletest_WMArts_AFMath.latency < 300 && script.currenttrial != "toofast") {values.toofast +=1;

    } else {values.toofast = 0;

    };

    if(block.incompatibletest_WMArts_AFMath.latency < 300 && script.currenttrial != "toofast") {
        list.RT300.appenditem(1);
    } else {
        list.RT300.appenditem(0);
    };
]
</block>

Notice that in these particular blocks you are incrementing n *twice* per trial. Because the first increment (italicized in the above) is unconditional, you'll also increase n in any trials where n shouldn't be increased at all (latency > 10000 or any "toofast" trials). This will obviously completely throw off the calculations.

RE: D scores not calculating as expected

crstevenson — Thu, 15 Oct 2020 23:15:02 GMT

+x[quote]

[b]Dave - 10/15/2020[/b]

You have mistakes in your script that lead to the exclusion of certain trials from the D-scores which should not be excluded.

You uses the picture IAT template as the basis for your script. https://www.millisecond.com/download/library/v6/iat/iattemplates/pictureiat/pictureiat/pictureiat.iqzip

Look at this:

<block compatibletest1>
/ bgstim = (targetAleftmixed, orleft, attributeAleft, targetBrightmixed, orright, attributeBright)
/ trials = [
    1=instructions;
    3,5,7,9,11,13,15,17,19,21= random(targetAleft, targetBright);
    2,4,6,8,10,12,14,16,18,20 = random(attributeA, attributeB)]
/ errormessage = true(error,200)
/ responsemode = correct
/ ontrialend = [
    if(block.compatibletest1.latency <= 10000 && block.compatibletest1.currenttrialnumber != 1 ) {
        values.sum1a += block.compatibletest1.latency;
        values.n1a += 1;
        values.ss1a += (block.compatibletest1.latency * block.compatibletest1.latency);
        values.n_correct += block.compatibletest1.correct;
    };

    if(block.compatibletest1.latency < 300) {
        list.RT300.appenditem(1);
    } else {
        list.RT300.appenditem(0);
    };
]
</block>

The latency for the first trial in the "practice" blocks is not included in the calculations. Why? Because here the first trial is an instructions trial.

Now look at your various "practice" blocks:

<block compatiblepractice_WMMath_AFArts>
/ bgstim = (faceAleftmixed, wordAleft, faceBrightmixed, wordBright)
/ trials = [
    1,3,5,7,9,11,13,15,17,19= random(faceAleft, faceBright);
    2,4,6,8,10,12,14,16,18,20 = random(wordAL, wordBR)]
/ errormessage = true(error,200)
/ responsemode = correct
/ onblockbegin = [
parameters.showtoofast = true;
]
/ ontrialend = [values.n1a += 1;
    if(block.compatiblepractice_WMMath_AFArts.latency <= 10000 && block.compatiblepractice_WMMath_AFArts.currenttrialnumber != 1 && script.currenttrial != "toofast") {
        values.sum1a += block.compatiblepractice_WMMath_AFArts.latency;
        values.ss1a += (block.compatiblepractice_WMMath_AFArts.latency * block.compatiblepractice_WMMath_AFArts.latency);
        values.n_correct += block.compatiblepractice_WMMath_AFArts.correct;
    };
    if(block.compatiblepractice_WMMath_AFArts.latency < 300 && script.currenttrial != "toofast") {values.toofast +=1;

    } else {values.toofast = 0;

    };
    if(block.compatiblepractice_WMMath_AFArts.latency < 300 && script.currenttrial != "toofast") {
        list.RT300.appenditem(1);
    } else {
        list.RT300.appenditem(0);
    };

]
</block>

Here, too, you are excluding anything that happens in the first trial from the calculations. But your first trial isn't an instructions trial -- it's an experimental trial.

Then you have a serious mistake in your various incompatible blocks:

<block incompatiblepractice_AFMath_WMArts>
/ bgstim = (faceBleftmixed, wordAleft, faceArightmixed, wordBright)
/ trials = [
    1,3,5,7,9,11,13,15,17,19 = random(faceBleft, faceAright);
    2,4,6,8,10,12,14,16,18,20 = random(wordAL, wordBR)]
/ errormessage = true(error,200)
/ responsemode = correct
/ onblockbegin = [
parameters.showtoofast = true;
]
/ ontrialend = [values.n2a += 1;
    if(block.incompatiblepractice_AFMath_WMArts.latency <= 10000 && block.incompatiblepractice_AFMath_WMArts.currenttrialnumber != 1 && script.currenttrial != "toofast") {
        values.sum2a += block.incompatiblepractice_AFMath_WMArts.latency;
        values.n2a += 1;
        values.ss2a += (block.incompatiblepractice_AFMath_WMArts.latency * block.incompatiblepractice_AFMath_WMArts.latency);
        values.n_correct += block.incompatiblepractice_AFMath_WMArts.correct;
    };

    if(block.incompatiblepractice_AFMath_WMArts.latency < 300 && script.currenttrial != "toofast") {values.toofast +=1;

    } else {values.toofast = 0;

    };

    if(block.incompatiblepractice_AFMath_WMArts.latency < 300 && script.currenttrial != "toofast") {
        list.RT300.appenditem(1);
    } else {
        list.RT300.appenditem(0);
    };
]
</block>

<block incompatiblepractice_WMArts_AFMath>
/ bgstim = (faceAleftmixed, wordBleft, faceBrightmixed, wordAright)
/ trials = [
    1,3,5,7,9,11,13,15,17,19 = random(faceAleft, faceBright);
    2,4,6,8,10,12,14,16,18,20 = random(wordBL, wordAR)]
/ errormessage = true(error,200)
/ responsemode = correct
/ onblockbegin = [
parameters.showtoofast = true;
]
/ ontrialend = [values.n2a += 1;
    if(block.incompatiblepractice_WMArts_AFMath.latency <= 10000 && block.incompatiblepractice_WMArts_AFMath.currenttrialnumber != 1 && script.currenttrial != "toofast") {
        values.sum2a += block.incompatiblepractice_WMArts_AFMath.latency;
        values.n2a += 1;
        values.ss2a += (block.incompatiblepractice_WMArts_AFMath.latency * block.incompatiblepractice_WMArts_AFMath.latency);
        values.n_correct += block.incompatiblepractice_WMArts_AFMath.correct;
    };

    if(block.incompatiblepractice_WMArts_AFMath.latency < 300 && script.currenttrial != "toofast") {values.toofast +=1;

    } else {values.toofast = 0;

    };

    if(block.incompatiblepractice_WMArts_AFMath.latency < 300 && script.currenttrial != "toofast") {
        list.RT300.appenditem(1);
    } else {
        list.RT300.appenditem(0);
    };
]
</block>

<block incompatibletest_AFMath_WMArts>
/ bgstim = (faceBleftmixed, wordAleft, faceArightmixed, wordBright)
/ trials = [
2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40 = random(faceBleft, faceAright);
1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39 = random(wordAL, wordBR)]
/ errormessage = true(error,200)
/ responsemode = correct
/ onblockbegin = [
parameters.showtoofast = true;
]
/ ontrialend = [values.n2b += 1;
    if(block.incompatibletest_AFMath_WMArts.latency <= 10000 && script.currenttrial != "toofast") {
        values.sum2b += block.incompatibletest_AFMath_WMArts.latency;
        values.n2b += 1;
        values.ss2b += (block.incompatibletest_AFMath_WMArts.latency * block.incompatibletest_AFMath_WMArts.latency);
        values.n_correct += block.incompatibletest_AFMath_WMArts.correct;
    };

    if(block.incompatibletest_AFMath_WMArts.latency < 300 && script.currenttrial != "toofast") {values.toofast +=1;

    } else {values.toofast = 0;

    };


    if(block.incompatibletest_AFMath_WMArts.latency < 300 && script.currenttrial != "toofast") {
        list.RT300.appenditem(1);
    } else {
        list.RT300.appenditem(0);
    };
]
</block>

<block incompatibletest_WMArts_AFMath>
/ bgstim = (faceAleftmixed, wordBleft, faceBrightmixed, wordAright)
/ trials = [
2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40 = random(faceAleft, faceBright);
1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39 = random(wordBL, wordAR)]
/ errormessage = true(error,200)
/ responsemode = correct
/ onblockbegin = [
parameters.showtoofast = true;
]
/ ontrialend = [values.n2b += 1;
    if(block.incompatibletest_WMArts_AFMath.latency <= 10000 && script.currenttrial != "toofast") {
        values.sum2b += block.incompatibletest_WMArts_AFMath.latency;
        values.n2b += 1;
        values.ss2b += (block.incompatibletest_WMArts_AFMath.latency * block.incompatibletest_WMArts_AFMath.latency);
        values.n_correct += block.incompatibletest_WMArts_AFMath.correct;
    };

    if(block.incompatibletest_WMArts_AFMath.latency < 300 && script.currenttrial != "toofast") {values.toofast +=1;

    } else {values.toofast = 0;

    };

    if(block.incompatibletest_WMArts_AFMath.latency < 300 && script.currenttrial != "toofast") {
        list.RT300.appenditem(1);
    } else {
        list.RT300.appenditem(0);
    };
]
</block>

Notice that in these particular blocks you are incrementing n *twice* per trial. Because the first increment (italicized in the above) is unconditional, you'll also increase n in any trials where n shouldn't be increased at all (latency > 1000 or any "toofast" trials). This will obviously completely throw off the calculations.

[/quote]

Hi Dave,

I am SO sorry for wasting your time - I had actually noticed these issues in the code and fixed them a few days ago. I guess I re-uploaded the wrong script to my online experiment so when I was running dummies, I was actually running the wrong script which had all these issues. I just re-uploaded, ran a dummy, and the calculations are all matching up. Thank you so much for all your continued help with this and I am so sorry again! You're the best!

Cassandra

RE: D scores not calculating as expected

Dave — Thu, 15 Oct 2020 00:53:03 GMT

+x[quote]

[b]crstevenson - 10/15/2020[/b]

Hi Dave,

I am SO sorry for wasting your time - I had actually noticed these issues in the code and fixed them a few days ago. I guess I re-uploaded the wrong script to my online experiment so when I was running dummies, I was actually running the wrong script which had all these issues. I just re-uploaded, ran a dummy, and the calculations are all matching up. Thank you so much for all your continued help with this and I am so sorry again! You're the best!

Cassandra
[/quote]

I'm glad that's sorted -- thanks for letting me know!

D scores not calculating as expected

crstevenson — Thu, 15 Oct 2020 00:53:03 GMT

Hello,

We have begun running our experiment, but D scores are not calculating as we'd expect. We hand-calculated some and the results are not coming out the same.
I'd appreciate it if you could let me know if this is a problem in the programming, or a problem in my manual calculations?

These are the steps I'm going through manually (after removing the necessary trials):
1. Compute the standard deviation for each participant’s “practice” trials
2. Compute the standard deviation for each participant’s “real” trials
3. Compute the mean difference between practice blocks and divide for each participant and by each individual’s pooled SD
Practice Difference Score = (Mean (Practice incompatible) – Mean (Practice compatible)) / Mean SD for Practice Trials (Step 1)
4. Compute the mean difference between the real IAT-blocks and divide by pooled SD
Real Difference Score = (Mean (Real incompatible) – Mean (Real compatible)) / Mean SD for Real Trials (Step 2)
5. Create D score by averaging the practice difference score (step 3) and the real difference score (step 4)

I've attached my program here so you can take a look. All the calculations look the same to me and all my lab members. I also checked that the right values/expressions were in each block and it all looks good to me, so we're really stumped as to why we're getting such different results. Any help is greatly appreciated.

Thank you so much!

Cassandra