Millisecond Forums

Well there are quite a few people who are emailing me saying they completed the task, they know all the details of the task, yet there is no data file for them there. If it was just one or two  I would say its a scam, but its actually a lot of them.

The attention check goes on to a next page that then does as you suggest already (e.g., you have failed to pay attention so you cannot take part in this task). No one has complained about this and there files still show up if the control quit or are kicked out.

Just to be clear, unless you've updated the script, the only choice is to click the invisible rectangle. You'll need to list "proceed2" as a response for that response to be recognized.

I'll look further at the logs to see if I can find more data to make sense of the feedback you're getting from respondents.

-Sean

Yes I fixed it but again it has nothing to do with the issue I am having here with the data not being uploaded.

I'm slogging through the servers raw hit logs, and it appears some of the participants are also running your "T2" study, which captures the MTurk ID in the link. With those participants, I can use the ip address to examine their hits with respect to the "Bundle" study to see what connection they made with the server. Would it be possible to share some/all of the MTurk IDs for respondents who say they completed the whole study but for whom no data can be found? It would also be helpful to know which date they ran the test.

If you'd rather not post them here, you can email them to me at seandr at millisecond dot com.

Thanks,
Sean

P.S. It appears that T2 is registered with Inquisit 4.0.6, and Bundled is registered with 4.0.7. You could save workers the trouble of downloading the Inquisit control twice by setting these to the same version.

-Sean

The T2 study is done now so no more people will be taking it. Didnt have any issues there, at least none were reported to me though that one wasn't run through my MTurk account and I think they just pay everyone regardless so that might be why there were no complaints.

 Here is a subset of workers which I am most confident actually did the study but whose data I cant seem to find. In terms of dates its from Nov1st to Today.

A1FTR8DJIKGD8H
A21A93NS7IJNZM
AQLO2RDFLX3DN
A34L4JED8UAIXC
A38Z4HGE16SX10
A19ED8FYO6CA36
A2JZQHB8N6LS7T

Ok, of those, only A34L4JED8UAIXC also ran T2. With that participant, Inquisit was for some reason unable to reach the server to verify the license. In that case, a message appears indicating the failure and giving the participant the option of running anyway even though data wouldn't be saved, which they seem to have done, since they did eventually arrive at the finish page. That would explain how they could have run the whole test without producing a data file.

I've identified 2 other cases where the license check similarly failed, although I can't map them back to the mturk IDs, unless you happen to know their subject id or ip address (not sure how you would). There may be other cases that I can't identify because they didn't reach the finish page.

I'm still investigating why the license check failed to reach the server. In the meantime, you could try setting the version back to 4.0.6 for this script, although that's just a shot in the dark, as I'm not aware of any recent changes to the code that could explain this.

P.S. As for the finish page timestamps missing from the logs, that appears to be a bug in the report. The hits are there in the underlying data tables. 


-Sean

Thanks for looking into this.

So basically something pops up and they just ignore it and click through blindly. I think that part is going to be hard to avoid but I can at least put a warning about it up to try to prevent it. Do you think if they try to install and run again it would work on the next pass? Is there alternatively anything I could add to the script that could check if this is the case then kill the session if it is?

Ah and yes a few reported that they got to the end but the finish page never popped up.

I don't know if trying again would work, but you could instruct participants to try, especially if they have access to another computer. 

I'm planning to update 4.0.7 to remove the option of running the study anyway given its potential to cause this type of problem. That way, if the script registration fails, the participant doesn't waste their time and your money by completing the study. I'm kind of surprised this hasn't come up before.

The earliest I'd publish that update would be tomorrow (it's 11pm Seattle time right now). 

I'm also trying to find a computer that reproduces this issue. Once we can reproduce it, we should be able to fix it.

Regards,
Sean