Millisecond Forums

Ooh, the mean-spirited lawyer I keep locked up in my basement (just in case) will a have a field day with this comment!!! :-)

~Dave

What? Quite the contrary!

You brought this whole long thread into focus (at least for me), and my ideas for new counter syntax were stolen straight from your explanation how latin square selection should work. So, thank you!

P.S. Looking forward to seeing how you solved this.

-Sean

Nice!

Of course you are absolutely right, Sean. And obviously I didn't think things through properly, thus causing confusion and dismay with my nonsensical post. Clearly, we're neither dealing with a bug nor a design issue -- the only thing that's wrong is my flawed logic (yes, my existence is undeniable proof of Unintelligent Design!). Sincere apologies to everyone for that.

That said, great suggestions presented by you and Malte to possibly make factorial selection easier to implement in a future release!

However, to make up for this utter f*ck-up on my part, and because Latin squares / factorials are a useful thing to have around, I've attached a script that should indeed produce a random, Latin square order of arbitrary size using the method described by Byers (1991).

Thanks,

~Dave

Well, for a start, I'd suggest

<counter COUNTER_latin>
/ items = (1, 2, 3, 4)
/ items = (2, 3, 4, 1)
/ items = (3, 4, 1, 2)
/ items = (4, 1, 2, 3)
</counter>

which would be a tad less error-prone and, to me, appears more readable then the matrix based version. Plus, with the matrix based version, there'll definitely be people to request matrices of higher dimension than 2.

Further, I could imagine something like

<counter COUNTER_latin>

/ items = sequence(1, 2, 3, 4)

/ items = sequence(1, 2, 3)

/ items = sequence(1, 2, 3, 4, 5)

/ items = replace(1, 2)
/ itemselect = replace|noreplace|permute|...

</counter>

For each draw, the counter would select a new "/ items" list, commit the current draw on this list and return the value. If "permute" is selected as the item select mode, Inquisit would iterate through the "/ items" lists and select a new element from the first list on every draw, from the second list only every 4th draw, from the third list every 4*3th draw, from the fourth list every 4*3*5th draw.

Just a thought,
Malte

Thanks for the test cases, Dave, they definitely clarify things. I've run through a few of them, and I think the problem is more of a design issue than a bug per se.

The issue is that counters select values on the fly, which might lead to situations where it is impossible for a counter to satisfy both the NOREPLACE and NOT constraints. Consider the following example:

       a    b   c    d
t1   2    4    1    3
t2   3    1    2    4
t3   1    2    4    2
t4   4    3    3    1

Things go along nicely until it is time for d to select a value on t3. D has two items, 1 and 2, left in the pool. However, both of those have already been selected on t3 by a and b, so what should it do? Currently, the NOREPLACE rule is given precedent, so it selects from the remaining pool. However, even if NOT were given precedent and it selected 3, you still don't get a nice neat factorial because d already selected 3 on t1.

I'm fairly certain there is no way to always satisfy both NOT and NOREPLACE constraints with "on the fly" selection. Depending on the size of the matrix, there will be some probability that a counter will be painted into a corner. Perhaps we could preselect various matrices until we find one that works, but I'm not convinced that will work either since we have no way of knowing exactly a) how many trials there will be, and b) on which of the trials a given counter will be used, because trials and selections may be inserted or skipped based on data that isn't known at the start of an experiment. 

I think the solution must allow you to define selection matrices. This could take the form of a multi-dimensional counter such as the following:

<counter all>
/ items = ( (1,2,3,4), (1 3 2 4), (4 2 1 3), (2 4 3 1) )
/ select = noreplace
</counter>

where subitems can be referenced with an index:

<text one>
/ items = ("a" "b" "c" "d")
/ select = counter.all.selectedvalue.1
</text>

<text two>
/ items = ("a" "b" "c" "d")
/ select = counter.all.selectedvalue.2
</text>

Or it could be accomplished by a new /select option:

<counter group>
/ items = (1,2,3,4)
/ select = noreplace
[...]
</counter>

<counter subgroup>
/ items = (1,2,3,4)
/ select = factorial(group)
</counter>

<text one>
/ items = ("a" "b" "c" "d")
/ select = counter.subgroup.selectedvalue.1
</text>

<text two>
/ items = ("a" "b" "c" "d")
/ select = counter.subgroup.selectedvalue.2
</text>

I think the first option is more flexible, explicit, and intelligible, although things get a little cumbersome as the matrix grows (a 20x20 matrix would involve a lot of typing). Possibly there are some better approaches than either of these - ideas are certainly welcome.

As a historical note, support for factorial selection has been on the feature list since it was first requested way way back in version 1.

-Sean

Some more details about the code attached to my previous post to help in troubleshooting the issue. The code in the examples is essentially supposed to generate a random, n-th order Latin square (over the course of n trials, although this is not essential). Here's an example and outline of the underlying reasoning for the case of n=4 elements:

<counter a>
/ items = (1,2,3,4)
/ select = noreplace
[...]
</counter>

<counter b>
/ items = (1,2,3,4)
/ select = noreplace
/ not = (a)
[...]
</counter>

<counter c>
/ items = (1,2,3,4)
/ select = noreplace
/ not = (a, b)
[...]
</counter>

<counter d>
/ items = (1,2,3,4)
/ select = noreplace
/ not = (a, b, c)
[...]
</counter>

Over the course of 4 trials, this is what's supposed to happen:

(1) Trial 1

• A random value (e.g. 3) is drawn from <counter a>, leaving three values in its selection pool for the subsequent trial (1,2,4).


• A random value is drawn from <counter b>, whereas it shall not be the same value previously drawn from <counter a>. The value drawn might be 1, also leaving three values in the counter's selection pool for the subsequent trial (2,3,4).


• A random value is drawn from <counter c>, whereas it shall not be identical to the value drawn from either <counter a> and <counter b>. Thus the value determined might be 4, with again three values remaining in <counter c>'s selection pool (1,2,3).


• A value is drawn from <counter d>. Given the imposed selection constraints, this value must be 2, with values (1,3,4) remaining in <counter d>'s selection pool.

(2) Trial 2

• A value is selected from <counter a>. Since 3 has already been selected on the previous trial, this value might be 1, with (2,4) remaining in the selection pool for the third trial.


• A random value is drawn from <counter b>. 1 has been selected previously and the value shall not be identical to <counter a>'s current selection (1). So let's say the returned value from <counter b> is 3, leaving its selection pool at (2,4).


• <counter c> now has little options left: 4 has been selected on the previous trial, and due to the constraints 1 (<counter a>) and 3 (<counter b>) are also out of the picture. The returned value thus must be 2, with (1,3) remaining for subsequent trials.


• Again, the selection for <counter d> is fully determined at this point: 2 has been selected previously, and due to selection constraints 1,2, and 3 aren't available either. The value drawn can only be 4, leaving <counter d>'s selection pool at (1,3).

You get the idea... Summing up, over the course of 4 trials, here's the desired result (along the lines of the above example) in cross-tabulated form, with trial numbers as rows and counter selections as columns:

counter
a b c d

t1  3 1 4 2
t2  1 3 2 4
t3  4 2 1 3
t4  2 4 3 1

As we can see, we (should) have ended up with a nice, randomly generated Latin square. Now, if only this would (still) work. I've found the outlined counter-logic to function properly for smaller values of n (e.g. up to 4) even in 3.0.5.0 and the release 3.0.4.0 build, but according to my tests it fails miserably for larger values in both. As Malte confirmed (thanks!) the issue is not present in 3.0.3.2. However I am *very* sure that multiple '/ not' statements vs a single '/ not = (a,b,c,d,...)' did make a difference in 3.0.3.2 -- note that this might hold true for larger values of n only.

Thanks for enduring this tedious-to-read post!

~Dave

Yep, same here. The reason for my disbelieving response to Dave's posting was that I had tested both variants and was pretty sure they yielded the same undesired results. However, I tend to believe the things Dave states, so I tried both options on one of our lab computers which indeed delivered. But: my notebook - Inquisit 3.0.5.0, lab computer - 3.0.3.2. So Dave's correct, version 3.0.3.2 respects multiple "/ not" statements but not a single "/ not" statement, whereas version 3.0.5.0 disregards both cases.

Malte.

Okay, here we go: I had initially discovered the inconsistency between '/ not = (a, b, c, d)' and '/ not = a, / not = b, / not = c, / not  = d' when writing and testing the SOPT script. The Inquisit versions used at the time were 3.0.3.2 and various beta builds of 3.0.4.0. To recap the then-finding once again: Multiple '/ not ' attributes worked fine while the same logic implemented via a single '/ not = (a, b, c, d)' would lead to faulty results (double selections, etc.).

I've now revisited this scenario with version 3.0.5.0 and found that indeed -- as Sean noted -- both options behave equally in this version. However they behave equally *wrong*, i.e. there'll now be undesired, faulty selections in both cases. Bummer. A set of example scripts showcasing the problem for both not-options is attached to this message.

~Dave

Ok, I think the only bug is in the documentation, because I've checked both cases and they be have identically as we all would expect. Are either of you seeing different behaviors?

-Sean