Your calibration logic is fine. You can see this by displaying e.g. a rectangle sized according to expressions.letters on-screen and measuring its dimensions (it's the desired 12mm on my system and should be on yours as well):
<<<<this script should always shows letter 12mm height, regardless of participants computer display properties>>>
<values calibration>
/ cal_mm = 0mm
/ cal_px = 500px
/ letters = 12mm
</values>
<expressions>
/ mmtopx = values.cal_mm/values.cal_px
/ letters = 1px*round((values.letters/expressions.mmtopx))
</expressions>
<openended calibrate>
/ stimulusframes = [1= line]
/ position = (50%, 80%)
/ validresponse = (anyresponse)
/ monkeyresponse = ("195")
/ mask = positiveinteger
/ ontrialend = [values.cal_mm=openended.calibrate.response]
/ ontrialbegin = [openended.calibrate.buttonlabel= "Save results (in mm)"]
</openended>
<shape line>
/ shape = rectangle
/ color = (black)
/ erase = (250,250,250)
/ size = (values.cal_px,10px)
/ position = (50%, 50%)
/ halign = center
</shape>
<text Letters>
/ items = ("F","P","Q","J","H","K","T","S","N","R","Y","L")
/ fontstyle = ("Verdana", expressions.letters, true)
/ erase = true(silver)
</text>
<trial trial_A>
/stimulustimes = [0=letters, myshape]
/ validresponse = (57)
</trial>
<shape myshape>
/ shape = rectangle
/ size = (expressions.letters, expressions.letters)
/ position = (50%, 25%)
/ color = red
</shape>
<block calibrate>
/trials = [1=calibrate]
</block>
<block block_trial_A>
/trials = [1-10=trial_A]
</block>
<expt >
/blocks = [1=calibrate;2=block_trial_A]
</expt>
The problem with your approach is that fonts just don't work that way. To the best of my knowledge, there is no such thing as a "fixed-height" font, i.e. letters *will* necessarily vary in height due to their natural shape. For example, the letter Q requires more vertical space than the letter P (the red square is 12mm x 12mm):
If you measure the height of the Q, you'll find it to be relatively close to 12mm (also note that there's typically some amount of empty space around letters in a font).
If you really want all your letters to have identical, fixed height you will have to resort to a different approach. E.g. don't use fonts, but prepare images displaying the letters, with each letter sized according to your needs.