Secret Sauce Recipe

[genevoise]

I've always loved to play long puzzles and since the new scoring system was introduced I set a personal goal of attaining a 1000+ points average by the end of the month (currently struggling).
To that end I strive to get the maximum number of points from every puzzle. But for the life of me I haven't cracked the code. I have several times got a new 'best time' yet only received less than the maximum points, sometimes as much as 200 less!
So how do you get the maximum number of points? What is the secret sauce recipe?

[benjihall]

That is because the scoring isn't consistent across puzzles and only benefits large puzzles.

[admin]

So how do you get the maximum number of points? What is the secret sauce recipe?

Maximum points will very, very rarely be achieved for any given puzzle. Very fast solvers will generally get 75-90% of the maximum, and exceptionally fast solvers might get somewhere in the 90-96% range.

[sionnach57]

Genevoise poses the question: "What is the secret sauce?"

I have a partial answer, but you many not like it. My scores seemed to be improving gradually over the last
several months -- because practice really does help -- so I thought that aiming for an average score of 1,000
for the month of August might be a reasonable goal. So we had the same goal! Was it a reasonable one?

It was and it wasn't, for reasons I will explain below, probably in a post to follow this one.

First of all, it can be done. I think my current average score for August is somewhere around 1040, over
several hundred puzzles and 100% solving rate. So it's achievable. But you may decide it's not really worth
it. Because I am stubborn, and August is almost over, I will keep trying for scores over 1,000 for the rest of
the month. But not after that.

Why not? Well, let's start by stating the obvious. It is a mathematical impossibility to score over 1,000 for
puzzles with fewer than 20 clues. Let's remind ourselves of the MAXIMUM scores for the longer puzzles:

20 clues 1031
21 clues 1111
22 clues 1155
23 clues 1180
24 clues 1254
25 clues 1303
26 clues 1360

The stated intention is to avoid penalizing people who play longer puzzles (which generally -- necessarily --
take longer to solve). In fact, this distribution of *potentially available* scores affords long puzzle players
the chance to get higher scores overall than those who routinely play only shorter puzzles. Though, of
course, since there is not a monthly competition that recognizes high average scorers, people will track this
particular metric to satisfy some inner compulsion. There will be no gold medal for briang, a god among mere mortals.

Let's also agree that you are very unlikely to score over 1000 on 20- and 21-clue puzzles. As the admin has
said, nobody gets 100% of the possible score. (I had a flashback to every French class I have ever taken, of
handing in homework which I knew to be technically perfect, to be awarded a score of 19/20. Because, as
each successive teacher would remind me with that sweet Gallic shrug, "la perfection n'existe pas"). I
generally count on coming in at least 200 below the maximum, for my fastest performances, but often as not
closer to 300 below the maximum. Effectively, with my current solving ability, to have a decent chance of
breaking 1,000, I need to be choosing the really long puzzles: 24 to 26 clues.

But that's really not all. Here's the bad news. You are not automatically rewarded for having chosen a long
puzzle. You still have to *do well* on it, in some meaningful way. And that is judged, reasonably enough, by
some kind of grading on the curve, where your performance is evaluated relative to other people who have
tried that particular puzzle. The admin is reluctant to provide exact details of how this "grading on the curve"
is translated into points, and I don't blame him. (In graduate school, I had to teach statistics to potential
pre-med students, all of whom had what I came to regard as an obsessively unhealthy interest in the
grading algorithm, and an incredibly annoying willingness to engage in endless debate about the application
of that algorithm in their particular case, especially if they felt they were on the borderline between, say, a
B+ and an A-. So in later iterations of the course I learned that a certain vagueness about the exact grading
algorithm was desirable, because it held the more obsessive students at bay.)

That said, there are two potential issues that come to mind, each of which could affect the "fairness" of the assigned scores. The first is that, there is a small number of players on the site, who I shall refrain from naming, who consistently achieve solving times that seem frankly incredible. It is always a little disheartening to solve a long and challenging puzzle to find that player X has been there and set a record solving time that defies credibility. If such solution times played too big a role in the assignment of scores to everyone else, then that would be a concern. The second concern I have is about the distribution of scores. At the bottom of the page for each solved puzzle, there is a pretty little bell-curve, suggesting a nice well-behaved normal distribution of solving times. But all it takes is one look at the x-axis for this distribution to realize that the pretty bell curve is entirely bogus. The only way to get it to look like that is to engage in some kind of weird distortion of the scale for the solution times. Most common options for "grading on the curve" rely heavily on the assumption of a normal distribution, or at the very least, a symmetric distribution of solving times around their mean. This assumption is nowhere near to being true for any of the puzzles. The distribution in every case is highly asymmetric. (Sorry, I'm a statistician, I can't help myself).

*That said*, my general reaction to the assignment of scores is that there is an overall sense of coherence to how it works, and while it sometimes seems a little strict, it seems pretty fair to me. So this post should not be interpreted as a criticism of the scoring system, which in my time on the site has continued to evolve in what most people would consider the right direction. This is just my attempt to answer genevoise's original question about the "special sauce".

In a subsequent post, I will give actual data that gives a clearer idea of what might need to happen for a 24-, 25-, or 26-clue puzzle to have a chance of breaking 1,000. Spoiler: just matching the average solution time, or beating it by a slim margin, won't even get you close -- you'll be lucky to get a score of 800.

But I've probably rambled on enough for now

[jugl11]

Great post, Sionnach! I can relate to the anguish of teaching medical students. Many years ago, the hospital I worked for appointed me Lecturer in the Med School. Although I am not a physician, I had apparently achieved enough notoriety in my clinical subspecialty that they had me teach a course & do some Grand Rounds presentations. Lecturing to med students is a chore! They are trained to obsessively write down everything they hear so they can regurgitate it later. I finally got them to agree to put down their pens and LISTEN to me by promising to hand out a sheet with the points I wanted them to remember. About 30 years later (and in a much different setting), I was drafted to deliver a series of lectures on the limitations of statistics (example: Rule #1 when evaluating research -- find out who paid for the research). My audience was physicians, nurses, dentists, & psychologists. But these were experienced professionals, average age 50+, & had outgrown the obsessiveness of their student days.

[genevoise]

Recently I've stuck to playing 24, 25, or 26 word puzzles. I've found that choosing puzzles with the greatest difference between the average and fastest time increases my chances of getting a score closer to or greater than 1000.
Any other tips?

[sionnach57]

Can one consistently beat 1,000? Well, let's see.

A priori it seems the most promising place to start is with 26-letter puzzles. 1360 points available!!!
So *all you have to do* is not lose more than 360 points from the possible maximum score.
Easier said than done, as this little table shows. It represents my actual solving times and associated scores for a variety of different 26-letter puzzles. Importantly, the average time and record time are also provided for context. All solution times are in seconds. 'nr' stands for not recorded

Average time Record time My solution time Resultant score

1139 375 617 1065
1641 1153 738 1115
1752 565 621 1167

looking promising? Don't let's get too excited just yet.

1255 221 742 994
1043 nr 907 887
1315 nr 1303 820

from which I infer that *just matching* the average solution time does not get you many points at all. Depending on your point of view 820 might be a fine score for just beating the average solution time, but it does represent a full 540-point deduction from the theoretical maximum .

And if you don't manage even to beat the average solution time?
1255 537 1393 756

So, if I had to sum up the story so far, it's only in cases where I managed to solve *substantially* faster than the average that I had a chance of beating 1,000. But even when I managed to reduce the solving time to a quarter of the average AND shave almost 200 seconds off the previous record, my score of 1172 was still almost 200 below the theoretical maximum.

And here's the thing. I know that, realistically, my best solving times for a 26-letter puzzle are typically in the 500 to 800 second range. So if the average solving time for the puzzle is below a 1,000 seconds the chances that I can reduce that *substantially* are not great. If the average solving time is of the order of 600 to 700 seconds, I may be lucky to achieve it. So, if all I care about is getting a score over 1000, I will need to reject a whole bunch of 26-letter puzzles. Because my best bet is always a 26-letter puzzle with a high average solving time.

But this way madness lies. Because now an obsession with high scoring leads one to reject all but a very select number of puzzles. Which is why I am abandoning the entire 'score over 1000'mentality and just gonna take the longer puzzles as they come.

So, genevoise, in my opinion/experience, one can consistently beat 1000. But it's not worth it, because you end up limiting the puzzles you choose to do significantly.

I have tabulated my experience with 25-letter and 24-letter puzzles as well, but the general message is the same. Here are data for some 25-letter puzzles. Theoretical maximum score = 1303 points.

average time record time my time my score
1284 388 475 1110
1273 467 515 1092
2178 741 547 1172

but if I choose 25-letter puzzles with average solving times that are not so high, then

665 nr 634 806
705 464 564 886
777 nr 915 689

I know that it usually takes me anywhere from 500 to 800 seconds to solve a 25-letter puzzle. So, if I had to make a cutoff for 25-letter puzzles that give me a chance of scoring over 1000, unless the average solving time is over 1100 seconds I probably won't manage it. For instance

1069 195 629 996

And that's quite enough of this for today!
On edit: all my efforts at spacing those table numbers just didn't show up. So you need to bear in mind that the four numbers on any line are left to right: average solve time - record time - my time - my score

[duhmel]

I have analyzed scoring for over 1000 puzzles withe the following conclusion

points awarded = (1-.4*x)*maximum points

as long as the 'points awarded' is greater than 100 where x=(time to solve/average time). Note that the value of x that give 100 points changes for the different number of clues. If the equation gives a value less than 100, an 'arbitrary' number between 100 and 132 is the awarded points.

This gives an interesting outcome. I did a 21 clue puzzle at 2.25 times the average and got 109 points per the equation. However when I solved a puzzle with the same number of clues in a longer time (2.34 x average) I got 127 points where the equation would predict 72 points. Thus, taking longer to solve turned into a higher score. I have data that shows that this happens for every number of clues. How the administrator assigns scores for the equation would give less than 100 points remains a 'state secret' but I know it is not dependent on time to solve, percent solved, or solution time compared to the average. Actually it looks like a random number - the administrator wanted to give everybody at least 100 points for a solution, but instead of just giving 100, he has randomized the award to between 100 and 132. I'm just guessing on this but that's what the data shows.