Statistics

[Steve-O]

How does anything work here? How do points get calculated? I've scored more points being way over the average than I have hitting the average. Seems points are almost random. I cannot find any instruction on how things work. Don't get me wrong. I will never set a record. Speed, monthly points, anything. I just like to know how things work - especially when they don't seem to.

[jugl11]

Basically, you get 800 pts. for solving a puzzle in 150 seconds or less (regardless of the individual puzzle's difficulty). You get 799 points for a score of 151 seconds, 798 for 152 seconds, & so forth. Down to a minimum of 10 points for a solve, no matter how long it takes you. I personally don't agree with this scoring system but I play by the rules for the enjoyment of the game.

[compuspud]

What is the bonus for solving more puzzles than you've actually played?

[redpicker]

I collected stats on a couple of months of puzzles and discovered some interesting things about the scoring. First, it's a lot more complicated than what jugl11 states. He starts off correctly. That is, less than 150 seconds is 800 points. Between 150 and some greater number (820?) if you have used no hints, your score = 950 - Time In Seconds. If your time is over this number (perhaps 820), your score seems to be a random number between 130 and 100.

I have two puzzles that I got a score of 130, both finished in 820 seconds. It fits the formula.
I have one puzzled that I got a score of 131, finished in 950 seconds. It is the only one that does not follow this rule. It is possible I fat-fingered the score. Perhaps it was supposed be 121? I don't know, but there is no way to check it now.

I have a puzzle that I got a score of 100, and finished in 1004 seconds, which is the lowest score I have recorded for which I did not use any hints. That puzzle had an average time of 1244 seconds, so I beat the average by 4 minutes. In contrast, another puzzle gave me a score 123 points that I finished in 3898 seconds (average time was 961.7 for that one). That's why I say with times over 820 seconds, it seems your score is just some random number between 100 and 130.

Now, if you use any hints, your score drops quickly to 10. It is not instantaneous, though. One game I remember, I used a hint, which finished the puzzle so I hit SUBMIT as soon as the hints came up and my score 267 with a time of 683 seconds (which included and extra 180 seconds for using a hint). This is equal to 950-683. This tells me that it is possible to get more than 10 points if you use a hint, but you apparently have to be quick about it. I also have a game where I received 12 points when using hints. I'm thinking that using a hint merely accelerates the rate at which your score drops. The minimum score when using a hint is 10 points.

[PatienceCrabstick]

I don’t mean to contradict anyone here, but I do believe you’re forgetting one factor: GIGO. Can the data available from this site be trusted? There are frequently glitches — for example, posting dates/times for all puzzle comments have been reset to the single afternoon of March 29. Also for example, when the site’s servers go temporarily offline (I think all regular players have experienced this) scores, etc. for games then in play are negatively affected. Lastly, as with the discussion on “suspected anomalies in record times,” there’s no real way to arrive at any objective answer as there’s no transparency here — not that we should necessarily expect any, but still the site’s admin(s) are conspicuous in their absence.

[redpicker]

You may be onto something, PatienceCrabstick. I don't look at is as the data available can't be trusted, but, rather, that the data is not what we believe it is.

My response is rather long, so I will break it up into several posts. First, I want to address your thought that "the site's admin(s) are conspicuous in their absence." Frankly, I don't believe there is an identifiable administrator to this site. By that, I mean that I doubt there is a single person or team who is responsible for all aspects of its operation. Sure, there is someone behind the controls, but I suspect they are third-party IT professionals hired by a consulting firm that is hired by the owner of the site. Their job is to keep the board working. They (probably) do not design the puzzles, verify the accuracy of the clues/quotes, or respond to questions about how the site works because that is not what they are hired to do. They (apparently) do fix problematic clues (I saw a comment a day or two ago saying "A rubbish heap would be found in a dump, not a junkyard. Usually, a junkyard refers to a place were wrecked cars are dismantled for parts." The clue for the puzzle I solved was "Garbage dump sight"), but I doubt these guys do anything about scoring, player rankings, or other statistics. The site owners, who could have their IT group address such issues, probably don't want to. They bought a site that had a certain number of visitors, so it brings in so much income. Making changes to how it works could drive users away. They are going to only fix what is clearly broken.

The observation that puzzle comments are now all dated March 29, 2020 is interesting. I had found it odd that the comments on the puzzles I had finished recently were always recent comments, but had not noticed that they all had the same date. Obviously, a glitch.

I do want to finish this by saying that I do believe a significant change was made when the site was migrated to PuzzleBaron. Prior to migration, I believe it was possible to get the site to call a specific puzzle by ID, so a person could solve a puzzle as PlayerA, then log in as PlayerB and solve the same puzzle in record time. The site lost a few high-scoring solvers at that time who have not returned (or if they have, they use a new login). I do believe there were some players who were exploiting this bug and I am thankful the admin was able to correct it. I realize they can't delete suspicious records since they have no proof, but putting an end to the practice was a plus, in my book.

More on scoring stats to come...

[redpicker]

As I mentioned in a previous post, I have collected the scoring stats for a month or so of puzzles. As I finished each puzzle, I would copy my score, the scores given at the bottom of the histogram figure on the "Congratulations!" page. There are 6 numbers on that figure. I call them Minimum, Fast, Average, Slow, and Maximum, in addition to "My Time" which is indicated by a Star on the histogram. I also record my score, the success rate, the total number of clues, the total number of letters, and the number of hints I used (if any).

Now, any analysis of a sample has to consider the randomness of the sample. I consider my data set to be random based on the idea that the puzzles the site assigns you when you hit the "PLAY" button are random. I finish every puzzle I start (or, at least, I try to, 99.7% over 3.5 years and 4000 puzzles), so my data should be a random sample. Lately, I have only been collecting data that I find "interesting", so the last ten or so entries are not completely random. By "interesting", I mean something like a score greater than 10 when having used a hint, a time that equals the "Maximum" time on the figure, or some other oddity. These "interesting" entries are less than 10, of nearly 200, or less than 5%, so I assume their effect on my analysis is minimal.

One of the first issues I saw was what I was calling "Minimum" was always equal to the fastest time, and was always an integer. However, the "Maximum" was not always an integer, but was always an integer, or ended in ".5". Clearly, what I considered "Maximum" was not the maximum, as the time is only recorded to nearest second. On the few puzzles where my time equalled the "Maximum", this time was always an integer, but overall, the "Maximum" time ended in ".5 " about 68% of the time. Now, if the "Maximum" was the average of the two highest times, you would expect the value to end in ".5" 50% of the time. Furthermore, three of these 200 data sets, I my time equalled the "Maximum", which was always an integer. So, it would seem that the "Minimum" is the fastest time, but the "Maximum" seems to be an average, at least some of the time. For the case where you solve the puzzle with the slowest time, your time is reported for the "Maximum". I've never had a fastest time, so I don't know what happens in that case.

More to come...

[redpicker]

OK. I know I am rambling and, perhaps, hard to follow, but I hope this post will bring me back on point.

After noticing that the "Maximum" times were either all integers (which is what I would expect) or ended in ".5", I noticed the "Average" values were either integers, ended in ".3" or in ".7". This I found to be rather odd, and could not come up with a reason. I would have expected, given enough samples, all digits would be represented in the average. Furthermore, for every instance where the "Average" was an integer, the "Maximum" was also an integer. Since I had all of these values in a spreadsheet, it was pretty easy to determine that the "Average" value was always equal to the "Maximum" value divided by 5. This is true for all values, except for those when my time was equal to the "Maximum".

Now, this is where it all hit me. While these values may have a basis in reality, the meaning of "Average" and "Maximum" are certainly not what I would have assumed them to mean. It would seem that "Maximum" represents, at least some of the time, the average of the two slowest solving times, and that "Average" is defined as being one fifth of "Maximum". That is why I said it's not that "the data available can't be trusted, but, rather, that the data is not what we believe it is". Perhaps this is why they don't base the scoring on whether or not you are better than the "Average" time. The "Average" time is not the mathematical average of all the times of those who successfully solved the puzzle, but a calculation based on the "Maximum", which is not simply the slowest time.

Now, all of this does not diminish the challenge I get from solving the puzzles, nor will it make me less likely to improve my skill in solving these puzzles. It does, however, suggest to me why the scoring seems, perhaps, capricious.

[sionnach57]

I have always assumed that the reported "average" is more likely to be something like the median of the distribution of scores, rather than just the regular arithmetic average (sum of all scores/number of scores) as the latter will be unduly influenced by extreme scores, in particular very long solve times. The pretty bell-shaped curve that is always shown also seems likely to be entirely fictional, as I would guess the actual distribution is more complicated with a heavy right tail, because people who use hints, get interrupted by their cat, use the bathroom etc etc etc will all contribute larger scores. Actual symmetry of the distribution about the mean is impossible anyway, because scores are bounded at the lower end, but not at the upper end.
So I take all the summary numbers with a large grain of salt. If I were less human, I would pay no attention whatsoever to them, but of course one is always going to take a peek. To me, the most annoying feature is that one can beat the "average", sometimes by quite a bit (in my world, that would be anything over 150 seconds, say), yet not be rewarded in any way if the puzzle is long and complicated, because once your solve time gets over the magic 820 seconds or so, you won't get over 130 points.
By the way, ever since reading redpicker's May 1st post, with his genius exposition of the approximate scoring algorithm, I have mentally checked his proposed algorithm for each puzzle I have completed, and what he suggests has checked out every time. Major thanks for that piece of detective work.
I'm not even going to go near the whole issue of all the magically low solving times one sees, except to say -- no, not gonna go there!