Cheaters

[SGreenNYC]

There are too many people here who have ridiculously fast times. They are obviously completing the puzzle under one name and then quickly filling it in under another name. Even filling in the blank spaces in the time some of them post seems amazing. They certainly can’t actually be solving in the times they are recording. These people suck.

[admin]

Let's keep the discussion civil, everyone.

I've deleted all posts except the first one in this thread as things have gotten a bit testy and it is NOT appropriate to use this forum to "name and shame" users you think are cheating. If you feel a user is cheating, you can email me at puzzlebaron@gmail.com and I will investigate it.

Regarding potentially unrealistic record times, I've done a deep dive on the data and here's my conclusions.

Two record times (2 seconds) were either absolutely solved programmatically, or they were the result of some sort of glitch in the system. I've deleted both of these from the database.

Concerning the rest, I've run some stats. Out of approximately 50,000 puzzles:

- 140 have record times under 30 seconds (all are from our original 2013 library of puzzles, all have 14 or fewer clues but most have <10)
- 639 have record times under 40 seconds (99% are from our original 2013 library of puzzles, all have 17 or fewer clues but most have <12)
- 1562 have record times under 50 seconds (98% are from our original 2013 library of puzzles, all have 17 or fewer clues but most have <13)

Our earliest (2013) puzzles tended to be much shorter and contained a much smaller pool of clue answers, so this data makes sense to me as we have several "power users" who have been here for many, many years and solved tens of thousands of puzzles. Over time they are going to recognize certain clues and be able to pop the answers in with lightning speed. If a critical mass of clues in any given puzzle (generally, around 30-40%) can be instantly recognized, the rest of the puzzle becomes fairly easy to solve, and if the user has (a) excellent memory, (b) excellent pattern recognition skills and (c) excellent typing skills, there's no reason not to believe they can legitimately solve an 8-clue puzzle in 25 seconds. Especially if they've solved tens of thousands puzzles over several years here and have likely gotten several repeat puzzles and many repeat clues over that time.

On our cryptogram site we have similar complaints from time to time, when people see record times of 4, 5 or 6 seconds on a puzzle. But I can guarantee you that such incredible solve times are legitimately possible - I've seen it with my own eyes. Never underestimate what a person can achieve given enough practice and enough innate skill at pattern recognition. Just because you can't do it doesn't mean there aren't a couple of super talented people out there who can!

My conclusion: the vast majority of players who have sub-40 record times (35 different players) appear to have set those times legitimately, likely through a combination of sheer pattern recognition, excellent memory from having solved many thousands of previous puzzles and lightning-fast typing ability. Only one player appears to have possibly employed "shady" tactics in at least a couple of puzzles, and all those records have now been reset.

In the end, cheating is always a concern. We do have measures in place to detect programmatic solves, but I'm sure they're not foolproof - no system is. I'll keep doing what I can to keep the site as fair as possible.

I may also end up discarding (or reformulating, with our new much largest dataset of clues/answers) the original batch of 2013 puzzles. Will think some more on that.

[briang]

I'm happy to keep things civil and will refrain from naming individuals henceforth, but I have to beg to differ on your takeaways for a few related reasons:

1) While I certainly believe that individual players can on occasion achieve extremely fast times, there are several people who always get extremely fast times, except when new puzzles are released, and suddenly they get several much slower times. This to me is strong evidence that they are solving using an automated tool that is programmed to submit based on a fraction of the current best score, as I described in my earlier now-deleted post.

2) There is a small but definite number of puzzles that have typos and errors in them. These are pointed out here on a fairly regular basis. The same group of users described above often have an exceptionally low score on those as well! This simply doesn't pass the sniff test for someone laboriously changing letters one by one in order to find the one typo in a puzzle that is preventing a correct solve. A cheat using the exploit that I detailed in the deleted post would not have that problem, since it wouldn't even consider what the clues actually are, and would just look at the solve hash that is visible in plain text on the page source.

3) There are users whose score cards indicate that they have solved not only every single puzzle on the site, but sometimes somehow more puzzles than are even present on the site! While it is conceivable that someone shut-in, retiree, or similar could have put in the thousands of hours it would take to actually do this, again we have the problem of them somehow managing to solve every puzzle (even ones with errors in them that are are rendered borderline unsolvable, so quickly!) Look at Jeopardy! Even Ken Jennings sometimes blanked on clues or didn't know answers. Somehow these folks seem to know every single answer, even ones that aren't repeats, immediately, never getting stuck. And this doesn't even taking into account typing errors, time spent scrolling between the clues and puzzle, etc. Color me extremely skeptical.

4) And on a more subjective note: while yes, pattern recognition skills and typing skills can be improved through practice, the levels you are talking about are simply not attainable consistently by human players. I have a lot of experience in the world of crosswords and am considered a champion-class player. One of my close friends whose typical scores are almost identical to mine placed second in this year's American Crossword Puzzle Tournament, and I myself was a fixture on the NYTimes crossword leaderboard for some time years ago before cheating became rampant there. I truly believe I have a firm basis to pass judgment.

5) I'm particularly convinced of these points because I only play the very long (24-26 clue) puzzles. While it is conceivable that on the shorter ones, a player might know every single clue and be able to run through the entire puzzle in just a few seconds, this gets increasingly less likely the more clues there are. A longer puzzle also drastically increases the likelihood of inadvertent typing errors, the amount of scrolling needed (even on a large screen), and other factors that increase times. Yet I've seen sub-1 minute times even on these, and the people who have gotten them have without fail been among the group of names I had mentioned before.

I get that on a certain level it feels good to think that some people are simply capable of seemingly impossible, superhuman feats. Marveling feels good. In the cycling world, people felt the same way about Lance Armstrong, to the point of positively trashing Floyd Landis, who not only was Lance's teammate, but knew all the cheats himself, for pointing out the incredibly obvious evidence that Lance was cheating, too. We saw how that turned out.

[admin]

You're of course welcome to your opinion, and heck, you could even be right. But I'm not going to penalize players who may be cheating unless there is definitive proof. When there is definitive proof, I take action (and regularly do so). What you've laid out is not proof, it is a gut feeling based on several flawed assumptions.

One example: the data you've laid out in #5 is just incorrect. There isn't a single puzzle on the site right now with 25 or more clues that has a record time under two minutes. There is ONE puzzle with 24 clues that has a record time just shy of two minutes - all the rest are above two minutes and the vast majority a great deal above that.

Another example: Re: your javascript comments, that is only step 1 in the verification process. The javascript only does a rudimentary check to see if the solution is correct in order to advance to the next stage. More extensive server-side checks are done after that. Faking the javascript hash without also submitting the correct and complete puzzle solution will only get you to an error page.

[DogMa]

I have been solving acrostics here for more than 10 years, although my total number of solves is not terribly impressive compared with some. I have never seen a record time of less than 60 seconds, much less 20, which was the number mentioned in a couple of now-deleted comments.

[duhmel]

I have been solving acrostics here for more than 10 years, although my total number of solves is not terribly impressive compared with some. I have never seen a record time of less than 60 seconds, much less 20, which was the number mentioned in a couple of now-deleted comments.

This comment is just ridiculous. I just started a 14 clue puzzle that had a 58 second best time. My guess is that 75% of the puzzles with less than 12 clues have times below 60 seconds.

[DogMa]

I'm so sorry I made a ridiculous comment. Please forgive me. I do usually skip the puzzles that have fewer than 20 clues.

[admin]

My guess is that 75% of the puzzles with less than 12 clues have times below 60 seconds.

26.3% of all puzzles with fewer than 12 clues have record times below 60 seconds.

[admin]

Digging into the data a bit more, it looks like we can reduce the number of sub-30 records by 85%, and the number of sub-60 records by about 45% if we ignore all "replay" statistics (i.e. instances where a user solves a puzzle they've already solved before at some time in the past, and in doing so sets a new record time). The vast number of super-fast solves do appear to be made by power players who have solved so many puzzles that they've started to get repeats, especially back in the days when our puzzle library was only about 20% the size it is today.

I'm working on code now that will enforce this restriction (i.e. only the very first time any user solves a given puzzle will be considered eligible for statistical purposes - they will still get points, however, any time they solve a puzzle) and will post the results of reformulated record times once that's done. The only issue is that in order to enforce this restriction we'd have to ignore all solve statistics from non-logged-in users, which tends to be about 80% of all solves. So many of our newer puzzles will likely get knocked back into not having any median solve time data as they don't yet have a critical mass of purely logged-in user solve times.

Will post a thread here once this new set-up is ready for primetime, along with stats showing what the updated record time distribution looks like.

[duhmel]

The only issue is that in order to enforce this restriction we'd have to ignore all solve statistics from non-logged-in users, which tends to be about 80% of all solves. So many of our newer puzzles will likely get knocked back into not having any median solve time data as they don't yet have a critical mass of purely logged-in user solve times.

Will post a thread here once this new set-up is ready for primetime, along with stats showing what the updated record time distribution looks like.

My personal opinion is that you would be solving a problem that doesn't exist for the vast, vast, vast majority of users and would penalize the rest of us as a result. I'd bet very few people care about 'record' times - knowing they are unreasonable and unachievable. What is important to me is the average time. I've done nearly 4000 puzzles and I measure my performance against the 'average' time. I measure the percentage of times that I beat the 'average' and my average score. If you make a change that eliminates many 'average' scores, people who are essentially gauging themselves against the 'average' player will lose this measure of their success. All this to satisfy the very, very few players who are concerned with the ultra, ultra fast scores that are being posted.

[cocoadog]

FWIW I agree with duhmel in that I'm not too concerned about bogus fast times. It's not like there's money on the line.

Online chess has a cheating problem, and the top sites throw a lot of money, time, and technology at it, with a whole process for reporting suspected cheaters, and the use of AI and statistics to detect them, and a formal process for banning and unbanning, and so forth. I don't know if cheating on these acrostics rises to that level of need for policing. But I appreciate any thoughtful attempt to address the problem, and as long as I can keep enjoying the puzzles I will probably be fine with whatever you end up doing.

[DogMa]

One reason to be concerned about the undeserved fast times is that cheaters are displacing noncheaters in the High Score and Fastest Solver Challenges. From 2013 through 2016, I placed fairly frequently, never at the very top, but in the top 10 or 20. Now, despite scoring above average the vast majority of the time, I never place. Of course, there are more competitors now, and there appear to be a lot of people who spend hours every day solving these puzzles. If they push me out, then that's fair. But if there are many cheaters, that has to skew—unfairly—these high scores and fast times.

[duhmel]

One reason to be concerned about the undeserved fast times is that cheaters are displacing noncheaters in the High Score and Fastest Solver Challenges. .

I repeat, the vast, vast, vast number of puzzle solvers don't care about what the 'cheaters' do. I just hope the Administrator doesn't take actions that will hurt the joy of the the majority of puzzlers by reducing the number of puzzles that show 'average solution time'. Many more people would be affected by this action than improvements against 'cheaters'.

[OldManInNNV]

I've never noticed any times that seem inordinately low in Acrostics. When I see a low score, I usually attribute it to someone getting a repeated puzzle and remembering the clues or quote. It's really easy to type stuff fast that way.

The only place where I think cheating is going on is the Cryptogram section. I've seen record times of 2 seconds on many occasion. Even if it's a quote that you recognize, there's no way your brain will recognize the quote and move your fingers fast enough to type it error-free in 2 seconds.

But the larger question is: does it matter? There's no money at stake. There's not even bragging rights at stake, since there's nobody to brag to. I play for my own enjoyment, and my opponent is me.

Just my two cents, and worth about as much.

[Monty51]

Why is it necessary to even "grade" how fast a person solves a puzzle? For myself, I'm here just for the fun of solving the puzzles and don't care how others do. I don't feel this is a competition...or shouldn't be. Take away the notion of competition and there won't be these dramas I'm seeing here.