Difficulty Ratings

[admin2]

Hi folks -

We've implemented a new difficulty rating system that should be quite a bit more accurate, based on real-world solving statistics. This means you will now start seeing more short puzzles that are ranked "Easy" and long puzzles that are ranked as "Difficult". The new formulations are based off median solve times (by quote length), success rates, and user feedback.

Thanks!

[abra]

Wow!! Stephen, you've heard us, and answered, This is great news.

[imported_Lurker]

Thank you! Although, I'm going to miss all those points on short "difficult" quotes.

[marnita]

I agree, Lurker. Like many improvements, this one will take some getting used to.

[gryhnd51]

Now that we've had many quotes removed because some folks find them offensive, I've been paying attention to every single quote I solve and the ratings they've been given. I have to say that I've been truly amazed at the quotes deemed "offensive", especially since we are all given the opportunity to omit "possibly offensive quotes" in our settings. Therefore, I've taken every chance to negate the naysayers. I hope that the rest of you will take the small amount of time it takes to check them.
Also, the new scoring seems more arbitrary than rational...and based on a small percentage of folks who will always complain, no matter what. When I first started playing, those very long quotes were my "bread and butter" and the stepping stones to finally finding them more difficult and the shorter ones easier! I hope you'll all forgive me for saying that it feels a bit like "dumbing down" the site.
I know that Stephen is trying to be responsive to input, and I applaud him for that. I guess I'd just like to see a much larger sampling size before changes are made.
Ah well, I'm sure I'll get used to the changes and come to appreciate them.

[abra]

I thought it would be good to have the quotes rated according to the statistics. I know a lot of commenters had voiced that opinion as well.

But, so far I don't really care for the new system. Maybe, I just can't be pleased.

[LLapp]

Let's give it some time

I'm generally finding that the new difficulty ratings are more realistic. Yes, I still have trouble with some "Easy" ones (usually before coffee), and I still breeze through a few "Difficult" ones (usually after coffee), but for the most part the ratings are jibing with how hard the quote was.

Here is a great example of the new ratings being correct. This is a long quote I just solved that used to be ranked as "Easy" and that was upgraded last week to "Difficult." It was brutal, even though it was long, and the stats reflect this, and now so does the difficulty rating:

"Poker is to cards and games what jazz is to music. It's this great American thing, born and bred here. We dig it because everybody can play."

You solved this cryptogram in 76 seconds. That's worth 437 points!

This Cryptogram's Statistics:

Success Rate: 10.2%
Average Time: 176 seconds
Record Time: 29 seconds (nanrich)

This quote used to award no more than 149 points, and now it awards up to 449 points. I'd say that's much more realistic.

[wvwoman]

I'm generally finding that the new difficulty ratings are more realistic. Yes, I still have trouble with some "Easy" ones (usually before coffee), and I still breeze through a few "Difficult" ones (usually after coffee), but for the most part the ratings are jibing with how hard the quote was.

Here is a great example of the new ratings being correct. This is a long quote I just solved that used to be ranked as "Easy" and that was upgraded last week to "Difficult." It was brutal, even though it was long, and the stats reflect this, and now so does the difficulty rating:

"Poker is to cards and games what jazz is to music. It's this great American thing, born and bred here. We dig it because everybody can play."

You solved this cryptogram in 76 seconds. That's worth 437 points!

This Cryptogram's Statistics:

Success Rate: 10.2%
Average Time: 176 seconds
Record Time: 29 seconds (nanrich)

This quote used to award no more than 149 points, and now it awards up to 449 points. I'd say that's much more realistic.

wow--it surely is!

[Altoid701]

I'm all for this in theory, though I haven't yet played any quotes since the change was made so I'll have to get back to you with my real opinion! As someone who goes for speed rather than completion %, this'll likely mean I have fewer points overall, which I'm okay with as I don't play enough to make the top 20 in total points anyway.

[imported_nanrich]

I'm having a hard time figuring out the new scoring system. Some quotes that have quite a low solve rate now receive much fewer points than under the old system. I've pasted in an example below. The difficulty ratings marked by players was quite low (3 something out of 10, if I'm remembering right), so that may be why the puzzle now gets a lower number of points. But it seems to me that the solve rate of 8.5% speaks for itself and that, compared to puzzles with a high success rate, a lot fewer players would even have the chance to give a difficulty rating. On the other hand, I've seen some puzzles that would previously have been rated "easy" based on their length that now get a lot of points even though the solve rate is over 50%. Anyone else noticing this?

You solved this cryptogram in 21 seconds. That's worth 146 points!
"Indifference, then, is not only a sin, it is a punishment."
— Elie Wiesel
This Cryptogram's Statistics
Success Rate: 8.5%
Average Time: 92 seconds
Record Time: 10 seconds
Your Time: 21 seconds

[abra]

Wow, that one seems to have suffered, under the new ratings. I would think that a quote with a solve rate that low, should get an automatic Difficult rating.

[admin2]

Right now we have a three-tiered difficulty system, and one of the pitfalls of that system is that there are a lot of "edge cases". This is a prime example. We're marking as "Easy" all puzzles that fall into the 1-33 percentiles overall (based on a combination of completion rate, median solve time and user feedback). This puzzle here falls exactly on the 33rd percentile, so it literally straddles the line between Easy and Medium. It made it to "Easy" just by the hair on its chinny chin chin.

Most likely we'll beef up the scoring system so that we can break it down by 5-10 percentiles at a time (instead of 33 at a time) so that things will be a bit more reflective in the points department.

Thanks!

[abra]

It's great to know that you're working on it. Thank you!

[imported_nanrich]

Yes, thanks Admin for the explanation. And also for having such a great site!

[LLapp]

Confusion about the 1-to-10 star rating system

Stephen,

Thank you for all the attention you've been giving this site lately. We players appreciate it.

Since you plan to further refine the difficulty ratings, I want to point out that there may be a critical glitch in how you calculate user feedback via average star ratings for Difficulty.

Based on reading many quote comments versus what you said recently about how you treat a quote's average star rating, I'm pretty sure that many (or even most) users have misinterpreted the meaning of the 1-to-10 star scale.

I get the impression that you read the meanings of our "Difficulty" ratings in terms of "Easy vs. Difficult," like this:

1 (extremely easy) -- 3 (easy) -- 5 (neither difficult nor easy) -- 7 (difficult) -- 10 (extremely difficult)

(Did I get that right?)

And meanwhile, I believe that many of us use that same "Difficulty" rating to rank the quote only for difficulty, literally -- like so:

1 (not difficult) -- 3 (somewhat difficult) -- 5 (pretty challenging) -- 7 (very difficult) -- 10 (nearly impossible; I only got here by googling the quote)

(Other players - Have you had this confusion?)

If I'm right about this disconnect, then you can see how it might lead you to misread many users' intended rating of "somewhat difficult" to mean "quite easy" -- which could result in you assigning a difficulty rating out of sync with reality. And it gets more complicated when you consider that the misunderstanding may be widespread but not universal.

This confusion about the star-rating scale is most apparent in how people respond to star ratings for "Offensive." When an innocent quote gets an average "Offensive" rating of 2 stars, for example, commenters typically remark, "How could anyone find this quote offensive?" The perception is that even one star equals "slightly offensive" -- whereas I think you are reading one star to mean "completely innocent." I suspect the same disconnect happens in the "Errors" rating: I might indicate a real-but-not-fatal error by clicking four stars, but now I wonder if you read that feedback to mean "Quote seems okay as is."

So, if you plan to use our star ratings for feedback data, I strongly encourage you to first update the feedback section of the solution page, to clarify the star rating scales. I bet you would get significantly different feedback after the scales are clarified.

Thank you for reading this long message!

Lisa