Why Easy Puzzles Weren't Always Easy (And Difficult Ones Were Sometimes A Snap): Rethinking Difficulty in CrossEights
We've changed how CrossEights measures puzzle difficulty. Here's what wasn't working, what we discovered, and how the new approach is changing the playing experience.
Here's something that was bothering us about CrossEights: sometimes an Easy puzzle wasn't easy at all.
Players could breeze through a Difficult puzzle in a couple of minutes, then find themselves completely stuck on a Medium one. Even within the same difficulty level, some puzzles were much harder than others.
That wasn't how the game was supposed to work.
CrossEights is designed to reward good solving: recognizing word patterns, making smart deductions, using hints strategically, and moving quickly when the answers start to emerge. Difficulty should make those skills more demanding, not produce random surprises.
So we set out to understand why our difficulty levels weren't behaving as expected.
Our original assumption: Harder words make harder puzzles
When we first developed CrossEights, we classified puzzles primarily by the familiarity of their eight words.
We used established word-frequency data, including the Corpus of Contemporary American English (COCA), to measure how commonly words appear in everyday language.
The idea was straightforward. A puzzle containing eight familiar words should be relatively easy. One containing several less common words should be harder.
And to a degree, that was true.
But the relationship wasn't nearly as strong as we'd expected.
Some puzzles containing perfectly ordinary words were surprisingly difficult. Others, with several less familiar words, practically solved themselves.
We had been overlooking something fundamental about how CrossEights works.
It's not just the words. It's how they fit together.
Unlike a traditional crossword, CrossEights has no clues. You solve its eight interlocking words by recognizing letter patterns and using intersections to uncover additional letters.
That means the difficulty of a puzzle depends on much more than the individual words.
Imagine two puzzles containing equally familiar vocabulary.
In the first, your opening hint reveals several useful letters in multiple words. You recognize one word, its crossings help identify another, and suddenly the puzzle starts coming together.
In the second, the same opening hint reveals fewer useful patterns. Perhaps several letters appear in one word while the others remain mostly blank. You can recognize every word in the puzzle when you see it, but you don't yet have enough information to identify any of them.
The vocabulary might be equally familiar, but the solving experience is completely different.
We realized that difficulty in CrossEights emerges from the interaction between the words, not simply from the words themselves.
That discovery changed how we approached the problem.
Making the game harder wasn't the answer
Our first experiments focused on gameplay mechanics.
What if we reduced the number of hints? Revealed fewer letters? Limited the letters players could choose?
These changes generally made puzzles take longer to complete. But they didn't necessarily make them more interesting.
An inherently easy puzzle could still be easy. A stubborn puzzle could become an exercise in frustration.
Worse, those restrictions reduced one of the things we value most about CrossEights: the player's ability to make intelligent choices about how to solve.
We didn't want difficulty to come from withholding the tools players needed.
We wanted it to come from the puzzles themselves.
So rather than changing the way CrossEights is played, we decided to change how its puzzles are classified.
Three things that make a puzzle difficult
After examining puzzle structures and how people solved them, we developed a new difficulty model based on three factors.
1. How familiar are the words?
Word familiarity still matters.
Recognizing a common word from a few letters is usually easier than recognizing an unusual one. We continue to use word-frequency information to evaluate the vocabulary in each puzzle.
But vocabulary is now only one part of the equation.
2. How much do the opening hints reveal?
This turned out to be particularly important.
Experienced CrossEights players often begin by revealing common vowels and consonants. Depending on the puzzle, those choices can expose a surprisingly different number of letter positions.
But the number of letters isn't the whole story.
Twenty revealed letters spread across eight words can be much more useful than twenty letters concentrated in just two or three.
We now evaluate both how many letters a typical opening hint reveals and how those letters are distributed throughout the grid.
We call this Foothold Yield: how much useful starting information the puzzle offers.
3. How do the words intersect?
This is perhaps the most interesting factor.
Every CrossEights grid contains sixteen intersections where across and down words share a letter.
Some intersections make neighboring words easier to recognize. Others create ambiguity, requiring you to work back and forth between possible answers.
The placement of common vowels, unusual consonants, and recognizable word endings can make a substantial difference.
We call this Intersection Resistance.
Together, vocabulary familiarity, foothold yield, and intersection resistance give us a much richer picture of how challenging a puzzle is likely to be.
Testing the new approach
Of course, a difficulty model that looks good on paper isn't necessarily going to work with real players.
We created a separate testing environment, called DiffTest, and invited frequent CrossEights players to try puzzles classified using the new approach.
Their feedback helped us identify several problems we hadn't fully anticipated.
For example, some puzzles revealed plenty of opening letters, but those letters were concentrated in just a couple of words. The puzzles looked promising according to the raw counts but remained frustrating to solve.
We refined the model to consider the distribution of those letters, not just the total.
We also discovered inconsistencies when puzzles were transposed—switching their across and down orientations. The same underlying puzzle could receive different classifications depending on its orientation.
That needed to be fixed.
And we reconsidered our Special Blend category. Rather than making these puzzles difficult through increasingly obscure vocabulary, we wanted them to offer a different sort of challenge: unusual letter combinations and patterns that require a fresh approach.
Through several rounds of testing and refinement, we arrived at our new classification system, Model 2.4.
Did it actually make a difference?
The results from our testing were encouraging—and in some cases, dramatic.
Under the original model, the difficulty levels weren't very different in practice. Completion rates varied by only about six percentage points between tiers, and Difficult puzzles took players only about 10–20% longer than Easy ones.
More than 22% of Difficult puzzles were completed in under two minutes. That was a pretty clear indication that something wasn't right.
With the new model, the separation between difficulty levels became much more pronounced.
Here's how the playing experience changed:
Several findings were especially interesting.
First, difficulty now has a much clearer progression. Difficult puzzles take substantially longer to solve than Easy ones, rather than producing nearly identical solve times.
Second, opening hints behave very differently. A typical opening reveals about 21 letters in an Easy puzzle, compared with just 13 in a Difficult one. That's a meaningful difference in how much information players have to work with from the outset.
Third, hints are being used differently. Players average 1.9 hints on Easy puzzles and 3.1 on Difficult ones. More demanding puzzles are requiring more assistance, as we'd expect.
And perhaps most strikingly, those unusually fast Difficult solves—where a supposedly challenging puzzle suddenly collapsed in under two minutes—have virtually disappeared in testing.
None of this means we can predict exactly how long someone will take to solve a particular puzzle. Individual vocabulary, experience, strategy, and occasional flashes of insight will always matter.
Nor do we want every puzzle within a tier to feel identical.
But the results suggest we're much closer to our goal: difficulty levels that offer meaningfully different, more predictable challenges.
What changes for players?
Beginning November 16, CrossEights will use the new difficulty classifications across both Daily and Rival play.
The four difficulty levels remain, but each has a more clearly defined character.
Easy: Familiar vocabulary and generous starting information. Designed for an approachable, satisfying solve, particularly for players who are learning how CrossEights works.
Medium: A balanced challenge, with less immediate information and more opportunities for deduction.
Difficult: More demanding vocabulary, fewer obvious footholds, and a greater need for patient, strategic solving.
Special Blend: A different kind of challenge, featuring unusual letter combinations and unexpected patterns rather than simply harder words.
Importantly, the gameplay itself hasn't changed. The hints, scoring, and solving mechanics remain the same. We're simply being more selective about which puzzles belong in each category.
A little help from our players
This project benefited enormously from the people who regularly play CrossEights and were willing to test unfamiliar puzzles, report their experiences, and tell us when something didn't feel right.
We're particularly grateful for the thoughtful feedback from an accomplished professional crossword constructor, whose observations challenged us to look more closely at the relationship between vocabulary and grid structure.
It's easy to assume that difficulty is something a puzzle designer can simply assign. We've learned that it's something you have to understand, measure, and continually refine.
We're looking forward to seeing how the new classifications work in everyday play.
And if a puzzle feels particularly easy—or unexpectedly difficult—we'd still like to hear about it.
Because the puzzles are getting better partly because of the people playing them.
Curious what happened when we applied the new difficulty model to our entire puzzle library? Read 27 Million Puzzles: What We Discovered About Difficulty in CrossEights.