What an Early Access Beta Actually Teaches You

The Wrong Reason to Run a Beta

Most studios run a beta to find bugs. That's useful, but it's not the point.

Bugs you can find yourself — with enough time and a decent test suite, you'll catch most of them before anyone else touches your game. What you can't find yourself is everything you got wrong about how humans actually play your game. The assumptions baked so deep into your design that you stopped seeing them. The flows that feel obvious to you after two years of development that are completely opaque to someone opening the game for the first time.

That's what a beta is for. Not QA. Calibration.

When we opened Retropolis to our first wave of beta players, we expected to hear about crashes and edge cases. What we got instead was a mirror held up to every decision we'd normalized without realizing it.

How We Structured the Beta

We ran the beta in waves, not as a single open floodgate.

Wave 1 (30 players): Close contacts — other developers, gaming friends, people we could call directly. The goal here wasn't feedback forms; it was observation. We watched people play over Discord calls, asked them to narrate what they were doing, and took notes. This is embarrassing but essential. Watching someone confidently navigate to the wrong menu because our labeling made sense to us but not to them is the fastest way to fix something.

Wave 2 (200 players): Discord community members who'd been active and vocal. They already had context on what Retropolis was trying to be, which made their feedback more directional. They weren't just telling us "this is confusing" — they were saying "this doesn't match what you described in your devlog post."

Wave 3 (open beta, invite required): Broader access, but still controlled. We gave out invite codes and asked existing beta players to share them selectively — friends who might actually play a location-based game, not just anyone they knew.

Each wave had a structured offboarding: a short survey (5 questions max) and an invitation to join a focused feedback thread in Discord. We didn't want exhaustive written reports. We wanted the three things that bothered them most and the one thing they liked. Short answers force prioritization.

What Surprised Us

The onboarding cliff

We knew our onboarding needed work. We didn't know it was effectively a wall.

The first session is dense in Retropolis. You're dropped into your real neighborhood, asked to navigate a map, claim a territory, build something, collect resources, and understand what all of it is for — in roughly that order, with minimal hand-holding. We'd designed it that way intentionally. We didn't want to over-explain; we wanted players to discover.

What beta players taught us: discovery only works when you have a reason to explore. Players who didn't understand the core loop within the first 10 minutes didn't continue. Not because they were confused by specific mechanics, but because they didn't have a compelling question to answer. "Why am I doing this?" is a more important onboarding question than "how does this work?"

We rebuilt the first session around a single goal: claim your first building. Everything else moved to later. Retention in the first session went up significantly.

Vocal minority feedback is real and dangerous

Our most active Discord members gave us the most feedback. They also gave us some of the worst product advice.

Highly engaged early players have strong opinions shaped by their unusual dedication to your game. They want depth, complexity, and power-user features. They will tell you the game is too easy, too slow, too simple — because for them, it is. They are not your median player.

We nearly over-tuned the mid-game difficulty based on feedback from about 15 very active beta players. When wave 3 arrived, the same section of the game had the highest drop-off rate we'd seen. The 15 loudest players had convinced us to make it harder. The 1,800 quieter ones showed us, through behavior rather than words, that we'd gone too far.

The rule we adopted: behavioral data overrides opinion data. What players do is more reliable than what they say.

What players loved was always something we built quickly

The feature players mentioned most positively in beta was our notification system — the little push that told you when someone was attacking your building, with just enough information to make you want to open the game. We built it in a weekend as an afterthought. The features we spent months on — the item crafting system, the gang war mechanics — got used, but they didn't generate the same spontaneous positive feedback.

This is a common pattern in game development and we should have seen it coming. The features that generate immediate emotional response are usually simple, immediate, and social. The features that generate long-term engagement are usually complex, delayed, and solitary. Both matter, but don't assume polish time correlates with player impact.

What We Changed

Onboarding: Rewrote the first session entirely. Single clear goal, deferred complexity, earlier gang creation.

Economy balance: Adjusted resource costs down across the board. Our internal playtesters had optimized over hundreds of hours; new players were resource-starved in a way that felt punishing rather than challenging.

Map UX: Added a "your buildings" filter on the map. Something so obvious we can't explain how it wasn't there at launch.

Notifications: Expanded the notification system because players responded to it. Added alerts for building income ready to collect, rival gang activity in your territory, and new contracts available.

What We Didn't Change

An early access beta will also tell you things you should ignore.

Several players asked for PvE-only servers — a version of Retropolis without player conflict, where you couldn't lose your buildings. This is a real request and we understand it. It's also incompatible with the game we're making. Retropolis is a territory game. Tension and loss are load-bearing mechanics. We noted the feedback, understood the underlying need (too much anxiety around losing progress), and addressed it differently — by reducing punishment severity on building loss rather than eliminating the mechanic.

Feature requests that would require abandoning your core design aren't feedback. They're a different game. Smile, thank the player, file it away.

The One Thing We'd Do Differently

We started the beta too late.

By the time wave 1 happened, we had six months of polished game development that turned out to be oriented around a flawed first-session design. The onboarding cliff I described above? We could have found that in month two with five playtests. Instead we found it in month eight after building an entire progression system on top of an assumption that wasn't true.

If you're building a game: get it in front of strangers as early as you can stand it. Not friends. Not colleagues. Strangers who will be confused by things you've normalized and politely tell you, or rudely tell you, or just quietly stop playing — all of which is useful information.

Ship the ugly version. Learn earlier. The code can be cleaned up. Time cannot.