“Luck is when preparation meets opportunity,” Seneca, the ancient Roman philosopher, once said. For many of us, luck feels more like pure chance—the result of outcomes blindly doled out by the universe. Your coworker scores coveted tickets to the World Series, your friend finds the love of their life on Tinder, or your neighbour’s child gets admission to the best preschool in the area. You shrug and assume they just got lucky. But what if Seneca was right, and luck is more than just chance? What if you learned that there are hidden markets all around us—ones that we play in every day—that dictate who gets what? Is it possible to for us to navigate these hidden markets to sway the odds to be in our favour?
According to Wharton economist and market designer Judd Kessler, we do have the power to make our own luck and tip the scales in our favour. In LUCKY BY DESIGN: The Hidden Economics You Need to Get More of What You Want, Kessler reveals the secrets of how hidden markets work and how to successfully manoeuvre within them. In familiar markets—such as the stock market or your local farmer’s market—what we get depends on how much we’re willing to pay. Hidden markets don’t rely on prices; instead, they follow a different set of rules, and you can’t simply buy your way into a better position. As Kessler, who has spent a career studying and designing these very markets, explains, “Without understanding the rules of the game, you might assume that your outcomes are determined mostly by luck. People who end up unhappy about what they get conclude that they were unlucky or that the system was rigged against them. After enough of these experiences, they believe that’s just how the world works.”
In this interview I speak to Judd B. Kessler, the inaugural Howard Marks Endowed Professor of Business Economics and Public Policy at the University of Pennsylvania’s Wharton School. We discuss the hidden markets that determine almost everything in our lives, and how to tip the scales in our favour.
Q: What do you hope we all get to learn about hidden markets?
[Judd Kessler]: When it comes to hidden markets, I want people to know they exist, that they allocate many of the things we really care about, and that the trick to succeeding in them is figuring out the rules that guide them and then deciding what strategy you should play to get what you want.
It might feel—if you haven’t yet learned the rules of these hidden markets—that the outcomes are based on luck, chance, or things beyond your control, but you actually have a lot more agency in them than you think.
Q: What is a hidden market?
[Judd Kessler]: Let me define hidden markets, and then I’ll give you examples, because I think it’s easier to see them once you have a framework.
I call markets where price allocates goods or services visible markets. When you go into a store and decide whether to buy something at a given price, all you do is pick it up off the shelf and ring it up, that’s easy to do business with. That’s a visible market—price is allocating the scarce resource.
Hidden markets are markets where price is not doing all the work. There might be a price, like for a live-event ticket where there’s a face value price on the ticket, but that’s not doing the allocation. They don’t have enough tickets for many popular shows for everyone to buy one at that price. So, you need some other way to decide who gets what. Sometimes it’s first-come, first-served; sometimes it’s a lottery. The same thing comes up for restaurant reservations and very popular products like clothing drops from high-end clothing lines. It used to happen with new iPhones. There was a fad product this summer called Labubus that everybody wanted to buy but nobody could get. These are cases where there is a price, but price is not determining who gets access to the goods.
Q: What is the role of luck in hidden markets?
[Judd Kessler]: … luck will sometimes play a role, like when a product is allocated by lottery, and you can get lucky in the lottery or not.
But there are often things you can do to enhance the likelihood of success. Some lotteries allow you to enter multiple times or allow you and someone you’re working with to independently enter the same lottery. You might both enter to win, say, two passes to go hiking in Half Dome in Yosemite in California or two tickets to the theatre. If either of you wins the lottery, you both get to go, so if you both enter, you double your chances of success. Those are lottery-based systems, but you actually have some control over the outcome.
You might look at those settings and think, oh, it’s just a lottery, it’s just based on luck. But there are ways you can improve your chances. And that logic extends to all the other ways we assign things in hidden markets. First come, first served races, lines, and waiting lists; centralized clearinghouses where you rank your preferences and an algorithm matches people: these potentially have some luck component you can improve upon.
There are also hidden markets where there’s no price at all. Seats in elementary schools for my son and daughters don’t have a price—you don’t pay for a spot—there is some other process to decide who gets which seats in which schools. Any time there’s some kind of government benefit they want to give away for free, or in the case of medical care, which differs a bit across the US and UK, there’s going to be some access to doctors or specialists where they might not rely exclusively on price.
Any time you’re in a situation where the price is “too low”—and I put that in quotes because it might not feel low to you, sometimes things are very expensive, but there are more people who’d like to buy the thing than they have the ability to serve—that’s when a hidden market will crop up. The econ term is excess demand; whenever there’s excess demand—more people who want the thing than items available—you’ll get a hidden market. The goal is figuring out the set of rules that determine the hidden market.
Thankfully, we have so much experience with hidden markets in our lives that once you think about it that way, it’s not hard to identify which set of market rules is applying. We all know what standing in line is like—that’s a first come, first served allocation rule where you have to physically be present to keep your place—that’s one set of market rules. There are also things like lines where you don’t have to stand there; you just put your name down. We call those waiting lists, and we’re very familiar with them. And then there are ones with similar features to a waiting list, where the earlier you get on it the better, but they do all the allocations instantaneously—we call those first-come, first-served races. We’ve all experienced playing in those.
So, when you see the environment you’re in, then you can figure out: okay, what are the rules that apply to this particular market? And once you know that, you can start to figure out what’s the optimal strategy for you in that market.
Q: Do hidden markets have to be fair?
[Judd Kessler]: … often hidden markets are not fair. This is what I talk about in the book as hidden markets that are designed poorly. The goal of a hidden market, from my perspective as an academic, is to satisfy what I call the 3 E’s, being efficient, equitable, and easy for market participants. Achieving those three things is the platonic ideal of a hidden market.
The problem is you cannot achieve all three E’s simultaneously. Early work in the subfield of market design shows it’s very hard and very unlikely you’ll get a hidden market that can successfully allocate things efficiently and equitably in a way that’s also easy. That’s what makes it challenging and interesting—because a lot of times you face trade-offs.
You can make something more fair and more equitable, but you’ll often sacrifice efficiency or ease. Take the lotteries we talked about earlier: if you had a fair lottery where everybody got one entry and that was it, that would be equitable—everyone would have an equal shot—but it wouldn’t give any extra chance of success to folks who really, really wanted to win, who had a high valuation for the thing being lotteried. So, you might worry you’re missing out on efficiency by making it as fair as possible.
Q: Do we need to rethink what it means to win in a hidden market?
[Judd Kessler]: Some strategies are very market-specific. For example, when it’s a first come, first served race or a first come, first served waiting list, being early is important—the obvious strategy is to get there as quickly as possible. Some strategies, like being fast, are very specific to those market rules.
But one strategy that runs through many different types of hidden markets is the intuition that the optimal strategy might be pretending that your second choice is actually your first choice—acting as if something you like less is the thing you want most. The reason that this strategy works is that when you’re in competition with other people for scarce resources—which happens in hidden markets—you can often do better by going for something other people find less desirable.
Let’s say it’s a first-come, first-served race for restaurant reservations. We all know the 7 pm reservation is more desirable than a 4 pm reservation for dinner, except perhaps if you’re retired and want to be in bed by six. If you try to get 7 pm when reservations become available, you might not be the fastest clicker on the website, and you could end up out of luck. Then you might turn around and say, okay, let me try for 4 o’clock now. But while you were clicking for 7, others were playing the strategy of pretending something less desirable was their first choice and were clicking for the 4 pm reservation. They will end up getting the 4 pm reservation rather than you.
I call this strategy, of going for the 4pm instead of the 7pm reservation, settling for silver—as in the silver medal—as opposed to going for gold, which would be trying to get the 7 pm reservation.
This comes up a lot; there are many settings where the settling-for-silver strategy is optimal.
[Vikas: In your book, you also related this back to dating!]
[Judd Kessler]: Yeah, so dating is interesting because it’s slightly more complicated, in a way that’s great for us as a society. If you ranked restaurant reservation desirability, we might basically all agree on which reservations are better. Dating is, thankfully, not like that.
In dating, there are two parts to people’s preferences. This is how economists think about these things, so I come to it with that framework—it might seem, I don’t want to say “unromantic,” but it might seem strange to think about preferences this way—but hey, I’m an economist.
One part are general preferences that we all agree on. We have definitions of people being classically handsome; we understand there are things many of us value in partners. A lot of people like when their partners earn more money or have more prestige in society, and that’s a general preference probably shared by most people.
But we also have idiosyncratic preferences. People have hobbies, and if their hobbies look like my hobbies, I might be really interested in that. If we like the same kinds of movies. Those aren’t general preferences; we can’t rank hobbies from best to worst and have everybody agree. We like the things we like. We’re also attracted to certain physical types or certain features in partners that might not be commonly shared by everyone. The thing in dating markets is that the optimal strategy is to focus more on your idiosyncratic preferences—the things you particularly like.
There’s a great study in South Korean dating markets where researchers gave participants—think of it as a Tinder-style setup—profiles to “swipe right” on. They also gave them “special swipes,” which they called roses, to send with a proposal to match with somebody. What the researchers found is that if you use a special swipe on a person, on average they respond by being more likely to match with you. But they really only respond if they aren’t the most desirable type based on general preferences. Very attractive people—who everybody agrees are great—don’t care if they get a rose or not. It’s the people unsure if your interest in them is sincere who respond. When they get the rose, they react very positively to it and are much more likely to accept your proposal. Giving roses to people you like because of your idiosyncratic preferences, rather than general preferences, helps you succeed on the dating market.
Q: What about the hidden markets in businesses?
[Judd Kessler]: I of course see hidden markets everywhere, including in business.
I think a lot about the labour market. Consider going after top talent. It’s possible that if you’re trying to get the best AI researcher for a new initiative, and everyone is going for the absolute superstar of the market and bidding up that person’s price dramatically, the settling-for-silver strategy might work in that environment. If there are people just below the superstar level who aren’t getting the massive influx of demand from companies chasing those superstars, they might be a good fit for your firm—both in terms of cost-effectiveness and recognizing that people like being wanted.
If you’re going after someone nobody else is pursuing, that might carry a lot of weight with them once they’re actually at the firm. Often, what you’re looking for is not just someone you can hire, but someone who will stick with the firm long term and not be easily poached. You want someone interested in you and your firm, and you can win a lot of goodwill with that person by pursuing them even when no one else is.
And it’s natural, by the way, for there to be a pile-on—everyone going after the same people, creating correlated preferences. You might want to take a step back and ask, am I pursuing this person just because everyone else is? Is there actually a good alternative with less active demand right now?
Q: How is AI helping (or hindering) hidden markets?
[Judd Kessler]: …I think AI is potentially creating problems in some hidden markets because we used to rely on certain actions as signals of people’s interest and AI is changing the costs of those signals. Think about an application to a firm. What is the hiring manager trying to figure out? They’re trying to assess if this person is good enough to be at our firm, do we like them? And, also, is this person likely to accept our offer if we make one and then stick with the firm for a while? That’s the person I want to identify here.
We used to have ways to suss that out with things like: did they write a detailed, thoughtful cover letter that shows how their skills could add value to the firm and that they’ve thought about the job and what they could contribute? The reason the cover letter was so useful is it was a costly signal of interest. Writing a good cover letter took hours, often required researching the firm and position, and you’d only do that for jobs you were excited about. If you were sending 400 resumes, you couldn’t send 400 thoughtful cover letters; you’d send 5 or 10. That was the firm’s way of knowing this person is genuinely interested in this job, and if we offer, they’ll probably accept and stay a while.
Enter large language models: now the cost of writing a detailed, thoughtful cover letter has basically dropped to zero. With ChatGPT, you could write 400 thoughtful cover letters that explain what you add to the firm. That signal of interest is basically gone. It’s worthless. So, in that way, AI has disrupted some of the ways our hidden markets function, forcing us to rethink what will work as a signal in that market now. I think it has to be something a large language model can’t produce. It has to be developing a relationship with an actual person in the real world: showing up at industry events, networking, getting referrals. All those things AI can’t replicate are going to become more and more important in the hiring process.
Q: Can we apply these principles more broadly to policy?
[Judd Kessler]: … hidden markets are often created by accident by people trying their best to allocate scarce resources often without carefully thinking about how to do it efficiently, equitably, and easily for market participants. It’s a little ironic, because when the government decides to allocate something without the aid of price—when they try to give something away for free or not let money enter the allocation—they’re often doing it because they think it would be inefficient or inequitable to use prices.
We think it would be inequitable to sell organs for transplant to those willing to pay the most. As a society, we decided we don’t want that type of market. So, there’s a hidden market instead, and we pick some set of rules, but often without carefully considering the efficiency, equity, and ease of those rules. We see it with organ allocation, we see it in healthcare markets, but also with library books and public housing. All of these markets allocate resources and we could potentially improve how the allocation is done.
Q: Do you think we therefore need better statistical literacy?
[Judd Kessler]: … one of the things that motivated me to write the book was the belief that it behoves us—whether policy makers or academics who study these types of hidden markets—to help market participants do better in them. Often that means, when the system is too complicated to understand fully, giving advice about what strategies to play. It’s hard to get people to fundamentally understand the laws of probability, but we can help by providing advice or by deciding whether to have a government lottery for folks to participate in or whether to legalise gambling in certain places.
And in these hidden markets, if the market is complicated, then in the book I explain how the market works but always make sure to also say: look, this is the strategy you should play here.
For example, in certain markets, the best thing you can do is rank your preferences honestly and make sure you include at least one thing you’re very likely to get. That thing could be at the top or bottom of your list, but you should include it. You don’t have to understand all the ins and outs of the market—allocating, say, elementary school seats—to know the optimal strategy.
It’s a little like this: yes, I’d love it, as a professor of economics, if everyone fully understood probability and we could train people to grasp that the probability of winning a lottery is infinitesimal and you’re probably throwing your money away, but it’s hard to teach a lot of people those deep concepts. We may just have to design the world so it’s easier for folks to play the strategies that are best for them.
Q: How do we turn ourselves into a hidden market?
[Judd Kessler]: Well, first, I think you don’t have to pretend. You are the market designer for the hidden market of your time and attention. You have a supply of hours in a day, and there will always be more demands on your time than you can meet. A lot of people will email you asking for things—you have friends, potential romantic partners, family, your boss texting you.
You have to decide how to allocate those scarce resources, and there are a bunch of principles you could use to optimise that hidden market. You could prioritise making your allocation efficient, equitable, and easy—the 3 E’s I’d like all market designers to use—but there’s also a 4th: elevate yourself as the market designer. Make sure the market rules you set up to allocate your time and attention are the best possible for you. That might mean prioritising the things most important to you; it might mean prioritising friends and activities that energise you and make you better able to bring your best self to everything else you do.
In the book, I discuss research I did with Alvin Roth, my graduate advisor who won a Nobel Prize in economics. We looked at mechanisms that provide extra access to scarce resources for people more likely to provide them. One example I’ve studied extensively is giving priority on organ donor waiting lists to people who registered as donors when they were young and healthy—when they were 18, they said, “I’d be an organ donor if I was ever in that position”—and we reward them with higher priority to get a kidney if they end up needing one. You could do a similar priority policy for your time and attention: prioritise your time for people who give you energy and opportunities to bring your best self forward. That might mean yourself, self-care, or friends and family who do that for you, while pushing away those who drain your time and attention. That’s a life strategy I think really applies the book’s concepts.