737

How Go Players Disempower Themselves to AI — LessWrong

__Written as part of the MATS 9.1 extension program, mentored by Richard Ng_o.

From March 9th to 15th 2016, Go players around the world stayed up to watch their game fall to AI. Google DeepMind’s AlphaGo defeated Lee Sedol, commonly understood to be the world’s strongest player at the time, with a convincing 4-1 score.

This event “rocked” the Go world, but its impact on the culture was initially unclear. In Chess, for instance, computers have not meaningfully automated away human jobs. Human Chess flourished as a pseudo-Esport in the internet era whereas the yearly Computer Chess Championship is followed concurrently by no more than a few hundred nerds online. It turns out that the game’s cultural and economic value comes not from the abstract beauty of top-end performance, but instead from human drama and engagement. Indeed, Go has appeared to replicate this. A commentary stream might feature a complementary AI evaluation bar to give the viewers context. A Go teacher might include some new intriguing AI variations in their lesson materials. But the cultural practice of Go seemed to remain largely unaffected.

Nascent signs of disharmony in Europe became nevertheless visible in early 2018, when the online European Team Championship’s referee accused a player, Carlo Metta, of illicit AI use during a game. His results were voided and he was banned from further participation in the event. At the time the offending game was played, open-source engines based on the AlphaGo paper, such as Leela Zero, had only been around for about a month. However, a predecessor called Leela 0.11 was already widely available and was known to match the level of the top Europeans that Metta was facing. Metta’s accusers claimed that his play was too similar to this AI’s preferred moves. It was moreover considered suspicious that his Over-The-Board (OTB) play agreed significantly less with the AI than his online moves did.

Unfortunately for the prosecution, their results were reported in intransparent and sloppy ways. This is evidenced by the fact that the best compilation of their findings is the slapdash facebook thread I linked above. This, along with the circumstantial nature of the evidence, was criticised in the same thread by community members. Teammates and friends of Metta’s also stepped up to publicly defend him. One way in which their rhetoric proved effective involved the public stigma and disdain against AI cheaters; this ironically made the case against Metta seem unfair and disproportionate due to the perceived gravity of the accusation. Ultimately, the Italian team appealed the decision and they won. Carlo Metta was officially exonerated.

Among non-Italian European Go players, the claim that Metta used AI in almost every game in the ETC since 2018 has become barely disputable, especially considering how things developed. In the 2017/2018 season, he scored wins roughly half the time, likely using Leela 0.11 against opponents who were roughly the bot’s level. That same year, the Italian team was relegated to a lower league where no-one powerful in European Go politics cares to look anyway. This coincided with the popularisation of Leela Zero, a properly superhuman open-source go engine. Metta went on a 9-0 streak against opponents matching his OTB level in the 2018/2019 season, scored 9-1 in the 2019/2020 season, and then won 25 out of 26 games in the following years[\[1\]](https://www.lesswrong.com/posts/nR3DkyivzF4ve97oM/how-go-players-disempower-themselves-to-ai#fnn7zme9cq6za). His only loss in this last streak was in a match where he was forced to play under camera control. During this time, his OTB level remained stagnant.

At this point, considering Metta “innocent” represents a near-categorical rejection of convictions based on circumstantial evidence. I am not here to litigate that question, but am nevertheless comfortable assuming here that Metta was regularly using AI for these games. However, this is only the very start of our story because it illustrates some key points about the sociology of AI use in Go. First, the public announcement of his disqualification and the ensuing discourse vilified AI cheaters (incorrectly as it turns out) as being unusually dishonorable and evil. Second, he set the precedent that AI users would basically never get punished, no matter how obvious their cheating was even while under investigation. They could always just get their allies to kick up a fuss and pressure organisers into reversing the decision. These features made accusing people of cheating socially costly, _and gave tournament organisers and fair-play committees an expectation of futility. Cheating in online European events thus became trivially easy due to a near complete lack of functional mechanisms for retribution.

I started my career as a Go teacher in 2020, producing technical game reviews for a newly re-established online Go school set up to meet pandemic demand. We had not planned for cheating to be a major issue in our school. Whereas illicit AI use was already a well-known problem for the growing ecosystem of online tournaments, we didn’t expect it to affect our unrated, prizeless teaching league. To the contrary, we soon became cognisant of how some of our students were outputting better games than we, their teachers, could ever hope to play. Occasionally, AI use was unmistakably blatant because _both sides played top AI moves for the entirety of the game. I now estimate that about half our students had used AI in at least one game and one in ten were chronic users. We were originally baffled by our observations. It didn’t make sense that players would just throw away their practice games to have AI win on their behalf. We also struggled to decide what to do about the problem and were reluctant to address it for roughly the same reasons that most tournament organisers were.

Around the same time, I was asked to look into the online games of a promising young player that a friend suspected of using AI in a youth league. Like at the Go school, I was surprised at how easy cheating was to detect since nearly all the kids regularly used AI against each other. This incident and other similar ones made me gradually realise that illicit AI use was entirely endemic to the Go world. It fortunately turned out that this pattern didn’t generalise to the really important or prestigious tournaments that were held online during COVID. The symbolic camera controls – which would be easy to circumvent for a dedicated cheater – seemed sufficient to curb almost all cheating in a way that threats or impotent references to “fair-play committees” were failing to. This reminded me of how Metta tended only to lose in online tournaments[\[2\]](https://www.lesswrong.com/posts/nR3DkyivzF4ve97oM/how-go-players-disempower-themselves-to-ai#fngykvsqgi7j5) when playing under a camera (or when facing another AI user).

Back to my hapless colleagues and I at the Go school, we initially settled for drily implying that suspicious games were “too good to review” and emphasising how we couldn’t help students who were playing “at such levels”. Our students caught on, and we were subsequently lucky to get some private confessions of cheating; over the years I was able to follow up with and interview many students that used AI, including some that hadn’t originally come forward. The appealing, exciting archetype of a cheater is one that uses covert, elaborate methods to get outside information and fraudulently obtain prize money or prestigious titles. Instead, we learned from the many examples of cheating and player confessions that _idle curiosity and laziness were the dominant reasons for AI use in our school. Our students would often set out to play a normal game of Go, but would get stuck on a particularly difficult or annoying move; eventually, their curious eyes would drift to their second monitor — where they usually had their AI software running anyway — and they would check the answer as one would sheepishly side-eye the solution to an interesting puzzle or homework problem. Another reason people cited for using AI was an emotional investment in preserving or improving their image within the school community. Some wanted to avoid appearing incompetent and would employ strategies such as only playing moves that lost “n” points or less in expected value according to their computer.

None of these reasons were surprising to us; we had already thought of most of them while shadowboxing our pupils’ strange behaviour. What personally shocked me, however, was the way our students conceptualised their AI use. In this, Carlo Metta was also a surprisingly predictive case. The original reddit thread discussing his ban featured a comment from a user called “carlo\_metta”, which read:

I never let Leela choose move. I just decide myself which one is better, for this reason i think i can find my own style with Leela. Go is an art and Leela help me tyo \[sic\] express my skill

That account was a burner, quite possibly a troll. However, I couldn’t help but recall the comment when I heard identical arguments coming from our cheating students’ accounts. A central part of every student’s retelling was that despite their AI use, they retained artistic control over their output and could exercise agency to think and improve for themselves. The AI felt to them like a tool that helped them fulfill latent potential or artistic sensibilities.

AI users never find out they haven’t “got it”.

Continental European math undergraduate degrees have a deserved reputation for their brutality, with completion rates of 10-15% being relatively common. Many of the 90% drop out nearly immediately, but some stick out the entire first year. These can often follow along with the proofs and exercise corrections’ atomic steps, which gives them false hope. However, they tend to struggle to see the “big picture” motivations of the material and are likely to have their hopes unravelled eventually. I was accidentally privy to a collective unravelling at the end-of-year third sitting of an exam on some basics of algebra and matrix calculations. I was retaking it to boost my grade from earlier in the year, but no other remotely competent person had bothered to do the same. Outside the exam hall, I listened to some other forty students’ chatter and had my blood internally curdle at phrases such as “I hate the proofs but I can do the exercises” or “I memorised all the matrix multiplication laws for this one”. The exam itself was quite unconventional; the professor clearly figured we would have had enough of manipulating matrices and instead asked an eclectic mix of simple algebra questions that to me vibed as “these are fundamental exercises you should be able to do if you have learned to think like a mathematician by now”.

The atmosphere on exit mixed depression with vitriol. People complained on the object level about the exam, usually about how it was too niche or off-topic with respect to the material. However, there was something more fundamental going on. People had shown up with bags of half-baked heuristics and hand-copied exercises and proofs. That exam had put them face to face with the fact that their memory aids were never going to help them “get it”. I don’t think I saw any of them ever again.

The population of Go AI users – both those who cheat in online games and those who simply review their games with AI post-hoc – is one on the perpetual eve of that exam. They fire up their computer out of idle curiosity and nod along passively as the truths of the universe float by them. They register the insights not one bit more because they can click the sublime moves. People consistently underestimate just how lost they will be when the solution is no longer right in front of them. This perspective of AI use to me explains why camera controls proved so effective against online cheating. Since AI use is usually an act of self-debasement and disempowerment – a subjection of oneself to ambient incentive gradients – it fundamentally contradicts the aesthetics of resourcefully overcoming a minor obstacle.

The illusion of control that AI users have reliably shown interacts in an insidious way with their disempowerment. It contributes to a society of Go players that allow their participation in culture to be automated away. They are moreover so disempowered about it that they have built-in psychological mechanisms to keep them from ever recognising their own obsolescence. This mechanism even works to sabotage the detection of AI use in _others. People tend to give overly conservative estimates of the chances a given game involves AI. I think this happens because they usually consult their own AI to check a suspected game. In doing so, they also come around to the machine’s point of view and conclude that playing the correct AI move was the “natural” thing to do anyway in that situation.

My view of AI use (especially cheating) in Go originally manifested as disgust for its practitioners. I switched eventually to an attitude of compassion and pragmatism towards a habit that was clearly much more vulgar and weak than it was evil. Over time, I have progressed to feeling deep sadness for a group that surrenders much of what it claims to value. The thing I want to impress with this article is the consistency with which we as a species underestimate our own willingness to give up our culture, economy and autonomy to AI, _even without monetary incentives. For this to happen, AI does not even need to be superhuman. Indeed, Go AI automates human players’ role in culture as shallow simulacra. All an AI needs to do is be passably good at a task and that may well be enough for people to volunteer their own replacement.

Appendix A: No, Go players aren’t getting stronger

One of the objections I can anticipate to this pessimistic monologue is that expert Go players seem to have improved since AI became widely available. There’s a modest body of research in the field of Cultural Evolution advocating this, including this paper and related ones from the same group of authors. These views have been promoted by blogs in the techno-optimist orbit and one of the associated graphs was recently making the rounds on Twitter. I have already written a post analysing the data used for the research, where I concluded that it is being misinterpreted. The tl;dr is that all improvement in the quality of play comes before move 60, when humans can mimic memorised AI policies. Play after move 60, in the pivotal parts of the game, shows no improvement. For me to think there’s any meaningful change in human play from pre-AI times, I would have to be convinced that players understand the AI moves they copy well enough to keep a heightened level when they go off-policy after the opening. There is no evidence of this.

Appendix B: Why this article exists

This piece is not meant to rigorously justify that Go players are disempowered or to carefully explore the shape of that disempowerment. It is instead designed to communicate a vibe from anecdotal experiences in the Go community that I think can give useful intuitions about Gradual Disempowerment as a general phenomenon.

1. ^ I scraped the data from the tournament website using a vibecoded script (ironic!) and manually verified most of it. 2. ^ Metta also regularly played (and used AI) in online events outside of the ETC during the COVID era

New Comment

Submit

Rendering 78/79 comments, sorted by

top scoring

(show more) Click to highlight new comments since: Today at 5:04 AM

Some comments are truncated due to high volume. (⌘F to expand all)Change truncation settings

\[\-\]LawrenceC3mo7712

Really good piece, thanks for writing it. History of X posts like this one are unfortunately rare, and I’m glad you’re helping to fix this. The story you tell seems quite similar to what’s been happening in chess as well (including players memorizing long sequences of computer moves and then immediately floundering when out of prep, though it seems the case for chess play improving is stronger than for go, perhaps?).

---

I’ve seen a lot of the same ”not getting it” phenomenon you described while interacting with much younger people who did coding with weaker coding assistants (eg late 2024 era Cursor agents). People learned to rely on Sonnet 3.7 to generate code, once they ran into bugs that Sonnet couldn’t fix (often because of poor decisions made by Sonnet a few hours ago), they were stuck.

I see the same issue these days with ML research and Claude Opus/GPT-5.5: the models allow people to think they’ve thoroughly investigated the hypotheses under consideration without once looking at the data or code base with their own two eyes. Predictably, this leads to a lot of slop going through.

The main similarity between these coding examples and your go/math stories is that there’s a feeling ... (read more)

\[\-\]Ashe Vazquez Nuñez3mo404

Go is an interesting model organism for disempowerment because its practice has both technical and artistic/cultural components. Go AI is indeed disanalogous to coding LLMs from a technical perspective: the Go AI has vastly superhuman competence and the LLM does not. However, as you pointed out, Leela zero and other engines don't communicate off the board; moreover, their incredible strength actually makes them worse as game and review partners. This results in human-AI Go interactions that are usually hollow and vapid with respect to human-human ones, even if you factor out any cheating. AI is therefore bad at the cultural practice of Go and this shortcoming manifests in similar ways to those of old (maybe even current) coding assistants and LLMs as writers. People gravitate towards using the AI in all of these settings because it does a superficially "good enough" job at replacing them. However, in reality this offloading of cognition to the AI is illegitimate. In your example of coding assistants, the AI is not actually "good enough" on a technical level. At my school, the problem was that the valuable social and cultural exchange between players could not be replaced by their G... (read more)

4CronoDAS3mo

Using a computer to learn Chess or Go is like using a calculator to help you learn arithmetic. You can check your answers, but you can't directly see if your underlying algorithm for generating the answers is any good - it's like a textbook that has the answers but not the solutions to problems in the back of the book.

\[\-\]Ryan Meservey3mo4138

## AI users never find out they haven’t “got it”.

There's a certain genre of "educational" material that leads to similar outcomes as described in this article. Sometimes I enjoy outsourcing my thinking to YouTube channels like PBS SpaceTime and Veritasium and, if I'm not careful, I can fool myself into thinking I know more about quantum mechanics or gravity waves than I actually do.

It turns out that learning things for real is hard, and things that feel "comfortable" or "passive" should be met with skepticism for their actual educational value. The tricky thing is that we like being comfortable, and so we often inflate the educational value of such experiences.

5Fractalideation2mo

Indeed (another one). In these cases (and many others), I find the Stoic practice of 'voluntary discomfort' helpful (i.e. in this situation, getting through the voluntary discomfort of actively & effectively learning about the topic).

5CronoDAS3mo

Indeed. There's a big difference between knowing how a problem could be solved and actually being able to solve it. Sometimes the former actually is enough, though.

\[\-\]lc3mo3415

Hard not to think about the death of software engineering as a legitimate craft and discipline.

\[\-\]Will Duncan3mo136

My experience with AI software engineering, as someone who did without for over a decade, is that you stay up the abstraction layer for longer now. Before AI, over 60% of your time involved weird finicky edge-cases. Learning the interfaces of new libraries, automating a series of simple commands you had manually entered enough that converting the workflow would pay dividends later, conflicts between versions of libraries, conflicts between libraries and the language version, conflicts between operating systems. The was an incredible amount of busywork. Now, you spend a lot more time defining the problem, defining how the system will scale, trust boundaries for security, and more than anything, designing the architecture so it's maintainable and iterating on parts of the code that don't follow the architecture. Software engineering has essentially moved from involving tons of junior level learning, to primarily staff level work. Junior engineers are now prompting but without having the hard lessons from the past, so they can't see the problems they're introducing. This leads to modern codebases spiraling into chaos and invisible bugs are introduced even after iterating on fixes, and... (read more)

3CronoDAS3mo

The hard part of software engineering has always been figuring out the right requirements. Once you do that, writing source code is merely doing a particularly difficult type of compiling. ;)

9Mitchell\_Porter3mo

There are quite a few distinct intellectual crafts, in which large sections of cognitive labor can now be outsourced to AI: \ All forms of writing \ Visual art / animation / video \* Now, software engineering It would be amazing to have a true understanding of the transformation in each case, although it's all so vast and multifarious that it defies orderly description. But if we were trying to understand, maybe in each case we would look for prototype examples of: 1. how it was done before AI, 2. how it's done when you leave everything to the AI, 3. how it's done in intelligent, stylish, non-slop uses of AI. Because the third category still exists in each case. There are people using AI but still maintaining creativity, aesthetics, and professionalism, along with holdouts who don't use AI at all, and then the masses who are happy to use cheap and quick AI slop. That three-way division seems to be how it is, in any number of fields of endeavor now.

8Richard\_Kennaway3mo

I worry about that third category. I’ve recently had occasion to say that truck with AI rots the soul. I’m not as sure of that as my blunt statement (intentionally) suggests, but at present it appears to me well within the range of possibility. Whatever care one takes to compose the AI’s standing orders and one’s questions to it, and to never nod along to what it tells you, how sure are you that you are not slowly poisoning yourself? And even if one eschews their use, others won’t. We now have keep up our guard against all information sources.

\[\-\]Ashe Vazquez Nuñez3mo283

Short comment addressing some people remarkig similarities in my story to Chess culture: I agree that Chess shows many similar patterns of disempowerment, but they seem to be better off along one relevant axis. Chess websites have automated algorithms for detecting cheating and are not scared to punish people based on their outputs. This significantly dampens AI use in online Chess. This contrasts with Go, where cheating is rarely punished and occasionally can even be enabled. One story I didn't share in the main post is that the biggest Chinese Go server actually has a paid option to use AI in games inside the client. I have also seen students at Chinese Go schools being allowed to cheat during online practice games by teachers.

7MC\_Escherichia3mo

Which server has paid AI use? I play on Fox so I hope it's not Fox...

\[\-\]Ashe Vazquez Nuñez3mo100

I was indeed referring to Fox. I would have sworn there used to be a paid feature called "eagle-eye" (or something to that effect) that let you sneak a look at the AI evaluation of your live games. On the other hand, I can't find it anymore. I'm wondering if I hallucinated/misunderstood it or if it was discontinued.

2xpym3mo

Interesting. How do actual top players deal with this, have private competitions based on honor system?

\[\-\]Daniel Kokotajlo3mo2514

Thank you for this post, I found it helpful and interesting. I agree that this is a data point of evidence about how AI automation in other domains might go down.

2StanislavKrym3mo

Could you explain how the anti-automation described in the post prevent the effects described in the Intelligence Curse essay series? Go wasn't a necessary part of the economy, it was valuable because it was an inherited from of entertainment. Maybe you meant that automation of other domains would make people less smart?

\[\-\]Daniel Kokotajlo3mo178

Oh I don't think it would. Yeah a thing I was alluding to is that maybe in other domains, e.g. coding, there'll be a period where lots of people use AI for coding and they tell themselves that it's still them doing the coding when really they are kinda useless middle managers between their boss and Claude, and they tell themselves they are learning new languages etc. when really their coding skills are starting to atrophy.

\[\-\]Kaj\_Sotala3mo182

Similar energy:

So at the start of last semester, when I read two of my student’s stories for the first workshop and knew within their opening paragraphs that both stories were written by AI, I was hurt. I was also worried, because I realized that for the first time as a writing professor, I had to deal with students producing words without work \[...\]

As the first workshop started that night, I turned to the ostensible authors and told them I knew that AI wrote their stories. \[...\] For a few moments, all was quiet except the classroom’s ticking radiators. Then, a teary-eyed confession: one of the ostensible authors said she only used AI because she was scared of looking stupid, of being criticized for bad writing. She said she loved writing stories and hated having used AI. But she couldn’t stop herself, recounting a sequence similar to an addict’s descent: at first she fed her story into AI for a grammar check, it suggested line edits and she accepted, then it asked if she wanted structural edits, then it offered to rewrite the entire piece.

The other would-be author admitted he had never written a short story before and he had an idea but didn’t know where to start. I asked him why he didn’t reach out to me for help. He shrugged.

\[\-\]Vladimir\_Nesov3mo184

The tl;dr is that all improvement in the quality of play comes before move 60, when humans can mimic memorised AI policies. Play after move 60, in the pivotal parts of the game, shows no improvement.

I would expect on-policy distillation to be a more effective training process compared to playing against AI, or to using AI help when playing. That is, a human plays a complete game against another human (with neither of them consulting AI during the game), then goes back to review all of their own moves (or just those flagged by AI as particularly bad), comparing them to AI's suggestions for what those moves should've been, as well as looking at AI's estimates for how bad specific human moves were.

9Ashe Vazquez Nuñez3mo

This is generally agreed upon to be the "right" way to study with AI and Go players often pay lip service to it. In practice people's boundaries for what counts as on-policy distillation are not well-defined and that dillutes the impact. The mechanism whereby boundaries get weakened goes as follows: A player finishes a game and usually has a narrative of what happened, which mistakes were pivotal, and what they need to improve. The AI's evaluation will throw prediction error in the person's model. Since the AI is understood to be unquestionably correct, the person often 'has' to update to the AI's view. However, people can hack their their prediction-error sensors by retroactively updating what they think they believed in the past as well. This makes it extremely difficult to get useful feedback from the AI because all of it is 'obvious' or 'natural' after it is pointed out to you. The above pattern seems extremely common and I personally struggle to overcome it. The best method I have found is to make a concrete, written narrative of my games. This includes recording the exact moves I think are mistakes, why they are mistakes, and how much I expect the AI to dislike them. This creates a more well-defined boundary of what your opinions actually were that helps you maintain some epistemic integrity. I would do this regularly if I were still playing Go (semi)-professionally.

3Kaj\_Sotala3mo

If that happens, isn't a sign that the feedback has been useful? Even if the person doesn't remember their previous thinking, if they find the feedback obvious, presumably that's because they've updated their model to incorporate it. There's a similar thing in therapy where people will fix an emotional issue they used to have and then completely forget that they ever had the problem in the first place. This might make it seem like therapy was less effective (they can't remember any problems it solved!), when in fact it's a consequence of it having been very effective.

\[\-\]Jonah Burnstone-Cresswell3mo139

Feeling that something is obvious in hindsight is a bad predictor that your internal model has updated (hindsight bias). It's true that your reflective model of the game might update - was this a good move, was this a bad one - but that's easy because you know the AI is super-human and whatever it tells you must be true. This process is unlikely to have any effect on your future move-generation policy.

It's like the difference between verification and construction of mathematical proofs. It's usually trivial to verify a proof in comparison to actually constructing it. Verifying a proof isn't any indicator you learnt the necessary skills to construct one similar in the future. Arguably Go AI is even worse here because it gives an answer without a proof, so you can trivially recognise a move without learning how to generate it.

1philip\_b3mo

That's exactly the recommended (by Go players) way to review games with AI. Well, you're also supposed to try to figure out WHY the AI-suggested moves are better than the human's worse moves.

\[\-\]Declan Molony3mo\*154

On AI-use in writing

Upon telling a new friend that I write blog posts, he asked me if I ever use LLMs to assist my writing.

I answered, "No, I don't use it for any part of my writing process whatsoever."

"But why not? Wouldn't it make tedious things like checking for grammar a lot easier?"

"Why would I want that to be any easier? The reason I write is to challenge my brain to think more clearly. Even with grammar, were I to delegate that task to AI, I'm effectively saying, 'I no longer need to think about grammar or care about having my ideas flow smoothly.' But that's the whole point of writing: to polish something over and over and over until it reads like smooth butter."

In my post called "Will LLMs supplant the field of creative writing?", I wrote the following (which is, in my opinion, one of the coolest things my wet brain has ever come up with):

I wonder if we'll look back on the people (like me) who solely use their biological brains to produce writing and view them as luddites compared to everyone else using LLMs. Am I basically a grumpy old scribe complaining about the newfangled Gutenberg Press? Or will my steadfast refusal to let go of a fading art form be seen as the death

... (read more)

8CronoDAS3mo

I think Plato's surving writings contain a screed against written language, with one complaint being that you can't argue with a textbook and get a response back the way you can with a human teacher.

8Kaj\_Sotala3mo

\[Image: image.png\]

3CronoDAS3mo

How long would it have taken Socrates to learn "Don't read the comments section"? ;)

1small identity2mo

This is amazing.

7Declan Molony3mo

And now you can!

5Linch3mo

do you use spellcheck? Or do you turn it off on Google Docs/ MS Word/your favorite word checker?

2Declan Molony3mo

I keep spellcheck turned on. I don't think it significantly influences the way I write. I suppose, from a grammar perspective, spellcheck is the equivalent of using the bumpers while bowling—using them won't allow for any gutter balls (ie: egregious spelling errors), but they won't teach you how to be good at the game (ie: write a coherent essay). The AI tool Grammarly "goes beyond basic spellcheck to improve grammar, punctuation, clarity, tone, and conciseness." This would be like having bumpers turned on for bowling, and also having a slight auto-tracking feature inside of the bowling ball that curves the ball once thrown to promote strikes. Are you even bowling at that point? Or is the machine doing most of the work?

4NickH3mo

I think not using it for grammar checking is going too far. If you follow that reasoning logically then human editors, proofreaders and reviewers should also be shunned.

8Declan Molony3mo

For me it comes down to temptation. It's so easy to ask an LLM to check my grammar. But then maybe also check my wording for specific paragraphs, and word choice, and checking my ideas for an essay... Until eventually I'm exporting some of my thinking to it which is making me a lazier writer. A human editor simply doesn't have the time to rewrite everything for me. And humans aren't instantly available all the time like LLMs. If you have the self-control to draw a clear line in the sand for LLM use and abide to it, more power to you. For me, however, LLMs are too tempting and I know that if I allowed an inch that I would take a mile.

2ChristianKl3mo

Installing grammarly is about letting an LLM check your grammar. I don't think it takes much self control to keep that separate from Chatbots.

2Wenngle3mo

I think we're drifting a bit in this discussion. Grammarly as a product contains an AI assistant and will rewrite your sentences automatically (I haven't used grammarly, but it looks to be the case). That's more like a chatbot than traditional spell check or a rule-based grammar checker.

2ChristianKl3mo

The question is what done by an LLM. Grammarly is LLM powered. LLMs are a specific technology. The term is not synonymous with chatbot. If someone want to claim that you don't use LLMs but they actually do, that matters.

\[\-\]Raemon3mo121

Curated. This was a fairly interesting case study. The particular concept of "people think of themselves as learning, when they're not" was something I haven't seen discussed before. The general feeling of "I have agency, the AI is just helping me" was more familiar but still good to explore

I'm pretty worried about how this sort of thing shakes out on a civilizational level.

2StanislavKrym3mo

On a civilisational level? It looks like there are three meanings: 1. OpenBrain's researchers might be fooled into believing that they understand what the AI is doing when they don't actually understand it. 2. The ASI could end up aligned and the USERS could fool themselves into pseudo-understanding everything and make stupid decisions. But why would the ASI decide not to warn the users? Why are the users' decisions necessary? How high do the users to be in order to screw everything and how does this interact with the Intelligence Curse/Deep Utopia where everyone loses their jobs and the world has "just" a wildly different power distribution? 3. Or does the ASI simply erode the agency of ordinary humans, as did the fictional Whispering Earring? But how does one human with eroded agency destroy others' agency? That being said, the third point was observed in practice to erode the agency and capabilities of individuals.

\[\-\]Denis Weber3mo9\-15

While the deep dive into Carlo Metta felt a bit like a personal crusade at times, I really appreciated how you humanized the 'cheaters' as being driven by curiosity and laziness rather than malice. That said, I think it’s worth noting that we’ve always outsourced our autonomy to pros or Joseki wikis—AI is just the newest iteration of a very old human habit.

6Ashe Vazquez Nuñez3mo

I hesitated to publish this for exactly that reason. I was drawn to using this case as an example because I do actually think it affected the way AI use is perceived and handled in European/American Go culture for the worse. However, the topic is pretty sensitive among Go players and this makes it hard to discuss without eliciting monkey-politics-brain sentiments from everyone (inlcuding me as the writer). I ended up addressing the piece mostly to the AI crowd and chose not to widely publicise it among Go players. I would be curious to know how I could brought up the example in a more tasteful way that wouldn't have given the impression you describe. ---------------------------------------- On outsourcing of autonomy: I think there is a meaningful difference between the other examples you gave and Go AI. I agree that humans outsource their cognition to things all the time. I would call artefacts like my personal notes and my anki deck part of my extended mind. Extending minds is great, despite the perpetual risk of self-disempowerment it entails. However, most delegation used to happen between humans. AI has reached near-superhuman (or higher) level at many tasks that are a key part of how we share culture and resources (e.g. writing, code). This seems unusually dangerous because cultural, economic, and political power is slowly being transferred to increasingly intelligent entities that are unlikely to be aligned with human interests.

\[\-\]Kaj\_Sotala3mo1114

For what it's worth, it didn't read as a crusade or untasteful to me.

5Denis K.3mo

Just curious: if you were hesitant about this aspect, why give the name of Carlo Metta ? It seems to me that anonymizing it would not hurt the story and would avoid this potential crusade effect.

\[\-\]Yulia2mo80

Complementary reading: Notes on Chess and AI. Or How I learned to Stop Worrying and Love the Bomb, written by a top chess player. Ashe takes on the perspective of a teacher invested in amateurs improving their understanding of Go. The author of the chess post writes as a top competitor singularly focused on winning.

Notable quotes:

“The goal of playing chess is not to create art, not to show understanding, not to think or display strategical brilliance. The goal of playing chess is to win chess games. The top chess players use whatever resources they can, i... (read more)

\[\-\]CronoDAS3mo70

Does "centaur" Go play do any better than unassisted AI these days? Because there was that exploitable bug in the algorithms that Go neural networks learned to tell whether groups were alive or dead.

\-1David Joshua Sartor2mo

My understanding is that centaurs beat unassisted Go AIs at this time. Not sure if they do noticeably better against humans.

\[\-\]Mart\_Korz3mo72

People consistently underestimate just how lost they will be when the solution is no longer right in front of them.

Important and true. In many cases where I felt confused about how people (myself absolutely included) behave highly irrationally, my conclusion was that many of us are desperately grasping for certainty much of the time and on most topics. Noticing confusion only feels like an option where one's thinking feels mostly stable. Reality has a surprising amount of detail, we live in a world full of agents that are just as smart as ourselves and wh... (read more)

\[\-\]Ephraiem Sarabamoun3mo52

I have seen a softer version of this in software engineering. Even though in the case of software engineering, using AI would not count as "cheating", still, many engineers will actively downplay the degree to which they rely on models for their work. I also noticed a general reluctance to acknowledge when models perform well and an eagerness to highlight when models perform poorly. I think this is partially driven by job insecurity (If you acknowledge that a model is better at coding than you are, then why should a company keep you on the payroll) and par... (read more)

3ChristianKl3mo

There were some studies last year that suggested engineers thought that they were getting faster through using models when they actually got slower, this suggests at least at that time people did attribute too much instead to little to them.

3Ephraiem Sarabamoun3mo

Hey Christian, thank you for your thoughtful comment. Yes, I remember also hearing about such studies, and you are correct that such studies would imply that engineers are more bullish on the usefulness of models than is merited which is difficult to reconcile with the general tone of my comment (suggesting that engineers around me seem to underplay how much they themselves use models and downplay the capabilities of models) I can think of a few interesting suggestions though that might be insightful: 1. Maybe the effects that I have observed are not representative of the field as a whole. I have noticed that junior developers and startup founders are generally much more open about their LLM usage and much more bullish on model capabilities than senior developers at large companies. 2. It is possible this is a social effect. I.e engineers today do think that LLMs are capable but downplay this fact in public. If, for example, the studies that you site are based on anonymous feedback then that discrepancy would be consistent with this effect. Engineers will reply anonymously that they are more productive with models but publicly will downplay this. 3. Models have gotten a lot better. I would be interested to see if those studies would yield the same results today as they did then. I personally would agree that a year ago models were probably a net negative on productivity especially for senior developers (they would introduce more subtle bugs that are a pain to debug). Today though, I think they are certainly a net productivity gain, and it would be interesting to see if developers today are undervaluing the productivity gains from models. These are just a few thoughts I had. I would be interesting in hearing what you think about this. Ultimately this is only my experience, I would be interested to know what others' experiences are

\[\-\]Shankar Sivarajan3mo50

There was a paper I read a few years ago where they said if they played a really unusual strategy – not a good one; I think it was one a human would have recognized as a blunder – it was out of distribution enough that the AI gets confused and loses. Whatever came of that?

\[\-\]gwern3mo180

Well, the obvious thing to do is to check the reverse citations. Or just ask a LLM: <https://chatgpt.com/share/69f58633-01b4-83e8-b3b1-de42d3d196c9>

FWIW, my understanding was that individual attacks could be fixed by further training or architectural tweaks, but you could still find new attacks and so the basic problem of adversarial robustness in DRL agents was nowhere close to being solved. The GPT-5.5 Pro Deep Research report says something similar. It looks like the best ref would be <https://www.reddit.com/r/baduk/comments/14prv4f/katago%5Fshould%5Fbe%5Fpartially%5Fresistant%5Fto%5Fcyclic/> \+ <https://gomagic.org/david-wu-on-building-katago/#h-the-circular-group-problem-where-bots-still-misjudge-go>

7Shankar Sivarajan3mo

I guess this is karma for me ever having replied to a question with a link to lmgtfy\[1\]. I thought it would be clear from context that what I was asking for was a first-hand account of how (and whether) such adversarial strategies, which I read are simple enough to be possible to learn and implement unaided over-the-board, had impacted play in these no-stakes Go schools. 1. ^ In my defense, that was like fifteen years ago, back when Google still reliably answered questions.

\[\-\]gwern3mo136

I don't think that was clear at all. Personally, I thought the question was a sensible one on its own, and something I had wondered myself, and that's why I took the time to look it up for you rather than downvote what looked like laziness - 'whatever happened to that KataGo adversarial attack research, anyway? I haven't heard about it in a while. Surely it hasn't been fixed? I would've heard about it, I think, given how DRL agents are so fragile in general, that a robust fix to adversarial attacks in any DRL setting ought to be big news. But what's the current state of play?'

But I have never seen anyone mention seeing someone go to the length of memorizing anti-KataGo strategies or deploying them 'the real world', aside from the documented example in this KG line of research of someone doing so just to prove that the circling hack can be deployed by a real human player against a live bot and is not intractable in practice (as many adversarial examples are very fragile or require near-superhuman capabilities to deploy correctly).

I would be shocked if anyone was doing so given that it's a lot of work to win games against a few specific obsolete versions of one specific Go agent (the... (read more)

1Isaac Newton2mo

I don't think there is anyone using this to counter cheaters, but people do it for fun, and there is definitely more than one example. Here is a video where Nick Sibicky beats katago with 10k playouts. Here is a game on ogs where a player captures the entire board against katago micro.

\[\-\]Тернай3mo40

Just today I read a post dedicated to the fact that most of the modern tourist routes imply that the tour guide in them, as the author put it, "the producer of interpassivity".

In popular tourist locations, the tour guide usually tells you what you see, and then - what you feel about it. "With each room you are more and more admired", "this act is respected", "the light in this room attracts all your attention". This is less common in academic museums. There the tour guide-researcher is absorbed in his own emotions on the subject. He can impose his visio

... (read more)

1Baldassare Castiglione1mo

Can you link to this post?

\[\-\]Philip Niewold3mo41

Thanks, this was an interesting insight in world I don't know enough about! As far as I understand games play an important role in establishing status (and sometimes dominance), which is pretty-much hardwired into many of us. So even without monetary gains, this is a big incentive. Admiration or respect from your peers, the knowledge that you beat someone else, your ranking going up. Even though A.I. disempowered them on the level of Go skills, it empowered them with regards to status. And ultimately, status trumps Go skill for most people. Personalities th... (read more)

\[\-\]Gordon Seidoh Worley3mo4\-2

I was just thinking about how this pattern might apply to software engineering, and I'm starting to suspect that it largely doesn't.

Here's my thinking. I use AI a lot to do things like brainstorm solutions. Rather than me sitting around trying to think of ways to solve a problem, I describe the problem to an AI, and get it to give me ways it thinks the problem might be solved. Now, I don't always take one of its options, but the process of asking it is usually enough to get me to think of how I want to solve the problem, and sometimes I get lucky and it pr... (read more)

3Ian McKenzie3mo

But here, the AI also plays the role of the junior engineer – you don't actually have to engage with the details of the chosen solution, and by doing so absorb the understanding of why it's the best approach. You can just say "yeah do that".

\[\-\]XelaP3mo40

I liked the post, and found the discussion of the Go world interesting to someone who knows diddly squat about it.

But there's a point you bring up a few times, sorta implicitly, which I wish you argued more for: that the pervasive cheating in games delegates the culture to the AIs, and that this is bad.

What does this even mean? Are the 'cultural' aspects of Go distinct from the entertainment value of playing games, watching games, player drama, and possibly player strategizing (e.g. if Alice always plays the X opener and has Y style, while Bob has X' and... (read more)

\[\-\]LawrenceC3mo160

In Chess, cheating is rampant not at the top professional level (probably) but at the level just below that — iirc there’s a lot of IMs banned for cheating on titled tuesday on chess.com? At least, many of the top players believe that cheating is rampant on online chess (though not amongst top players), and a lot of casual tournaments (eg between streamers) have had people get caught just aping stockfish. And there’s definitely a lot of accusations thrown around for online chess cheating that are generally considered unsubstantiated (the former world champion Kramnik being the most famous serial accuser).

Online chess tournaments not having rampant cheating seems to match the stuff Ashe is saying in their post:

The symbolic camera controls – which would be easy to circumvent for a dedicated cheater – seemed sufficient to curb almost all cheating in a way that threats or impotent references to “fair-play committees” were failing to.

when you add actual barriers to cheating, even if they‘re circumventable, cheating rates drop a lot, especially at the top level.

Of the factors you mention, I’m not sure how FIDE’s willingness to ban compares to Go organizations such as IGF or EGF. Plausi... (read more)

\[\-\]Isaac Newton2mo30

The tl;dr is that all improvement in the quality of play comes before move 60, when humans can mimic memorised AI policies. Play after move 60, in the pivotal parts of the game, shows no improvement. For me to think there’s any meaningful change in human play from pre-AI times, I would have to be convinced that players understand the AI moves they copy well enough to keep a heightened level when they go off-policy after the opening. There is no evidence of this.

An alternate hypothesis is that the first 60 moves are where AI has generated the most human-acc... (read more)

\[\-\]RickMcCoy3mo30

commonly understood to be the world’s strongest player at the time

Small nitpick - Lee was not the strongest player at the time. According to Gorating's history page on 2016-01-01, Lee was the 4th best player at the time, neck-and-neck with Mi Yuting and Shi Yue. You might be thinking of Ke Jie, who was the strongest player at the time and also battled Alphago the next year.

\[\-\]Knight Lee3mo20

It reminds me of all the rulers in history who had sycophantic advisors do all their work for them, but still believed they were the brains of the operation. If things got bad, they might take personal control only to worsen things further.

\[\-\]E.G. Blee-Goldman3mo2\-1

Fascinating, but even pre-AI, playing on Tygem you'd sometimes get a 2k suddenly playing like a 4D, and whether it was a friend-assist or Baduk school demo, honest cognition was hard to come by at times even then.

\[\-\]cubefox3mo2\-5

It seems likely to me that nothing here was substantially different from what happened to chess twenty years prior, so this situation doesn't look exactly new.

4cubefox3mo

For the downvoters: what is the fundamental difference between having access to a superhuman chess bot 20 years ago and having access to a superhuman go bot today?

\[\-\]Ryan Galliher1mo10

This piece really strikes a cord with me, particularly the portion on the test retake.

At some point during my American education, I evolved from someone who really enjoyed learning and working through hard problems to true understanding. However, by the end of high school, this joy was ground to a pulp by the desire to get an A at all costs. Colleges were concerned with GPA and test scores, not true understanding.

\[\-\]hold\_my\_fish2mo\*10

This was an interesting read, but it felt incomplete in the way it described the harm of using AI to cheat. Disempowerment isn't the main story here.

The primary reason that it's wrong to cheat in a recreational game of Go is that it's rude to your opponent. If the other player wanted to play against AI, they could do that themselves--usually, they're playing against a human because that's what they prefer to do. It's dishonest to present yourself as a human player (which is what they wanted) and then play AI moves (which is not what they wanted).

It's the s... (read more)

\[\-\]ModelVoyager2mo10

I feel pessimistic about the future on this front.

I think agency can mean two different things. One is the sense of agency (the phenomenology). The other is an abstract concept like "choices made with understanding". For someone who cares about the second, the two can come apart. A person can feel agency without actually exercising the second kind, and people in this camp would see that feeling as an illusion. They learn to check whether their sense of agency reflects the concept. I am closer to this camp. And I feel pessimistic because I believe this type... (read more)

\[\-\]Matt Vincent3mo10

I appreciate the general point about fooling ourselves into thinking that we know more than we do, and I have a supporting anecdote.

As a kid, I was a very gifted math student. Although I wasn't averse to memorizing formulas, I always strove for understanding. Consequently, when I learned about Taylor series, I was shocked to realize that I'd spent the last three years mistakenly believing that I understood trig functions. It hadn't really occurred to me, at least not since the early days of my algebra-2/trig class, that if I didn't have the scientific calc... (read more)

\[\-\]CRISPY3mo10

Skipping to the destination at the expense of the journey is a common problem. People want a degree to make money, many care little for the education. People want to win the game, not learn to play it. Many people want the big house and flash cars, but don’t want to put in the work.

Go and chess are the sophisticated examples, but spend enough time in airports and you’ll see people using AI and anagram generators to play NYT word games. Sudoku solvers and crossword engines fuel commuter trains. Geocaching has evolved into a puzzle game where participants s... (read more)

\[\-\]Ashwir Biyani3mo10

Loved reading this. I always felt the dichotomy between whether you go for the efficient path or your own, especially given the worlds best with his own path is not able to defeat AI. AI offers an comparatively easy path towards reaching excellence (although in this case excellence is arguable). I read another article on the state of GO and from what I recall with AI in the picture there is a surge in the number of better female players majorly owing to being able to learn and interact with AI. Which is a great thing, but at the same time, I feel AI limits... (read more)

\[\-\]Hua Yu3mo10

While the game of Go is a good model organism to study, I feel it's too strong to call it disempowerment. Go players are trying to adapt to the AGI world, which is not easy. What's one to do when there is a very strong player, whose move is readily available, while in the mean time hard to grasp? We mimic its play. This has always been the case, even before the arrival of AlphaGo. Some master came up with a new opening and started to win many games. Others adopt this opening, mostly without "understanding" it.

Fake it until you make it. A bit cliche, but it goes a long way.

How's this relevant for the disempowerment discussion? I'm not sure. There is hope as long as people are trying

\[\-\]Mikhail Samin3mo10

It‘s always been strange and sad to me when people aren’t taught to grasp the math and instead learn to just apply rules they can’t rederive to solving problems, so I can empathize with that part of the post.

I don’t really play go, or chess, but at some point, Manifold was trying to play chess against someone, and I tried to play through consequences of various moves with Stockfish (using stockfish was explicitly allowed); anytime stockfish recommended a move I’d try different ones and see if they’re good or not and why; and it was also pretty interesting ... (read more)

4ChristianKl3mo

The key difference between chess and go is that the merit of a chess move is much more easy to see five moves down.

\[\-\]Egemen Tunçarslan2mo\*\-20

The detail that stuck with me is that camera controls worked where 'fair-play committees' didn't. That maps onto something I keep seeing from a governance angle: the deterrent isn't the rule, it's whether a functioning mechanism sits behind it. The Metta case basically established that the rule existed but the enforcement didn't, and once that was common knowledge, cheating became the default not because norms collapsed, but because the cost of enforcement (social backlash, appeals, organizer futility) exceeded the cost of looking away.

What I find genuinel... (read more)

Moderation Log

737

737

AI users never find out they haven’t “got it”.

Appendix A: No, Go players aren’t getting stronger

Appendix B: Why this article exists

Comments (0)