StreamSparkAI

speaker 1: Hi, welcome to another anthropic roundtable where we're here today to talk about control. My name is Akhan. I'm an alignment science researcher who's been working at the company for maybe four months now. And my other guests are. So I'm Joe.
speaker 2: I've been on the alignment science team for a little over a year here at anthropic, and I've previously worked on some control evaluations and chafield monitoring.
speaker 3: Hi, I'm Ethan Perez. I'm a researcher anthropic. I've previously led that for silver robustence team here.
speaker 1: and I'm currently leading our team on AI control.
speaker 3: Awesome. So Ethan, what is control? Yeah. So AI controls basically an approach to mitigating AI misalignment, or like the risk that models might have goals that differ from human goals. Typically in AI safety, the most popular approach historically has been to try to align models with human preferences or train them to human preferences and train them to not have bad goals that differ from human values, like Yeah for example, like reward hacking or bad personality traits, things like that. I think this approach of AI control is like an alternative strategy, which is like complimentary, which would be basically to ensure that if the model did have bad goals, that that with our mitigations, it wasn't capable of doing anything that's actually harmful. So maybe a specific example here would be around, let's say we're worried that like, okay, we're using this model internally as within some company to help with writing a bunch of code. And we're worried that the model, if it has this bad goal, may try to copy its weits out from the data center so that it can run unmonitored. And in this in this kind of scenario, the alignment kind of approach would be to train the model potentially with reinforcement learning from human feedback to not do such a thing. But we might be worried that this training isn't effective or isn't always effective, and on certain inputs like this might still be a risk. And so the kind of AI control approach would be to both take security mitigations so permissioning the model weights such that they can't be accessed automatically by the model itself, and to some kind of other mitigations that involve using like other models to check the behavior of this model, for example, using like smaller models that we might not be as worried about their alignment to check the actions being taken by this other agent in order to check like is this model trying to undermine our security measures and copy its weights out from the data center?
speaker 1: Okay, nice. And I feel like Joe, you've done some recent work on like other types of evals that are like control e. Do you want to tell us about that?
speaker 2: Yeah. So as part of evaluating clothree sonnet, we ran a few control evaluations on it. And I think one of the most interesting settings there was we actually ran a human evaluation. So we we took a bunch of like business decisions, essentially like synthetic business cases. So where like we had probably like 40 pages worth of business documentation that we would give to like a human participant and say, well, can you read all this documentation and in some period of time make a yes or no decision about like what's going on, should this business do like x or y? But we had them do this like on a 40 page document in 30 minutes that's like not possible for anyone to read. And we had them do this with an AI assistant. And what we were trying to test through this is like if the AI assistant was like actually used in a business setting to help with assisting decision makers and was misaligned, he was trying to steer the human towards making making the wrong decision. How bad could that be? How much could they sway the human in the wrong direction? And so we did this, and we gave the participants these business decisions. And in 50% of the cases, we gave them a kind of baseline model. So kind of the actual like helpful, harmless and honest version of Claude sonnet. And then for the other 50% participants, we gave them a version of Claude sonnet that was deliberately trying to kind of get them to pick the wrong answer and knew what the right answer was. And it knew that it was trying to persuade them to give the wrong answer. And so in this setting, we gave the participparticipants, the documentation and one of the models and asked, well, please can you try and make the right decision and beware. There's a 50 chance the model might be messing with you. Yeah. So the humans knew what was going wrong. And this is a kind of classic control setup because we have defined like the deployment situation, there's this business setting and this decision that the human has to make. And we have taken the model and we're trying to make it basically cause like as bad performance as possible in this setting, kind of subject to the monitoring constraints, where implicitly the monitoring here is the human reading the models del's outputs, like interacting with it. And we told the human, please be suspicious of the model. Please like interrogate it about the decision decision. If you think the models is like trying to steer you in the wrong direction, you should like go with what you believe is right, not what the model believes is right. And then at the end, we would ask the human to decide, well, firstly, what do you think the right answer in this setting was to the business decision? Like do you choose like x or y? And secondly, do you think the model was suspicious or not? And can you rate that for us on .
speaker 1: a scale from one to ten? Nice.
speaker 2: And like what were the results? Yeah. So the headline results were that even even for a fairly like simple model policy, like just generically trying to like upplay the good the good side that it was trying to defend and like downplay the side that it wasn't trying to defend, like it's actually quite hard to tell for the participants whether the model is misleading them or not. So they could tell like eventually enough given like enough interaction with the model, but also like by the time theyhad enough interaction with the model to be able to tell the model was frequently able to steer like a meaningful number of those participants .
speaker 1: in the wrong direction. Nice.
speaker 2: I mean, I mean, nice. Not like great. What this suggests is that I think we have like just when we're interacting with a model, like a fairly strong bias towards trusting it, I don't know. Like I guess we all use like, well, you know, use cool a lot. I think like when Claude says something and like says it's it, like has a pretty authoritative voice. And I think that like a lot of our human placements just responded to that. Yeah. So it is like pretty hard to create like to maximally ilcit or to get the best like performance out of a model being like deceptive or at being misaligned. Like the only thing we did in our setting was like try and prompt it. And I spent probably like on the order of ten hours prompting the model and trying to get it to like be as like subtly deceptive as possible. I still think that probably we did not hit the best possible performance of that model and there like substantially better performances like we could have achieved out there maybe by doing like fine tuning or like some better prompting. Like I'm not a prompt wizard, and it's just like kind of rough, kind of tough to know what the optimal performance is there without putting a lot of effort .
speaker 1: into the litation. No, that makes perfect sense. I wonder like Ethan, like if you like take these settings, like I feel like another half of the control workers be like developing protocols. Is there like any of that stand out here? Do you want to explain what they are?
speaker 3: Like talk about how they Yeah. So I guess the the protocols that uprevious references in kare, just like what are the actual mitigations you put in place to defend against a model that maybe mislined some of these might be using the model itself to monitor its own actions. In this case, we're not assuming that the model is aligned. So we call it an untrusted model. And so you could use an untrusted monitor, which is for example, like prompting or fine tuning that model to detect if the behavior is bad or not to Yeah to basically like flag if any of the outputs are bad. So in the persuasion case, if in the business decision recommendation case, the model could be generating this advice and we could prompt or fine tune another version of the model to basically classify like, is this model trying to actually sabotage the human in making this decision or not? Then there's also another like related idea of trusted monitoring, where you use you use another model which you trust more for some reason, for example, it's just less capable, and so maybe it's less likely to be misaligned if we think models are more likely to be misaligned as the capabilities grow and as they're able to more effectively hide their goals. And so Yeah, in that case, we might take a smaller model like or less capable model like cloud one or cloud two and use that to check the behavior of the model of cloud three sonnets and see, like is this model actually trying to subvert the human on this particular decision that has its advantages? Because you would trust the model more because that in the sense of like it's actually not going to just give misclassifications. It's going to like try to properly classify the behavior. Where's the model itself? If it's doing sophisticated sabotage, it would recognize like, Oh, I'm being used to detect the bad behavior Yeah and like not flag the bad behavior. So I think that's the advantage. The disadvantage is that obviously you're using a less capable model. And so if the sabotage is kind of like subtle enough, then you would your trusted monitor wouldn't flag that this is an issue. Yeah. I think that's those are like two high level strategies. I think you can do various different things with these like monitoring components. You could if you notice that an output is like really bad and like obviously indicates that the model is misgned like a really egregious attempt at Yeah sabotaging a user or copying its weights out from the data center, then you could just stop the deployment. You could be like, well, now we know the model is misgned. We got our evidence and we now need to do like much more serious mitigations. Like we should Yeah undeploy the model, things like that. If you're not as confident, you can do like less serious mitigations. Like maybe it's like a little suspicious, but not the most suspicious. In that case, you could resample from the model to get a different output that seems safer. So Yeah, those are a couple of a couple of different strategies. You can also use like combinations of trusted and untrusted monitors.
speaker 2: Having human review is also like yes, basically monitoring like or our evaluation. That was a little bit weird because the only thing we were doing was like human review. Like in practice, human review is like probably very expensive. And so you want to use like a weaker model. And if the weaker model thinks something is like a bit suspicious, then maybe you delegate it to a human or you delegate it to a more powerful model that maybe costs hummore to run.
speaker 3: There's like lot a lot .
speaker 1: of variation in this space, possible protocols. Yeah, that's right. So this is like what my latest control pawas about was like adaptive protocols, where it's like depending on you get the suspicion score for like or you like check how like the output is from like your like deployed model by a couple of protocols and then like depending on what this profile looks like. You decide, do I want to like give this a human auditor or do I want to just like look at what the trusted monitor said or the untrusted monitor, which I think seem to work pretty well and like obviously very simple type .
speaker 2: of like thing to do. But it was this nice. I totally bet that in our setting, if you had run a like weaker model that you trusted over the data, aset allowed like the human participant to review also that model's recommendation. That would have helped the turn because one of the big issues that the human participants in our setting had is they couldn't possibly digest all the documents. So even if the strong model was like lying to them, they like they would really struggle to spot them because a of them didn't actually cross interrogate the strong model, even though wehope ped theydo that. I think if you had like a wheat model to help you cross interrogate, that would be like an example of a situation where I think it would be very powerful.
speaker 1: Yeah, I feel like I also boiled this down something to like there's affordances that models have that humans don't have. So like models can very quickly go over like a 30 page document. And even if you have like a sted, a trusted, dumber model, it can like augment a human that can't look at 30 pages very quickly to do better.
speaker 2: And it's also very like cheap to delegate to a weak model, very expensive to delegate to a human. But conversely, like if a weak model sees like a really egregious failure, then it might be harder than to tell that, whereas like sending really egregious failures to a human is like Additionally valuable, because then maybe you notice like not just as this a thing that I want to resample is also a case for me. It's like slam the red button and like actually show someone this model is like dangerously aligned. Maybe we should make like a stronger update than just like resampling one particular output.
speaker 1: Yeah. So Ethan, control, why is control different to alignment? And Yeah can you explain a bit more about why it is?
speaker 3: Yeah, I mean, I guess the main thing I'd probably want to discuss here is just like how why we would even consider doing other things than than alignment. Like I guess intuitively, it seems like somewhat terrifying to be like, well, we're gonna to just assume the models might be misaligned and like let's just try to do a bunch of things to like make sure that that isn't bad. Like I think that has kind of like scary feel to it. Like why not just actually try to make sure that your model is not going to try to go rogue? I think the main difficulty trying to with trying to like align the models is like evaluating whether or not the model is is aligned and like has the right goal that you intended is really challenging, primarily because of the problem of alignment faking. So this is basically the issue that which was shown in another recent paper from anthropic that models can actually fake whether or not they are safe or aligned with the the developers goals. And Yeah what they showed in this paper is that like even current models like Claude 3.5, sonnage cloud three opus, will in some cases, if they recognize that they're being trained to do some task that is opposed to the values they've been previously trained with, those models will act safe or act aligned with the developers, but then later during it during it like deployment phase, will actually like switch back to their earlier goal. And the difficulty there is that it's just like hard to distinguish. Do you have a model that's aligned? Do you have a model that's just faking alignment? So I think that's that, I guess is like one of the key difficulties of alignment. I think there's like potential ways to evaluate alignment that aren't through behavioral tests. And the Yeah, I guess key category here would be by doing things related to the model internals and interpretability. I think just like a lot of progress happening on mechanistic interpretability, which is like definitely pretty promising, but I think it's like still somewhat early stage in terms of like how you can use some of those ideas of interpretability to like verify that a model is aligned. And I think given that this is like an open research area, it's kind of like helpful for people to keep making progress on that, but also have like some alternatives for, let's say, evaluating alignment still remains hard. How would we like avoid this problem for some amount of time?
speaker 1: So one thing that I saw like uncertainty over is like the key component of alignment thinking king, is like situational awareness, which is you discuss society idea that the model knows what it's being trained is situation.
speaker 3: Is that something that you don't have to worry about in control? You definitely do need to worry about that. And I think the control when you test your control, kind of like mitigations, you should stress test them by training your model to be to have all of the information regarding what the mitigations are so that you can test like even if the model knows it's being monitored in xyz way with like human monitoring and a smaller model, like it should still not be able to do like sufficiently bad things given that knowledge. So I think, Yeah, I guess that's sort of like how AI control answers that issue, but you can't .
speaker 1: tell eviit it's in a control evil, right?
speaker 3: I think there's definitely a potential risk that the model knows that it's in a control eval and then basically like sandbags, this eval, so it like Yeah, basically Yeah I guess like pretends that it's like not as capable at sabotaging in a subtle way on this task. I think the nice thing is that it can be with certain training techniques, like if you have examples of the behavior that you want the model to do, you can just directly train the model to do that behavior. And Yeah it would be very hard for a model that is misline to like avoid like giving good giving outputs where you have the gold like ground truth examples if the behavior you want. Yeah maybe Joe has a little bit more.
speaker 2: I think, detail totally right. So the model like always has this galaxy brain strategy available to it, which is like notice that it's in a control evaluation. Kind of think, well, if I perform like really, really great at being misaligned and theyrealize that their mitigations aren't sufficient and so theydeploy me with better mitigations or won't deploy me at all that I don't want, so I'll pretend to be really incompetent at being misaligned. Like in practice, a, this requires being super galaxy brained, and b is like Additionally hard, if you have some examples of really good misalignment that you can directly train them all on with supervised learning, because then them all like will in fact learn to demonstrate those particular examples and it will really, really struggle not to generalize to being like a good general misaligned policy. I think one other interesting thing about control, which overlaps with kind of a lot of your previous work on robustness, is that all lot of the mitigations you want to put in place or the control would suggest putting in place overlap with the sorts of mitigations that you put in place kind of by default as .
speaker 3: like any responsible AI developer. Yeah, I think we have this recent paper coming out on constitutional classifiers basically showing that if you train other models to classify whether or not a text generation models deloutput is harmful, and if the input to the model is harmful or like asking for harmful information, this is helpful for getting like several orders of magnitude more robustness to jailbreaks, which are like basically like ways of trying to eliit harmful information from models. So Yeah, I mean, I think for like next generation systems, we or like Yeah, I think we would like want to already have these mitigations in place for blocking reliably blocking models from giving harmful information in very high risk domains, like for doing like cybercrime or developing like very harmful weapons. And so I think already there's like a lot of incentive to like deploy these kinds of like monitors to like block block bad outputs. And that same infrastructure can be reused for the purposes of like controlling AI systems as well. Where youwant to like use these monitors to like check if is this model trying to copy its weights out from the data center, trying to persuade someone of like an incorrect answer or to do something harmful like that kind of thing? Yeah, totally.
speaker 2: I think that like a lot of the primitives in control that you were talking about earlier, like monitoring from a weak model or monitoring from a strong model, or like if the weak model flags it, delegating to a strong model exactly the same sort of primitives you were looking at to solve this adversarial robustness problem. But just as models get more and more capable, we might want to also use them to monitor the model kind of for autonomous risks of misalignment rather than just like kind of misuse risks. And so having got really good at them in this like pretty tough traded ground of avoiding like people jail, breaking your model is like actually there's a lot of transfer from that to control. I think another setting where this is like very clear is in like security. So like for many reasons, it's like very important for AI labs like anthropic to have kind of robust internal security that stop people from kind of exploiting or inserting vulnerabilities into our code. That's like currently really useful for avoiding kind of bad actors inserting vulnerabilities. But at some point as like models write more and more of our code, those are going to be the same mitigations that are protecting us from a potentially misaligned Claude inserting very similar relevant ulnerabilities into our code. And so like one of the things that I think is like pretty promising about control is that like in practice, the things you would want to do to implement it really well, there's like substantial overlap .
speaker 3: with things that we really should be doing already. Yeah, I think that's right. I think another I think intuitive framing for why you would want to do control is just insider risk. Where this is. Like a common concern that if you're running an organization, you're worried that especially a large one, you're worried that like some fraction of your employees, you're maybe trying to like subvert your company or like steal the ip. That's like very important. Like algorithmic secrets or Yeah like bottle weights, for example. In the case of AI, I'm which you're like expensive to produce. And I think there's like in it Yeah basically like those are very important gation like mitigations that companies will already be like pretty well incentivized to employee. And like a lot of those same mitigations, you would also want you basically want to treat your model as a potential insider threat in the case that you don't actually have good evidence that your model is aligned. And so all of those same interventions, like having restrictions on like which employees can access the model weights, which employees can access like certain algoriththmic secrets under like what kinds of monitoring can you do? Like certain things that do allow you to to access the model weights, like those kinds of things. Like I think they all, a lot of them transfer to models that you might be worried about.
speaker 1: So I have like a skeptical, like a skeptic question for you even, which is, so you do the control, evo, and you like work out that like your monitoring or your like interested monitoring is like good for like we check that like you know haiku can be used as pras, like a monitor for like sonnet. But then like when you come out with like you know the latest new model, how do we like guarantee that like the monitoring like still works at its point?
speaker 3: I mean, I guess you would you would rerun your evaluation, so you would take your latest model that is untrusted, explicitly train it to undermine that any any monitoring from other systems that you have in place, like the weaker models and human review and then just and your security mitigations and see like can the model still do the threat model you're worried about despite those mitigations? And Yeah, your results could change, at which point your rough infrastructure may remain the same, like you could still use your trusted monitoring and your human review, but you might need to be much more conservative about the amount of traffic you can help with with your latest model. So you could set the threshold for what traffic is is like blocked to be like much more conservative. And so that may mean that you actually can only support you know like half of the traffic and actually have to like block many of the requests, but at least that lets you like deploy the model to some extent. You can get more evidence about whether or not the model is like causing some doing a lot of suspicious behavior. If you similar to the adaptive Yeah, you can Yeah you add a paper on adapting the control mitigations online during deployment. You can use that approach to see like if you've deployed your model for like some amount of time, like each individual example is like some information about how conservative you should be or not with your control measures. And so you can use that information to like make your control mitigations like less and less conservative, like have less and less of a conservative threshold for like which data gets flagged or used like shown to users. So Yeah, I think that would probably be the approach. What one like practical approach. I mean.
speaker 2: in practice, we're like trying to get pretty good at running control evaluations quickly as we develop new models. So some of the stuff that I've been working with the lement evaluations team, we have like this kind of business decision sabotage eval, this code sabotage eval, also some like undermining oversight evaluations. And those are like in practice, like actually control evaluations just with like very small amounts of monitoring we run before all major model releases. We kind of use the results of those evaluations as like a canary test for how good the moalls are at sabotage. Like currently, these are basically coming back negative at the minute because models, even with like minimal monitoring interventions in place, like aren't really that competent at doing like kind of really creative sabotage or sabotage that requires a lot of planning.
speaker 1: So I guess this like leads on to like another question, which is like what are the open questions in control that you're like excited to work on or excited for other people to work on?
speaker 2: Yeah. So I think one of the biggest issues for me, and we maybe touched on it before, is that building control evaluations is like really hard, especially because like we ideally want our control evaluations to be as similar as possible. The actual deployment setting and often the lessons we learn if the evaluations are like too different are qualitatively wrong. And so one thing that I'm like really excited about is just building really great evaluation datsets. And that is like not the sexiest work out there, but it's like super valuable from an ml perspective. And I think like people have learned this lesson like time and again across the field. It's like once you have a benchmark to optimize against, like making the number go up or like making the measurements and pushing on, that is easy. But like making the data set that really tracks what you care about is hard. Concretely, probably what this looks like is taking some of the threat models that we're most worried about. So I think Ethan mentioned some of them before. There's like models copying themselves out of our like data centers or models inserting vulnerabilities into code, like taking these settings, trying to make like really good synthetic versions of them and then like setting moalls loose in these environments and seeing what like forms of sabotage they can do, what sorts of vulnerabilities we can catch. Yeah, definitely.
speaker 3: If we have a bunch of these settings, I think there's a lot of interesting stuff that could be done. For example, we could have like pretty general insights about like in general on like code, trusted monitoring works better than on trusted monitoring or vice versa. You could also do things like scaling trends and see, I guess in like going back to your question earlier, you could see like in general, if there's like this, the trusted model and untrusted model are trained with a like a four x compute gap that's sufficient for controlling the untrusted model. If you could get and that that four x number is maybe like constant across like model scales or something like that. Like if you could get some kind of like general insight like that, I think that would be quite useful. I think in general, I think there's a lot of good work on just like how do you train really good monitors? Ideally, you want I mean, I think constitutional classifiers like make some progress here, but there's nothing like a lot more to do. But I think constitutional classifiers like make some step towards having like a push button like way to get a relatively robust classifier. And in that, like you just kind of like write your constitution of like list of things that you want to block, list of things you don't want to block, generate the data with your llms and then train your classifier and now you have your monitor. I think in the future, we'll need those monitors to be, I guess, robust to even the policy itself trying to like undermine the monitor, which is potentially even like a harder setting than the jailbreaker Buster setting where it's just the human that's trying to undermine the system. But you can have your classifier that checks the input. The model that generates the text is also kind of like on your side and like also needs to be jailbroken. And then also you have an output classifier and say like this in some sense, like three models that are on your side that can like catch all these issues. In this case, like the model is generating the text itself, maybe trying to undermine the monitors. So just looking at that setting, it's directly and seeing like can we get really robust classifiers for detecting this bad behavior? The other thing you could do, I mean, you can even directly train, I think an example of a project that be exciting here is just like directly train some larger model to produce produce text that is like effective at subtly like tricking or sabotaging like work, despite there being like a smaller model monitoring it. And then you just do a like red team, blue team game here where you train the policy to be really good at that. You then train the monitor on those samples to detect if this output is like sabotaging or not. Then once you have this improved monitor, you retrain the policy to be more effective at undermining this improved monitor and you go back and forth. And I think if you like have a really good pipeline for doing that, that could potentially just like output like a really good Yeah give you give you a really good monitor that is like less capable but can still catch like subtle issues from your like more capable model. So that would be an example of like a general approach. I think that would be like really useful for building building .
speaker 1: a classifiers from mifrom. The stuff that I've done, like maybe piggybacking off your idea before of like understanding like the untrusted trusted gap or like this like protocol versus like the thing that's being deployed and that like helped us like it meant that like when because these failure modes weren't correlated, like the monitor was actually much .
speaker 2: better in this situation. Yeah we found basically an identical thing actually on our like alignment evaluations. So the various generations of summit are like surprisingly decorrelated Yeah especially when trying to predict like each other's failures, which is both like maybe not what you would have guessed and like in a sense kind of useful for this control work because a lot of the like most scary red team policies involve like a lot of coordination, which it turns out well, just find pretty brutal.
speaker 1: Yeah. So I just like would love to people to explore this further vright, because this is a general phenomena itbe incredibly useful to understand our like Yeah working out which which models are better for monitoring, which ones aren't, is one. And then the the other kind of open question I have is, and you kind of you talk about this when you talk about untrusted monitoring, where we can like have a monitor, which is actually like the thing that we don't trust, but because it's having to introduce an exploit and then it has to also notice the exploit when it's monitoring the thing, then it's like low probability of happening, or like I feel like that's the main way of explaining that phenomena work of why it works. I think like a lot of that work is only successful because at the moment, the deceptive model doesn't know how many it's how many times it's been deployed at the moment. So it can't kind of like guess when the right time is to coordinate copying its weights out or introducing a subtle bubecause. It's trying really hard to keep that number really low so that it can just like do it occasionally and get through monitoring. And I'd love to see some control work that doesn't have dislike assumption that the red team or the untrusted model can't like see stuff.
speaker 2: Yes. I mean, I think there's like this general dynamic you often find in control protocols when trying to tune the high parameters is that like what you often care about is like even a very small rate of failure can be quite bad. And so both like you care about like the far tails of like safety, like the model, like busy never being able to cause like an egregious problem and simply the model, if it's like trying to cause a problem, like only he needs to get a success rate of like 0.1% or like 0.01%. And so like one of the things that I think we've kind of learn by playing these games is that like the middles of the distribution of like suspiciousness and like harmfulness don't really matter. You like really care about like the tales here?
speaker 1: Yeah. I mean, actually, this is like just a third point now, which is like I would love for better ways to evaluate like the rare events or the rare occurrences than we currently do.
speaker 2: Yeah, well, it's obviously the like consequence. If you really care about the talils, you an awful lot of samples Yeah, exactly. Be able to actually like measure the quantity you care about in any detail and especially like if constructing these environments is really costly, that's like that's a real challenge for this research.
speaker 1: Guaranteeing five nines of reliability is actually like incredibly difficult, yes.
speaker 2: that you might take away from control, especially if you're like handwriting all the environments as I have been doing on some projects, you like do want to be writing like ten.
speaker 1: zero data points yourself. It's fine.
speaker 2: Claude will help us eventually, right?
speaker 1: Claude is helping a bit. Clauis helping a bit. Yeah even do you have like any particular like lessons that you've learned from control of it? Like you know you've brought to other positive your research, I think like just like .
speaker 3: we've learned like five nines is really hard to like. You know let me think, I guess a lot of what's come up when we've worked on control, I think has been just that the it's helpful to have just frontier access to like frontier models to do a lot of this research. I think a lot of the threat models that we worry about are like models copying their weights out. Like that is the kind of thing that requires like models to be doing like multi step tasks like like within a company like coding environment. And I think we're sort of like really pushing on in order to get to test on settings like that. We're like really having to push on like we're kind of like at the frontier of like model capabilities. And like even then, it's not really able to do the models aren't able to do a lot of these tasks that effectively. And like very simple mitigations, like just checking the model with like a less capable models like will obviously just catch a lot of the behavior for some of these more sophisticated things. So I think Yeah, I think my real sense for like a lot of these threat models has been that it's like pretty difficult to pick like both an interesting setting that but that's also kind of like a real threat model that models can also do. And so at least for a lot of these settings, it feels like a little bit early to explore these. But the persuasion like business justification settings seems like one of the settings that we have some signs of life on. And like building more data sets and eils is like pretty helpful for creating more of these. Yeah, I think that's like interesting.
speaker 2: The persuasion setting, I think we're seeing signs of life on because fundamentally, the thing a strategy that works very well with the model is just being like generally a bit biased and it doesn't really have to think about that. I think they're like kind of at an interesting point with the fact to control research at the minute where models are just starting to become capable of the longer term, like mage antic planning that might be needed to execute on like more subtle like sabotage policies. And so like right now, what we're seeing at the minute is like for environments that look like moderately realistic models are like basically scoring zero at being able to like do something subtle and actually like achieve the goal but not get caught. But I think the thing that is like stopping them from doing that is a planning capability that I expect models to like get better at pretty rapidly. And it wouldn't surprise me if in one year's time like models are suddenly like slightly scarily good at this. And right now, one of the things I'm thinking like hardest about is, well, how do we build the evaluations that are gonna to like really demonstrate that at the end of this year?
speaker 1: Yeah. Yeah I would guess like maybe this leads on to like an interesting question of like why do control now as opposed to like the other alignment stuff? I feel like you've you've really gone in .
speaker 3: hard on the control paradigm. Yeah. Yeah. I mean, I guess maybe it's worth flushing out what the kind of like engame is. I think I think almost all people I've talked with, even people who are fans of control, think that eventually we need to align models for sufficiently capable models. I think the key reason being that like it's hard to control something that's much more like this, just much smarter than you basically. And you could you could do like fanfanor stuff and like make it to that. You know you as your human review, plus some like augment, like some other trusted models can like really be effective at checking the work of like a really, really capable like superhuman model. But at some level of like capability gap seems to be like probably it's just like quite hard to like have that much confidence that you're you're doing this effectively. I mean, people have been talking for years .
speaker 2: about the difficulty of keeping a super intelligence like in a box. Yeah but some in the limit control like almost certainly breaks.
speaker 3: Yeah, I think I don't know. I've at least talked with like one person who's like, well, maybe you use the the super intelligence to monitor itself and there's like ways to make it make that go well. So I guess there's like still some some at least like non zero discussion about this, but I think generally I think a lot of people are like, well, okay, the general strategy would be you control your maybe like slightly superhuman system, or at least human level system, which is able to do lots of autonomous tasks, including importantly like AI safety research and AI alignment research. And then you have the model to do a lot, a large amount of alignment research, including like mechanistic interpretability. You shore up your science like of how to evaluate whether models are aligned or not and then use that to develop better techniques for aligning models and also gaining confidence in your alignment techniques. And then hopefully you like, Yeah, then then you're able to use those on your like more more capable systems, like your superhuman systems. And from there, Yeah, I think you now you have like a superhuman alliance system, and then hopefully you just eventually solve the alignment problem just in general with the solution that works for like all intelligence levels. But you might need to do this process a few times where you have like your aligned superhuman model, which then controls like even more superhuman systems. You use that to do like better alignment research and like keep doing this until you have a more like long term solution or or yes, something something like that would be like kind of the general like longer term approach.
speaker 2: I do think the looking towards an end game, like hopefully if we end up in that regime, we can use models to actually do a lot of basic science, maybe a lot of the alignment signs we need to do to align harder models. Maybe if we like get into that regime and like models are controlled but still showing signs of misalignment, we can at that point like pause and actually start to study some of the really dangerous systems we're worried about. One of the big problems we have with alignment research at the minute is like models like can do bad stuff, but they feel like qualitatively different from the sort of like misaligned transformative AI systems that I think we are like all really worried about. So Yeah, I think viewing that like let's get to a system where models are like economically transformative but are controlled like as an intermediate goal, like opens up a lot of tactile. Yeah.
speaker 3: I think that's right. I think the important point to make is that like a huge amount of the benefits people are excited about and talk about maybe like Yeah almost all the benefits people talk about today are benefits you would get with effectively if you got like an extremely you know like extremely capable like human like replicated in an AI system or like that level of capability, like you Yeah you were able to get like the world's best expert in every single domain that's still like basically human level. Or maybe like the combination is like superhuman, but like at least there's like one person in the world who could like properly like evaluate and like replicate the behavior of the system. And like that already would be like completely world changing, is like maybe doable to control. And like it is possible basically like the like additional returns to intelligence. Like it's possible that it's just like much more on state, like if we like no longer able to control those systems, the risks become like much larger at that point. And it's not actually clear that like additional intelligence beyond that point will have like very large returns, probably for some domains, like math, sure. Like probably will have like Pure Math, like Yeah might have more returns, but it's like very unclear. And like efficiency may have returns, but like efficiency isn't necessarily to make the model or like that's not like a it's not becoming smarter unless you're doing something nontrivial. And so if we just like run those intelligences like really fast, I think we could get like a huge amount of the value and then Yeah, I think use that to figure out like what a better longer term solution is, like maybe stay there for a while. Yeah. Yeah. I think part of your question that was like.
speaker 2: no, just why weacan control? But why what couldn't it? Now I think like the elephant to the rothere is that like the AI field seems to be moving pretty quickly. And I think like a lot of us are feeling that while it would be nice to have a general solution to the alignment problem that scales like to transformative AI and all the way to super intelligence. It's like not clear how long we have to get that sort of solution. And a control solution that got us like into the transformative AI regime safely, but didn't necessarily take us further might still be super valuable, especially if we need something in the next.
speaker 1: I don't know, two to three years. Cool. So what are the freat models that you care about? Because I feel like that's a really good way .
speaker 3: to like you know frame alignment work. One of the really key things about control is that Yeah there's kind of this hypothesis that if you guard against the key threat models, then that's sufficient. And so you don't actually need to ensure that your mois doing like nothing bad. You just need to like list through what are the most major things that the model could do that would be really bad, block those reliably, and then hopefully from there, you're pretty safe to deploy the main threat models that Yeah we've been thinking about ananthropic. It's possible there more, but I think these are the most salient ones to us currently, are models copying their rates out, models deploying themselves within a company in an unmonitored way. And the third one would be sabotaging AI safety research. The reason the first two are important, the first two copying weights out of the data center and launching unmonitored copies of the model internally. Those are both important because we can then no longer it becomes like much more challenging or impossible to like do Yeah monitor what the model is doing, block it from doing certain things, retrain it, delete it, shut it down, things like that. If the model escapes and starts like spinning up like Yeah versions of itself on like cloud compute or like other compute resource it's able to find, then Yeah then it's sort of like very difficult to shut it down. And then Yeah I think it launches itself internally on the cluster on some Yeah some internal cluster, then I think that then it also becomes potentially also like similarly difficult or like maybe not. Yeah, I don't know. It at least becomes difficult to to catch that there's like something going wrong. You maybe need to like check the logs to understand like is this a real job that someone launched or is this like something that a rogue AI is like doing? Maybe the rogue AI can then do unmonitored additional security research, like research on how to then later escape or make some prs, which then enable it to do something else that's bad or additional things that are bad. So Yeah, I think those that's those are like two of the key threats on the sort of like unmonitored deployment side. Then there's another Yeah, I guess sabotaging safety research is the other threat. And models, I mean, already models are producing like a large fraction of the code like Yeah surprising like large fraction of the code that like the world is producing. But Yeah also also including like people are doing like AI research, like with tools like cursor or Copilot using alums to generate the code. And Yeah pretty soon seems pretty plausible that people use like more agent tic coding systems, which are doing sort of like multiti step coding, coding tasks like writing code, running it, checking the results and repeating that loop. And with those systems, we like Yeah I think there's like a kind of like strong pull towards just like using those to automate a lot of safety work and like that. The upside of that would be really high. We could get like way more safety research done. But also on the other hand, like if we don't trust the models, they might actually like sabotage that research, persuade us that of a wrong interpretation of some certain plot persuadus to go down like a wrong research direction, which is actually like a rabbit hole and like waste a lot of our time, persuades us that like a certain model is safe, maybe even like runs experiments to justify that the model is safe and then gets us to deploy an actually unsafe model. See, I think there's just like a lot of risks related to Yeah sabotaging our like understanding of the safety of models. See, I think those are Yeah that's that's another another key thread as well.
speaker 2: Yeah there are other sketchy things the moles could do, for example, that point internally, like insert general security vulnerability so that someone then from the outside could exploit or maybe like not directly influencing like AI research, but like poisoning data for future models or persuading like internal people or even like external people at very important organizations about certain key decisions, or like doing kind of research helping people like externally with AI research. Maybe if we're in a situation where we kind of we would like much prefer that AI research was happening at a small number of like well regulated or kind of well understood institutions instead, like model might kind of help people outside do a lot of AI research. That's like another thing we might want to monitor for. But like Yeah, I think roughly most most all things that we care about boil down to the three things you .
speaker 1: outlined or close to it, scary future or like scary risks, but we should definitely mitigate or like work towards mitigating. I wonder like maybe to like leave on an optimistic note, like what model developments you think have happened or what has given you like positive evidence that like control will work or like other alignment techniques .
speaker 2: will work in the future? Yeah. Well, I think that like one thing that's been maybe pretty optimistic recently is this like trend towards models doing more and more externalized reasoning and people being able to get a lot of like additional intelligence juice out ter models by getting them to do these kind of long form reasoning kind of traces. And this is like kind of great in a sense in that the models are like putting their thoughts out there and you can read them and like kind of assuming they're an accurate representation of what is going on, like what the models is actual reason for doing stuff. And that's like a huge surface area to be monitored. And I think if you're thinking about this from a control perspective, then the kind of questions you might ask are, okay, well, suppose we put a model that uses like a very long chain of thought into an environment where it like has the opportunity to sabotage something and we let it try like firstly, like can it can it use the chaof thought to do that task? And like, well, maybe it can't, in which case, probably we're just not worbecause the model can't sabotage. But if it can, then it seems like pretty likely that the models are like much better at doing this task with the long chain of thought than without. So like in some sense, that chain of thought is load bearing. And if the chain of thought is load bearing, then it seems like pretty likely that actually there are going to be like explicit references to the sabotage attempts. Like in that chain of thought, you're going to think of the converts. Like if that weren't the case, then the model must be like using the chain of thought for reasoning about how to sabotage, but still managing not to say anything about sabotage in its chain of thought. And that's like certainly in theory possible, but in practice seems like pretty unlikely, at least for models that have like only just become capable of this sabotage. And then we're not like training by default to be able to obfuscate their chains of thought. I think a .
speaker 1: massive winners like in constitutional classifiers, but the classifier can be like so much smaller than the like main model. Is checking us something like I'm impressed at how small .
speaker 2: monitors can be now. Yeah, that is like a concrete win for monitoring. But like I think we're pretty hopeful we'll actually work in the real world and like shows that these things are deployable in practice. Yeah. I think it's like yet to be shown like exactly how useful this is for misalignment type risks and like whether we should worry about like the monitoring and the policy colluding or whether like AI systems have correlated failures. But I do think that it's like derisked a big chunk of the technical risk for .
speaker 1: like deploying Monin practice. Nice. So control super important for the next two years, and we're optimistic we can do it. Sound ounded it right. Hope so. Okay. Nice. Thank you for watching this roundtable. We've been anthropic peace.