Skyview

lcamtuf :verified: :verified: :verified:: ""AI safety" is such an interesting but nebulous c…" - Infosec Exchange

"AI safety" is such an interesting but nebulous concept. If you're thinking about building a career in this space, I think it can be broadly divided into several subcategories: 1) Traditional infrastructure security & IP protection for AI platforms. We know this, we kinda suck at it. Boring. There are some novel bits here about extracting some knowledge about the model or the underlying training data via prompting, but the bulk of it is just traditional infosec. 2) Brand safety: making sure that your LLM can't be made to praise Hitler, deny climate change, or generate porn. This is where the vast majority of commercial "AI safety" efforts are focused today. Because we cherry-pick contemporary hot button issues, the results are inconsistent and sometimes funny, but I get it. You can't have it any other way. There's certainly money to be made by developing standardized, comprehensive evaluation frameworks. 3) Functional safety: making sure that the LLM can't be tricked into performing unwanted actions once deployed in commercial settings (e.g., customer support). Arguably the hardest problem; if it doesn't outright derail adoption, it's going to be a major source of infosec lolz in the coming years. Specialize in this early and get ahead. 4) Doomsday safety. Intellectually stimulating but not terribly scientific, as it deals with the technology we don't have right now and that we don't know much about. Has two major flavors: - "Hard" doomerism, which posits that we won't be able to reliably contain a superhuman AGI, so we shouldn't attempt to build it. I think that's a pretty coherent view. - "Soft" doomerism, which posits that AGI is safe if aligned with human values. Seems pretty hand-wavy to me. Because we can't intuitively grasp what's going on inside an LLM, this gets reduced to #2 (brand safety): ask the LLM what it thinks about Hitler and puppies and don't release it if gets the answers wrong. The dodgy assumption here is that the answer is a reflection of an internal moral compass and that it will remain constant no matter what. I don't think there's money to be made on "hard" doomerism unless you want to monetize a podcast or a blog; as for "soft" doomerism, the prescription listed in #2 certainly holds, and it's something that the recent White House order will help you sell.