Hi there everyone! It’s time for the eleventh episode of my episodic updates!

London

I’m still in London; it’s a pretty cool city to explore! I might visit Canary Wharf and the area around there next.

Research

I wrote up some of my thoughts on making NLAs but for the gap between layers.

Currently I think before getting those things to work, I should figure out how to make it so NLA explanations are more faithful representations of the actual underlying state. Right now NLAs kinda work because there’s a lot of stuff in the activation, and the NLA only needs to gesture in the right direction. But residual stream updates have much less content in them, and I think it’s much harder to faithfully verbalize them.

I think part of the solution here is to make the NLA explanations more casually linked to the activations (or activation deltas for what I want to eventually get to). I’m not sure what a good way to do this is though. Currently I like the idea of having an LLM try to make predictions about the model based on the explanation. (Not sure exactly how to operationalize this re: deciding what questions to ask it.) However I fear this is harder to do for delta NLAs, because the effects of individual updates are much smaller. (It’s probably a lot slower to RL, since you need to run an LLM to compute the reward. Although you can use an LLM that much smaller than the one you’re trying to interpret.) But probably worth trying for normal NLAs to see what happens, then trying to use that to make a good delta NLA.

Security research

I finally got to write about some of the LLM harness Git security stuff I did earlier this year! The issues I found weren’t that bad, but I had a lot of fun exploring Git internals while working on this.

There’s some other fun security stuff that I’ve done that I’ll be able to write up in a few months.

Looking forward

I’m still going to be in London for a while; it seems pretty likely I’ll be here until at least November or December. It’s not as good as SF but it’s still a pretty good place. I’m going to spend some more time exploring London over the coming week. Please let me know if you have any advice for what I should do here!

I’m going to try to do a lot more writing over the coming month or two. Writing about research seems like a pretty useful way to help me think through it more, and it’s also useful to be able to feed my thoughts to LLMs. Hopefully I’ll do enough interesting stuff to be able to make these updates more frequent; I might even be able to achieve my Inkhaven level of daily posting!