Often if I achieve something arduous, I award myself some kind of predetermined nice thing, such as a sundae.

When I discuss this with other people, they tend to say things like “but I can just eat a sundae anyway”, which I hear as meaning that the issue is that that they have no ability to stick to their commitments and not acquire the sundae unless it is earned. But on further discussion, what this friend meant at least was that since there isn’t any rule against eating a sundae any time he wants without doing something arduous, it isn’t a very compelling impetus to work.

While I do usually treat the reward as disallowed in the temporal vicinity of it being a prize, that isn’t the important difference here in our models.

I’m just not at all expecting a sundae to be a sufficient incentive to work hard for hours. Winning at a challenge, and at achieving my goals in particular, is compelling in itself. The prize for winning is more like a decoration accentuating, concretifying and glorifying the win, which might otherwise be an abstract and anticlimactic point.

So if you think you are the wrong kind of person to make use of concrete prizes for achievement, due to not being incredibly compelled by the kind of stuff a child with no money would work hard for, perhaps reconsider.

Prima facie that does sound kind of objectionable. But is it?

People definitely do a lot of things without much explicit account of the point of each of them. For instance, many people go to university without taking a stance on whether it is primarily fulfills a learning function or a signaling function or an associating-with-elites function or or a getting the hell away from whatever they have been up to so far function.

It’s not that people think of going to university as pointless, they just have a vague sense of it being good in some rich array of ways.

Are activities motivated in this way much less useful than ones done with a distinct purpose?

I doubt it. But it seems complicated.

I’d guess that if a person thinks explicitly about their purposes in university, they might make university-related choices somewhat better. For instance, if they clarify with themselves that a big component is meeting like-minded peers, that might change which courses they do in counterintuitive ways.

But on the other hand, if they too much trust their breakdown of purposes, I expect things to get worse again: I’d guess a teenager who decides on a specific purpose for their university attendance to make worse choices than one just heuristically and hand-wavily trying to do what seems nebulously good.

Likewise, if I’m running a party, I don’t think it should have a point. Though I do think considering the possible sources of value from it might help.

On the other hand, if someone really wanted to cause a specific unusual event, and could give no account of why they wanted to do that except for evidently thinking it ‘would be good’, I might not be that excited to help them.

When should you know the point of your actions?

For a random person, I’d guess the unaccounted for effects on other people from their making different private decisions about their lives while being cared about by numerous family and friends might be at least comparable to those from marginally contributing to air pollution or city density or traffic congestion.

The speaker often thinks there is a fairly substantial risk of the AI thus produced killing or disempowering everyone, including Trump and China. And I imagine in a situation where a certain course of action were going to produce a 20% chance of Trump being shot in the head or China being heavily nuked, that these parties would actually be considered to be ‘following incentives’ to avoid it. Yet they talk as though the idea of Trump or China responding to such risks is akin to the idea of these parties suddenly becoming zealous proponents of universal selfless love randomly.

It’s like while believing in the risk, they also kind of believe that it’s a totally uncompelling story that nobody in real geopolitics would ever be touched by.

Or that these parties actual utility functions are ‘be the bad guy’, which, you know, sometimes involves extreme selflessness as long as it puts everyone else at risk too.

Any other hypotheses for what’s going on?

In the age of GLP-1 drugs and the like, being fat shifts to partially a sign of not taking GLP-1 drugs, which makes it increasingly a sign of different traits, such as lack of spare wealth and reluctance to chemically modify your body (albeit a very imperfect sign).

I wonder if such body modifications will reach a level of cheapness, availability and acceptance that it is hard to read much of interest into being fat at all, much as it is now hard to read much into someone having a purple sweater or not. And then people can be fat in peace.

It’s like how you can believe a fictional world has dragons, but it strains credibility if characters start being totally indifferent to social status apropos of nothing.

I can warily accept that boxes of wires can now talk like humans. But that huge official companies now proudly present products that are like crazy confused ladies that do a lot of tasks for you, many accurately, but also try to steal you from your spouse (or these days encourage you to kill yourself or believe in new spiritualities), feels like a scene written by someone with poor familiarity with the character of companies.

But it’s actually written in reality, so what don’t I understand? I guess just that if something is hypey and perceived to be in-future-profitable enough, normal standards of professionalism must fall to that?

I phoned CVS, hoping to move an antibiotic prescription to a different pharmacy address so I could pick it up before leaving on a flight. I was answered by an AI, but the smoothest, most charming and person-realistic AI answering machine I had experienced. It asked me in a friendly woman’s voice if I wanted this and that medication sent over, and since I didn’t really know exactly what my medication was, I agreed to everything it said. This seemed great. I hung up surprised that phoning CVS had actually led to the thing I wanted in short order, and reluctantly impressed with their phone AI.

I tried to pick up the medication. It seemingly hadn’t been sent to the new pharmacy at all, and they didn’t seem aware of anything like that being planned. Huh. I still managed to get them to move it from the pharmacy, and catch the flight. I got a message from my doctor’s office saying that they had received a request from CVS for two other antibiotics, but they didn’t think that would be the best choice at this point. From what I can gather then, this AI assistant just made up a couple of different plausible antibiotics that I hadn’t been prescribed and tried to order them for me from my doctor.

Ways this could have gone wrong: a) I don’t get antibiotics in time and get a more serious infection, especially for instance if I’m old or don’t have heaps of time to follow up at pharmacies, b) my doctor’s office just trusts the ‘pharmacist’ request for meds and I get sent meds that were literally hallucinated by an AI, and take them.

So beyond dangerous, this all seems wildly unprofessional. If a person behaved in this way, there’s no way they could work as a phone operator for a respectable brand like CVS. Even if they were part of a huge union of people willing to work for free.

Why is this suddenly acceptable because it’s AI?

How to avoid such problems, from either side?

• A couple of people asked me for ‘one thing’ or ‘one good thing’ in my life lately, which avoids this problem nicely by giving more leeway to bring up anything at any level of importance, while remaining similar enough to the normal question to not seem super weird.

• Someone asked me if I was going to open a physical storefront, what would it be for. That actually worked pretty well to start a conversation, that was non-zero revealing about my state. At first the general category here seems like ‘extremely random question’, which for many people and contexts risks being too weird to pull off. But perhaps the relevant category is more like ‘concrete hypothetical question implicitly about the other person’s preferences and nature’, which is both relatively fun to answer, and relatively useful for learning about one another. More examples: ‘would you rather be rich or famous?’, ‘would you rather be one dog or two dogs?’

• This could be made further less weird by connecting it with the current situation. ‘If you were in charge of local parks, how many mysterious objects like that would you try to have in a park like this?’

• More ideas?

For one thing, you have to avoid all those people you don’t want to meet, which means not just accepting bids from strangers on the street. For another, reciprocally, you have to get past everyone else’s screening processes—that is, somehow distinguish yourself from a stranger on the street.

Consequently, the streets are full of lonely people, including many who would like each other, but can’t meet for lack of a good screening process that would identify them to one another.

In the old days and other cultures, the low bandwidth screening processes are an even bigger impediment to meeting people than now. For instance, it was improper to greet someone at a party without being formally introduced by an appropriate person.

In the future, if we manage to keep using AI without catastrophe, I wonder if it could herald a world of very high bandwidth screening processes, where you could just happily meet people on the street all the time, in any case where you are properly introduced by an appropriate AI system.

• If we pause at the last possible moment, then we will have the most advanced AI possible during the pause, which will be helpful for doing AI safety research during the pause

• Implicitly, there is some quantity of ‘pausing credit’, that will buy us a few months of pause say, and if we use them now, we won’t have them to use later, when it is important

• If we pause, and then AI doesn’t seem to be at dire risk of destroying the world, maybe the public will backlash against this and it will be harder to do any kind of AI safety (especially if it has major economic consequences)

• The models aren’t dangerous yet

This all sounds very questionable to me. I suggest instead that the following are at least as likely to be true:

• We can’t pause on a dime at the precise second that ‘we’ decide it is important to—pulling the breaks will take a while, during which time we will continue to rocket into danger.

• If we managed to pause now, that would greatly increase the chance we paused again later. At the moment, a major obstacle to people supporting this is their senses that it is impossible. Building the machinery to pause makes it much easier to pause again. Doing something once is in general extremely helpful for doing it later times—if you are taking the most important action in the world, you really don’t want to be trying it for the first time, when it matters.

• The public substantially hates AI, but feels incredibly disempowered, because for instance they buy the story that technological ‘progress’ is inexorable. If they saw vividly how much power our institutions have to shape the course of technology, they would become more activated against AI.

• Some of the models seem to be some amount dangerous, we can’t tell how dangerous the new ones are, they are improving very fast, and the point where we are confident that they are currently dangerous is very suboptimally late to start this project.

For instance, if I have a meeting this afternoon, it looms there in a certain way in mental space.

Whereas if I will get a drink later this afternoon, it doesn’t really take up any mental space and does not loom.

You can kind of intentionally cause the meeting to seem like the drink, adjusting how much ‘space’ it is taking up.

(It’s more obvious that other things can cause this change, for instance, if a much more important meeting shows up some other time this afternoon. But I think you can also do it intentionally.)

I wonder if this makes sense in others’ minds.

I think this is often because I tend to pick up constraints too easily. That is, I implicitly believe ‘I have to read this thing’ instead of ‘it would be some amount good if I read this thing’.

Having constraints instead of evaluations makes decision-making simpler: if you have to read this thing, then that’s what you are doing. And if you have to go to work during work hours, then that’s where you are doing it.

Up until you have too many constraints, and it becomes impossible to find a solution that satisfies them.

I don’t know if other people have this problem, but for me, it is helpful to notice this and explicitly drop some constraints. Remember that you don’t have to read this thing or wash your hands or not be considered an idiot or show up to your appointments.

(Inspired somewhat by a talk by Aaron Kaufman, along the lines of this blog post)

I’ve found it useful to think of some feelings as like post-it notes. I have a lot of things to remember to do, and thoughts to remember to think more about, and problems to remember to address. And I have written too many of them in a todo list, and another todo list, and uncountable pieces of paper. So if a new thing arises that I really need to remember soon, I don’t have any place I can put it where I have faith that I will address it. Except one place I’ll definitely notice it: as a continuous discomfort in my own mind. That is, just feel continually troubled or guilty about it until I’ve done something better with the information. This works a bit like chanting a number repeatedly until you write it down.

To be clear, I don’t mean to do this. But I think I can sometimes stop being anguished by recognizing that it is serving that function, and just recording the information better, or giving up on definitely remembering it.

(I probably owe this idea to someone else, but I don’t remember who. Maybe John Salvatier or David Allen.)

But I’ve been on a few weird dates too, including some pretty good ones (in terms of date format at least). This is one.

You go to a nice place with lots of stuff, such as a street full of interesting shops and cafes. You make a list of things to find, which will be a mixture of fun and intimacy improving, for instance:

• A picture you both agree is beautiful

• An object that one of you finds delightful and one finds cringe

• Something that reminds one of you of a thing you did as a teenager

• A dress that would look good on one of you

• A sign that makes one of you mad

• Something neither of you have eaten before

• A delicious cupcake

You can construct this list together at a cafe at the start, or one of you can bring it.

Possibly set a time limit or play it by ear.

Search.

If you find all the stuff, get a drink to celebrate. If you fail, do something else fun to anti-celebrate.

I’ve done this with someone I didn’t know well and it was pretty good, both in terms of fun and hearing about different aspects of one another. I could imagine it also working with someone I did know well.

Now the people get to share their message with some meta-information about its importance - for instance, they can say it more or less loudly.

What happens?

Lets say they all say it at a loudness that corresponds to ‘should get 10% of the attention’. The listeners then divide their attention between the messages, and each one gets 9% of the attention.

The speakers realize they spoke a bit too quietly for the importance of their message, and turn up the volume.

But the next time, they find that the ambient volume has increased, and their new volume again only gets them 9% of the attention. So they increase it again.

Eventually, everyone is shouting as loud as they can, just to get what seems like a rightful look in.

I see this pattern in lots of places. Too many emails titled ‘Important’, too many paragraphs beginning ‘One key thing to remember...’. Loudspeakers trying to breach travelers’ headphones with important advisories. Todo lists forever longer than you can possibly do, and marked with things like ‘NO REALLY DO THIS ONE’... Warning labels and ‘open this immediately’ signs. Perhaps even the length of writing: an author emphasizes with length what they think will be ignored, but then other stuff will be ignored, so they emphasize those too, not coming to terms with limited attention of reader. The proliferation of objects that should be findable. Literal speaking in crowds.

I’ve been in the AI safety and Rationalist and Effective Altruist spheres since about 2008 (or the beginning of EA), so generally go around feeling well acquainted with them and justified in opining on them. I gather that many other people do too.

But reckoning with how huge they are now reminds me that of course I only know much about little corners of each, and each has many corners that are fairly different. Nonetheless, I’m tempted to opine about them in general, and much more often hear people opining about each of these communities in general.

This reminds me of how people seem to much more often hate Berkeley than hate a particular part of Berkeley, even though the particular parts are fairly different.

Why?

If we wanted to, how hard would it be to create another institution with similar virtues? (Not to replace families in general, but to have more of that value, or a backup for when family fails, as it does for many people.)

People often consider close friends to to be their ‘chosen family’, but I think the inability to choose is relevant to some of the virtue of real families: you are stuck with each other almost no matter what, and you are forced to be on good

A rough method I could imagine: take a large set of people and use AI to sort them by some measure of personality similarity and physical proximity into groups of 4-10, across different ages. Give each group substantial incentive to interact and be on good terms with each other, for instance free use of local amenities if they show up together. (I’m imagining this is brought about by a national government, but alternately a large group of people could decide to do this with each other.)

This plan is probably missing something like the meaning and loyalty-to-ones-blood aspects of family. Though I could imagine the right personality assessment system might change that, if people felt they were being put in touch with essentially their platonic soul-mates.

One reason people purport to not be impressed by these chances of extinction is that they are subjective probabilities, not verified frequencies or arising from a more formal process.

My natural response to this concern is to argue about this, since I think it is very wrong and misguided.

But a different direction would be to just try to get admissible numbers on these things that some of our more normal processes of responding to risk can respond to.

It’s going to be hard to get straight up frequentist numbers due to extinction being not very repeatable.

But in other areas of life, we have systems that must deal with subjective probabilities and non-repeating circumstances, and take them seriously and quantitatively. The fact that a specific company can’t go bankcrupt a large enough number of times to get a good outside view on it happening in a given month doesn’t mean investors just write off that possibility as speculation. It gets a quantitative part in the price of that company. And the chance of it happening isn’t an output of a formal method—it’s a bunch of people’s guesses, from looking at a detailed situation. Somehow the subjective guesses about this and a multitude of other sources of future value and disvalue get processed into an extremely specific formal price number that official processes of the world take seriously. If the price of a company goes down because people think it might go bankrupt, a bunch of people will make different decisions as a result.

And in the end, very few uncertainties in the world can be turned into clear frequencies—even if you have a disease that a million other people get every year, you aren’t the perfectly average person with the perfectly average body and life and your prognosis and what you should do can’t be pulled out of a formal process. And this is a relatively good case for using pure frequentism.

So how else does human society turn subjective judgments into admissible forms of uncertainty?

~

Inspired by a talk by Paul Christiano.

The fifty year old is an odd one out though: nobody has any previous experience being as old as they are now! Not only that, but all of their experience is of being younger. And granted, some of it is only a little younger, but a lot of it is a lot younger!

Everybody just stumbled into being this old. It’s their first day.

Does this cause ‘acting ones age’ to warp over time? Today’s 45 year olds have an idea what acting 45 looked like for the 45 year olds that came before them, but they are also very used to acting younger than that.

Among most people in AI safety, it seems taken for granted that these are not relevant. And whether or not they are ultimately relevant, it seems very plausible from our current vantage that advanced AI can be very dangerous without them.

To intuitively picture this, I suggest people think about the game-playing AI systems we already have. If a system can trounce us in game worlds with instants of thought, why couldn’t a vastly scaled up version do similarly in worlds that matter to us—the physical world, the financial world, the worlds of public opinion and political power and geopolitics? Why would it need consciousness or some kind of free will to do that?

Few people think that game AIs are conscious, or make their own goals. But they can clearly have goals, and navigate bringing them about destructively.