One Hour One Life Forums

This game was never meant to feature a full-fledged combat system.  The protocol was designed to allow movement and actions from hundreds of simultaneous players with as few messages as possible, ensuring logical consistency (two players can't both pick up the same object at the same time and cause the object to duplicate), but not realtime interaction guarantees.  In fact, the protocol was design with extreme and variable latency in mind, because I always intended players from around the world to connect to the same server.  One of my test cases involves using the tools TC and NETEM to manually insert up to 2.5 seconds of latency for all messages sent to my test server.  The game still mostly works, even in those extreme network conditions.

So why is there combat in the game at all, then?  Because it's necessary for society to function.  How can you have rules or laws if you can't enforce them?  How can you enforce anything without, well, using force?  So yes, it would be easy enough for me to remove combat from the game entirely---just a few mouse clicks.  But then the whole thing would fall apart, and the game would be ruled and ruined by griefers, happily ever after.

As a simple example of an endless annoyance:  you set down a tool that you are busy working with, and a griefer grabs it and won't give it back.  This is kindergarten stuff, but it graduates to petty or even grand theft in the adult world, and in any case, we solve these issues by adjudicated force, at the end of the day.  But let's say we want no violence in our game.  How do you deal with the person who took your tool and won't give it back?

A game could "solve" this problem for you---and many games have tried---via a very detailed system of hard-coded privileges and rights and transfer operations, essentially hard-wiring a legal system into the fabric of the game itself.  Your tool would have an owner list, and that other player couldn't touch that tool unless you added them to the list, and they can't add other people unless you also add them to the admin list.  All to just lend someone a tool.  If this sounds fiddly and tedious, you're right.  We might call such a design a "trust system," but is is anything but, as I will explain in a bit.

Much simpler to let an understanding develop between players---who was using the tool first, when it's acceptable to grab it yourself, and when you should ask first---and give the player the power of force-in-numbers (in other words, law) if that understanding is repeatedly violated.  If you say, "Don't plant wheat here, I'm tilling these rows for milkweed," it's expected that the other person will listen.  If they don't listen, you complain to them, and warn them.  If they keep on with their violations, you get the backing of the town elders, and you take it to the next level.  After all, they can't keep violating the community standards if they're dead.  Unvarnished, it sounds barbaric.  But that principle is at the beating heart of every functioning society in the real world.  The less-barbaric-sounding version is imprisonment or banishment, but even those are carried out at weapon-point, for obvious reasons.

Even without the will of the majority to back you up, you can see how an understanding could develop between two isolated players over time, and how that understanding could blossom into a real sense of trust.  Real trust comes from freedom and power:  you could hurt me if you wanted to, but you choose not to.  If you are powerless, I may not need to worry about you, but that doesn't mean that I develop a trust with you.  Two free wolves develop a real trust.  Two toothless dogs stuck in neighboring cages have no need for trust.  That's why a hard-coded trust system, in the place of powerful and free players, actually squashes trust instead of fostering it.

And that is what One Hour One Life is about, from its very first moments onward.  Two strangers trusting each other, sacrificing short-term gain for long-haul mutual benefit.  Mother and baby are the basic example.  There's no hard-coded guarantee that you won't betray me, but I have your back, and you have mine, and you haven't betrayed me for the past 30 minutes, so even though you're a complete stranger to me, I sense that I'm growing to know you.  And just maybe---dare I say it---I'm growing to love you.  Because you could kill me, but you choose not to.

All that said, for this to function, we don't need a full-fledged combat system.  Violence can be a logical operation, or at least it should be, and was meant to be, in this game.  If I choose to kill you, I should be able to do that without any execution skill required, and regardless of network latency and so on.

Which brings us to that infamous dancing griefer.  Because the movement protocol is not a realtime one, it was possible to exploit it to make yourself nearly impossible to kill.  The server essentially had no "sync points" between players in the system until the players landed at the end of their movement.  Okay, the player's movement just ended, and we know where the player is standing, for real, and all clients and server agree about that.  Now they can execute an action on an neighboring tile.  Mid-move, due to variable latency, all bets are off about where a player actually is on each client screen.  Griefers could exploit this by moving continuously.

Even if the protocol didn't work this way, a moving target is still way harder to click.  And what happens if you misclick?  Well, you just right clicked on the ground, which means you just dropped your weapon.  Trying to finally hit a dancing griefer was an absolute click-fest of frustration.  So even if the town elders have collectively agreed that this person needs to go, they had trouble carrying out their decision.  And the griefer, who could sneak up on an unsuspecting person who wasn't busy dancing, would get the jump on each victim and be able to land the kill.

This update changes the semantics of the client KILL action.

It used to be an isolated action that would either succeed or fail depending on where you clicked and whether the server believed that the person was actually there and in range.

Now KILL is a state, not an action, and SHIFT-right-click puts you into that state, with essentially a death warrant for whoever was closest to your mouse when you clicked.  You get a forced-ANGRY face when you're in this state, and you can still move around freely.  But if you ever cross paths with your target server-side, so that the target of your request is in range of your weapon, the kill action is executed instantly.

This comes as close to a logical operation as we're ever going to get, while still preserving the effects of weapons with different ranges.  If you kill-target someone with your bow who is in a locked room, and you stand by the door, the arrow will fire instantly as soon as the door is opened.  If you target someone outside the gate with your sword, and then stand by the gate, you will attack them instantly the moment they walk through the choke point.

You can cancel this state by putting your weapon away momentarily, and your angry face will return to normal.  The face also gives the target some warning about your intentions---and of course, it can be bluffed with the usual angry emote.

This is a pretty big change, and it will dramatically change town griefing dynamics.  But there are some other big changes beyond that.

The temperature system in buildings has been improved again.  Now the heat simulation map is 13x13 instead of 8x8, allowing larger buildings to function as insulated spaces.  Furthermore, an enclosed building acts as a direct temperature bonus on its own, and also doubles the effectiveness of the clothing that you're wearing.  So buildings do more than just hold in the heat of a fire.  The best way to think about it is that they reduce some kind of inherent wind chill in cold areas of the world.  But for this to work, you have to be completely indoors, which means walls on all sides and corners, along with a full floor, and along with a closed door.  If the door stays open too long, the wind will blow in, making you colder slowly over time.

The way last names are passed down to babies, especially in the case of a generation gap where no name is given, has been standardized.  This should fix the lost-family-name bug.

Laying tracks for rail carts is now a lot cheaper---one kit can lay six tracks instead of just one.  Gates can be built across roads again, and unused property twig bundles decay away quickly, so you don't need to spend time cleaning them up.  You can finally move bowls of sterile pads around---mobile medicine can be a thing now.

Next week, I'll be tackling Eve overload around existing cities, and also the balance between close towns and stripped natural resources.

The Eve spawning algorithm has been changed from an ever-growing spiral to a dense grid packing, based on used natural spring sites.  That means that towns are much closer together than they used to be.  Walk in a cardinal direction from your town well for a bit, and you're bound to encounter other settlements.

But that doesn't mean civilization will turn into one giant, sprawling mush.  Just as I've brought you all together, I've also added a few other things to help you maintain some separation.  You're not building that tower to heaven together quite yet.

My friend once observed that as photographs age, they transcend their status as documents and move gradually into the realm of sublime art.  Even the most casual, throw-away snapshots, given eighty years or so, become nothing short of jaw-dropping---windows into a forgotten world that we can barely imagine ever really existed.  We also occupy a very special place in history---the place where photographs, as a relatively recent invention, only document very recent history for us.  We can dream about photographic evidence of the middle ages, but we will never see it.

Now imagine people five hundred or a thousand years from now, and how they will likely be able to look at photographs that are many hundreds of years old.  Even more mind-boggling, they will have access to moving images that are many hundreds of years old, where we don't even have movies of our grandparents when they were young.  It seems like these kinds of multidimensional documents will serve to make distant history more real and less mysterious.  That seems neat and all, but part of me likes the mystery.

And this game mostly preserves that mystery.  What happened long ago?  What did it look like?  Who lived here?  How did they dress?  Now we all have a very narrow window into the past, and your photographs will be messages in bottles that wash up on the shores of the future.  Damn, I'm really laying it on thick this time, aren't I?  But this is profound stuff that we're dealing with here.  Pictures of Grandma.  Forever.

And the first in-game photos have already started showing up.  Here's a good one for you:

You can see the full stream here:

http://photos.onehouronelife.mythiclair … front_page

And check the family tree browser for people who have been tagged by a little photograph icon.

This update is mostly about one thing:  New clothing, and lots of it.

But there are also a few key fixes under the hood.  Your family's last name is remembered, even if your mother doesn't give you a name.  So later, if you name your own child, they will get that "invisible" last name passed on to them.  This will make family trees more coherent, with a last name shared by every named person in the tree.  No more lost family names over time.

Even cooler is the new grave-mouse-over feature, which now works for all graves, not just graves for people who died in your lifetime.  Family history is now apparent directly inside the game, when you visit the family grave yard.  To make this work, the server remembers everyone who ever lived, and keeps a new database of grave positions for each of those people.  To prevent this from growing indefinitely, people get "forgotten" after a week, or when the server restarts, and the grave position database is flushed at server startup.  But if your family has been around for a while, you will see a lot of family history on the ground.

In other news, community member Futurebird has created a survey for players, which you can find here:

https://www.surveymonkey.com/r/3LKVW7P

Here are the results:

https://www.surveymonkey.com/results/SM-LRBDYL2FV/

There most likely won't be an update next week---I'm scheduled to be on vacation with my family.

The past few weeks have been an absolute whirlwind.

First of all, preparations for my GDC talk are complete.  I'm actually giving the talk just two days from now.  You can read the full description at the link below, but keep the intended audience in mind:  "Anyone who isn't ready to roll over and die just yet."  Hopefully, that describes you.

https://schedule.gdconf.com/session/201 … pse/864486

Also in the past few weeks:  a huge, confusing debate about the boundaries of the public domain and authorship, as it concerns One Hour One Life and the unofficial mobile adaptation.  There are very few precedents here, and lots of different opinions.  You can get a taste of this here, in what became one of the longest and most viewed threads in OHOL forum history:

http://onehouronelife.com/forums/viewtopic.php?id=5479

You can also read the opposing point of view, in which I almost became a Sith Lord:

http://onehouronelife.com/forums/viewtopic.php?id=5534

There are two upshots of this discussion.  First, the mobile developers have decided to eliminate this confusion in the future by hard-forking the game.  They will be eventually operating under their own unique title, and they've already started the process of designing their own mascot character, and some of their own background textures, which you can see as sample of here:

I recently realized that the "mascot" they had been using in their mobile icon---my Original Eve character---happens to be my cartoon rendition of my own mother:

The hard fork will also involve a content fork from here on out, so my weekly updates will no longer be rolled directly into their game.

The second upshot is that I've clarified my stance on copyright, moral rights, personal rights, and trademark.  In the past, I had only thought about copyright, and decided specifically that I did not want to wield it.  Over the past fifteen years, I've never faced the situation of a direct competitor using my work to make a almost-identical service that would be confusing to end users.  When I placed my work in the public domain, I never thought, "Hey, that means that someday, people may widely believe that someone else is the original author of my work."  Furthermore, I realized that competitors have a huge unfair advantage against me, as I toil away on weekly content updates that they can roll into their product with almost no work at all.  Finally, I realized that the ultimate financial nightmare---a completely free but identical OHOL PC service, maybe even operating on Steam---was a possibility, unless I clarified things.  Yes, I was hoping Steam wouldn't allow such a thing, but if I'm serious about retaining no rights, that's an intellectually dishonest thing to be hoping.  I've decided that trademark is the best tool to use to deal with such issues.  My clarified "no_copyright" file can be seen here:

https://github.com/jasonrohrer/OneLife/ … yright.txt

To summarize:  you can do whatever you want with my work, with no restrictions and no permission necessary, but don't mislead people, and don't use my non-code work to directly compete against the service that I'm operating.

Despite all of that, here I am, on the eve of GDC, with a small but amusing content update for you.

More emotions.  Lots of little bug fixes.  Enjoy!

This update has tons of fixes and improvements.  The biggest one is an overhaul to the way the map is loaded.  You may have noticed that, in the past, the first time you loaded a map, it was pretty slow, but in later lives, it was very fast.  This would be true even if you quit the game, as long as you didn't restart your computer.

And by "pretty slow" the first time, I mean very slow, depending on the state of your disk.  60 seconds or more wasn't unheard of, which meant that you were loading through a good portion of your childhood.  This has gotten worse over time, as more sprites have been added.  Subsequent map loads in future lives would be as fast as 4 seconds, thanks to caching.

Reading files from hard drives the first time is slow, there's no way around that.  The game was designed with a lazy, as-needed approach to sprite loading, only keeping the sprites that are absolutely needed in VRAM, and flushing any sprites that haven't been drawn for over ten seconds.  The idea was that, with 10,000 objects, all those sprites are never going to fit in texture memory.  Maybe not, but we're not there yet, and the total size of all the sprites in the game is currently only about 56 MB.  In busier map areas, almost all of these need to be loaded, so we're pretty much using that much texture memory anyway.

It turns out that reading 56 MB from disk isn't slow, generally, but when it's in 1800 separate files, caching prefetches can't help.  Bundling all of these into one huge file makes it much faster, and so does compressing them (TGA files that have a lot of transparent borders are very compressible).  These all fit, together, into just a single 6 MB file.  Might as well load the whole thing at startup, which is what the game client is doing now.  While we're at it, might as well do the same thing with the sound effects (which aren't at all compressible, but still benefit from being in one big file together for caching reasons).

So by the time you get around to "map loading," after logging in, there's really nothing to load.  This means that a progress bar isn't even needed--it's that fast (most of the "3 seconds" quoted above are spent finding the server and connecting to it).

And thinking about the future, we're definitely not going to have 10x more sprites than we do right now, and that worst case would be 560 MB, which still would fit in the VRAM of some pretty old graphics cards.  It might actually be okay to always preload all sprites.

This isn't entirely free, because the compressed glob file has to be made somehow.  Given that, between sprites and sounds, this represents about 25 MB currently, and given that these files will change with every update, building them server-side would dramatically balloon the download sizes of the weekly updates.

So, your client rebuilds these, one time, after every update.  This can take a bit of time, maybe up to a minute, depending on your hard drive, but after that, the game will load quickly.  And furthermore, this process happens before you even login, so it has no impact on your map loading experience.

Okay, what else changed?  Too much to list in detail here.

Everything after v199 here:

https://github.com/jasonrohrer/OneLife/ … ngeLog.txt

Everything on February 22, 2019 here:

https://github.com/jasonrohrer/OneLifeD … its/master

All reported and reproducible code bugs on GitHub have been fixed now.  I'm still in the process of working through all the content bug reports.

What made this bug so hard to find and fix was the fact that it affected so few people, relatively speaking.  However, for the affected people, it affected them all the time, and pretty much ruined the game for them.

The symptom:  in busy areas, apparent network lag would grow and grow, resulting in up to twenty seconds of delay between trying to do something (like pick a berry) and have the action resolve (like have the berry in your hand).  On its face, this sounds like classic network lag.  The first thought is that the server isn't keeping up with demand.  However, other people playing in the same area were not experiencing lag.  In fact, the affected player would often ask about lag, in-game, and be told by others that there was no lag for them.  Also, if the server was being bogged down, the lag would be experienced everywhere in the game world, not just in busy areas, because all areas are processed in the same loop.

Maybe they were in a remote part of the real world.  Maybe they were on spotty WiFi.  The problem would often clear itself up instantly if they walked out of the busy areas.  And certainly, the server is sending them fewer messages out there, because it filters the messages based on what is relevant to your location.  In a busy city, you need to receive a lot of information, because so many people are walking around.  In the wilderness, there's much less change happening.  So this symptom was generally consistent with network lag.

A while back, I built a /PING and /FPS command into the game, so that people could run tests if they were experiencing lag.  Sure enough, during these lag situations, ping times would balloon.  Normal ping times in the US are below 100ms, and not more than 400ms anywhere in the world.  But during lag, the ping would grow to five, ten, or even twenty seconds.  That's really bad, and probably transcends any normal network lag.

And for these people, things have only gotten worse when we moved everyone to bigserver2.  Big cities are much more common, so many of the affected people were experiencing unplayable lag almost every life.  Of course, for everyone else---those who never experienced lag---bigserver2 was great.

But finally, almost miraculously, I experienced this issue myself for the first time this week.  A unicorn!  I was playing in a busy city, on my slow dev laptop with a weak GPU, and sure enough lag.  Bad lag.  Really bad lag.  My in-game ping time grew to more than 14 seconds.  The game was totally unplayable.

During this time, I also noticed that my FPS dropped from around 60 down to 40 or so.  Frame rate and network lag aren't necessarily related, but my lag was very hard to reproduce---it would come and go seemingly at random, even in the big city, depending on where I walked---and it seemed to be correlated with this drop in FPS.

I set up a chaotic Eve-only city on bigserver2 on Friday to conduct a real stress test.  120 players all spawning in the same spot (0,0) is no joke, and I could very consistently trigger lag on my slow dev laptop.

I also found that my gaming rig would not see lag in the same area, but it is running at a solid 85 FPS (odd, I know, but it's a CRT).  So, same network, different CPU and GPU, higher FPS, no lag.  So yeah, with proper hardware, the client can easily handle 120 players all in the same area.  It was chaos, but buttery smooth chaos.

Someone pointed out that outside-game-pings (using the command line) aren't necessarily slow during an in-game lag, and I was able to confirm this.  Someone else suggested that I sniff the raw network packets and figure out exactly how quickly the server was responding to my PING with a PONG---just to rule out server-side lag.  Sure enough, while my client took 14 seconds to register the PONG, the PONG arrived on the network within the normal 70 ms, even on the slow dev laptop.  There was some kind of networking issue inside the client.

I spent quite a bit of time testing my underlying networking code and looking for reasons that network messages might get backed up, but found no issue in isolated network tests.  I also considered some kind of kernel networking issue (my laptop is running Linux, while my gaming rig tests were on Windows7).  No dice.

Meanwhile, someone else had been able to pinpoint the exact problem in the client, and they posted their fix in an old, lingering Github issue.  Finally, someone drew my attention to this fix, which was rather hidden on the Github side.

JRuldolf, we all owe you one!

Turns out that this problem has been with us since an update back in October, before the Steam release, when message frames were added.  A frame groups messages together that occur during the same server processing step, forcing the client to wait to react to any of these messages until all the messages in the frame arrive.  This prevents, for example, a message about a map change from being processed before the matching player update is received (if a player dumps a bowl of water into a bucket, the bucket on the map changes, and so does the bowl in their hand, and there are two separate messages, but they only make sense if they occur client-side at the same time).

This frame system was great, and fixed a heap of potential inconsistencies in client behavior.

However, there was also a bug in the way that frames were processed.  Each client step (each rendering frame), the client would read the next message and check if it was an end-of-frame message.  If not, it would put the message in the holding queue and go on to the next rendering step.

You can see how this can cause trouble when message frames contain more and more messages (which they do in busy areas):  a frame with five messages takes at least five client frames to fully receive, even if all five messages have arrived, because we only check one message per frame.  Once the 6th message is checked, the end of frame message, we call the frame ready, and process all five messages together.

What we need to do, instead, is loop as long as any messages are available, checking for the end-of-frame message, but if it's not there, continuing on to the next message, until no more received messages are available.  Thus, we process all received messages every client frame, regardless of how long the message frame is.  This even allows us to process multiple server message frames on a single client rendering frame, if several server frames are waiting client-side.

If we don't do this, during times with high message rates and large, multi-message frames, we can see how a message backlog would build up.  Assuming, of course, that more than 60 messages were arriving per second.

And if the FPS drops on top of that, you can see how it would get even worse, because we are running even fewer processing steps per second.  So players with weaker GPUs were pretty much experiencing the perfect storm in busy areas.  Lots more messages, and a slower client-side rendering loop that was effectively only processing one message per rendering frame.

The fix was literally two lines, putting a loop in there where it should be.

And suddenly, the client could handle the very busiest areas with absolutely no network lag.  Even if I artificially reduced the frame rate to 5 FPS, the game was completely playable in busy areas (yes, it was choppy, but each action was executed instantly, with no lag).  Before the fix, such a low frame rate would spell disaster in a busy area.

Now, how did such a devastating yet simple bug go unnoticed for so long?  Well, as long as the frame rate is high enough, and the incoming message rate is low enough, it generally doesn't matter.  We're processing at least one message every frame, and 60 messages a second is a lot, so we usually keep up, even if we don't process all available messages as soon as we have them.  I didn't write the code this way on purpose---the original code, before message frames were added, intentionally processed all available messages every rendering frame.  But the implementation of message frames quietly subverted this intention.

The move to bigserver2 made this very rare bug less rare, because the cities got bigger, and the message rate higher, causing slightly more people to experience the issue.  Including, finally and thankfully, me.

Bug fixes take a long time, but they are worth it.  More bug fixes next week.  The plan is to get clothing and heating working in a more sensible way.

What we had:  players spread out onto three or four servers for load-balancing purposes.  During peak times, this was necessary to prevent any individual server from becoming too overloaded.  During off-peak times, we kept sending players to all the previously-active servers to avoid any one server dying out unfairly (see the earlier Population Stabilization update).  But this meant that during off-peak times, even with plenty of people still playing, the population on each server got a little thin.

What we want:  everyone playing on one server, together, all the time.

The problem:  CPU overload when populations get high results in lag for players, not to mention Linode sending me warning emails (these server nodes are virtual servers co-hosted on multi-core machines---I don't want to be a bad neighbor to other users who have virtual servers on the same host machine).

It has been a long time since I examined this problem in detail, so I wasn't really sure where the issue was, or if there even was an issue anymore.  I was keeping the server population caps relatively low to avoid lag at all costs while I worked on other things.

So, I needed to do some stress-testing and some profiling.  Server1, with its ancient, gigantic map that has maybe only been wiped once in the past eight months, was historically the biggest offender in this department, so it made the perfect candidate for a stress test.  How many people can we put on there before it chokes?

Does the database engine need another overhaul?

Well, it turns out that with the existing database engine (which was written from scratch for our purposes and heavily optimized by me many months ago), we could pretty much house all the active players on server1 with no player lag.  CPU usage, however, was going above and beyond what keeps Linode happy, though.  At one point, our externally-monitored CPU usage was over 120%.

How is that possible?  Well, it turns out that a virtual CPU consumes additional CPU resources on its host CPU, apparently overhead from the virtualization process itself.  So, while I was seeing server1 sitting happily at 60% internally, it was well over 100% as far as Linode was concerned.

By running a busy-wait test program in parallel with server1 on the same node, I was able to push my internal CPU (viewed through top) up to 100%, and that brought Linode's CPU measurement up to 140%.  Yikes.  This likely means that my virtual server is so resource-hungry that the virtualization process is itself consuming resources from more than one physical core.  I'm not sure of the details here, but that's my best guess.

Regardless, we want to steer WAY clear of 140%.

But the lack of lag when 170 players were together on the usually-bedraggled server1 was promising.

Were there any unnecessary hot spots left in the code that could be eliminated?   Maybe the database engine needs to be rewritten again.  Keeping the database in RAM is one idea that might speed things up, but who knows?

This is where profiling is supposed to help.

But existing profilers do a notoriously poor job at measuring actual performance issues in I/O-bound processes.  My server is likely spending a lot of time waiting for data from the disk.  Asleep, essentially.  Not running code, in the way that a profiler might measure, but still slow.

After testing every profiling tool under the sun, and finding nothing that worked for this purpose, I ended up writing my own.  More details about that, and proof that it works, and examples of why other profilers don't work, can be found here:

https://github.com/jasonrohrer/wallClockProfiler

Profiling a toy program with a toy profiler is one thing, but profiling an extremely complex, multi-faceted server process is quite another.  This made an excellent test case that helped me actually turn my toy profiler into a working, useable tool.  At some point along the line, I realized that the text data that the profiler was outputting (essentially annotated stack traces) was too tedious to read through by hand, so I even wrote a conversion program that allows the resulting profile to be viewed in the Kcachegrind profile visualizer.

With all that working, here is a rough visualization of where server1 was spending its time while hosting 155 simultaneous players:

Now, before you tell me that I've lost my mind, let me reassure you that such an image isn't all that useful in practice.  It's just the best way to quickly represent the complexity of the profile visually.  In reality, I'm looking at sorted lists of functions and the amount of samples that hit each function.  But a screen shot of that doesn't make for a very interesting picture.

Anyway, from that image, we can see what looks like a pretty "clean room."  That big "empty space" in the middle is indeed empty space:  time the server spent waiting on epoll for incoming client messages.  We're doing that 54% of the time.  The rest of the clutter around the edges of the room is actual work being done.

The biggest forehead-slapper in the profile, which can actually be seen here in this image, is the 12% of our running time spent on recomputeHeatMap.  This is the bit of code that examines the environment around you to determine how cold you are (the thermal propagation simulation).  This is an expensive bit of code to run, but it's only supposed to be updated for two players every server step (thus spreading the load), so what's going on here?

It turns out that the wall-clock duration of a "server step" varies depending on the rate at which messages are arriving.  Big gaps between messages means the server sleeps longer before executing the next step.  Short gaps mean many steps happen in a short time.  The server is intentionally player-reactive in this way, actually using almost no resources at all if no on is logged in.

Checking the logs, I found that with such a huge population of players, with such a high inbound player message rate, the server step was being run something like 65 times per second.  Yikes.  Not only did this result in excessive calls to recomputeHeatMap (recomputing maps for something like 130 players every second, which isn't even useful), there were a bunch of other regular-interval parts of the server step that were being triggered 65 times per second as well.  We don't need to check whether a player's curse score is decremented 65 times a second, for example.

After finding the parts of the server step that weren't necessarily reactive, I put them on fixed timesteps so that they would only run if enough time has passed, not every single step.  Heat maps are now limited to 20 players per second, max, for example, regardless of how quickly messages are coming in.

The results are pretty dramatic.  Here's the new profile picture, after these changes, with about 150 players on server 1:

And here's a 30-minute monitor graph of both old and new (sampled every 5 seconds, for 360 samples total):

Yes, that's around half the CPU used per player now.  This should allow us to double the number of players that occupy a given server.

But even so, when we start getting above 60% internal CPU, external resource consumption can get up into the 90% range, which does not make Linode happy.

However, they did inform me that 2-core nodes (which are more expensive) are allowed to go up to 160% utilization, and 4-core nodes are allowed to go up to 320% utilization.

The server code is single-threaded, so it can't take advantage of more than one physical core directly, but the external resource consumption from virutalization, including disk access and so on, apparently can.

So, today I introduce a brand new server on the front line:  bigserver1.onehouronelife.com

2 cores, 4x the RAM, a bigger disk, and a bigger upstream network pipe.  Most of these extra resources aren't needed, but the extra core may help with external resource usage.  Four times the cost, though.  Is it worth it?  How many players can we put on this sucker before it starts to choke?

To give you a taste of the difference between internal and external resource consumption on a virtual server, bigserver1 currently has 155 players on it.  Internally, in top, it is using less than 1% of its CPU.  Something around 0.3%, to be exact.  Hard to believe, but true.  A fresh---and tiny---map database likely helps with this, for sure.

But externally, as far as Linod is concerned?  50% CPU.  Granted, I can safely go up to 160%, but still, 50% is way different than 0.3%.  My external networking and disk access graphs are relatively high, though, and my guess is that some of those aspects contribute to external CPU usage.  Again, my guess is that the process of virtualizing networking and disk involves extra host CPU operations that wouldn't be necessary on non-virtual hardware.

As another example, if I run a pure-CPU test process that busy loops, I see both 100% internally and externally, but that's a process that isn't touching the disk or network at all.

So, over the next few weeks, we'll see where bigserver1 can take us, in terms of a large population of players all in one cohesive world.

.-. .- -.. .. --- / - .-. .- -. ... -- .. ... ... .. --- -. --..-- / -.-- --- ..- / ... .- -.-- ..--.. / - .... .- - .----. ... / ... --- -- . / .--. .-. . - - -.-- / .... .. --. .... -....- - . -.-. .... / ... - ..- ..-. ..-. --..-- / .. ... -. .----. - / .. - ..--.. / - .... . / -- --- ... - / -... .- ... .. -.-. / ..-. --- .-. -- / --- ..-. / .-. .- -.. .. --- / - .-. .- -. ... -- .. ... ... .. --- -. / .. ... / .- -.-. - ..- .- .-.. .-.. -.-- / .--. .-. . - - -.-- / .-.. --- .-- / - . -.-. .... ---... / .- / ..-. . .-- / -... .. - ... / --- ..-. / -.-. --- .. .-.. --..-- / .- / ..-. . .-- / -... .. - ... / --- ..-. / ..-. --- .. .-.. --..-- / .- -. -.. / .- / -. . ... - . -.. / ...- .- - / --- ..-. / -.-. .... . -- .. -.-. .- .-.. ... / - .... .- - / .... .- ...- . / -... . . -. / -.- -. --- .-- -. / ... .. -. -.-. . / .- -. -.-. .. . -. - / - .. -- . ... / -.--. .. -. / .- .-.. -.-. .... . -- .. -.-. .- .-.. / - . .-. -- ... --..-- / .- / -... .. - / --- ..-. / .-. --- -- .- -. / ...- .. - .-. .. --- .-.. --..-- / .- -. -.. / ... --- -- . / ... .--. .. .-. .. - / --- ..-. / ...- .. - .-. .. --- .-.. -.--.- --..-- / .- -. -.. / .- .-- .- -.-- / -.-- --- ..- / --. --- .-.-.- / -.-- . ... --..-- / .. - .----. ... / -. --- .. ... -.-- / .- -. -.. / -... .-. --- .- -.. -....- -... .- -. -.. --..-- / .- .-.. -- --- ... - / .-.. .. -.- . / - .... . / .-. .- -.. .. --- / . --.- ..- .. ...- .- .-.. . -. - / --- ..-. / ..-. .. ... .... .. -. --. / .. -. / .- / .--. --- -. -.. / .-- .. - .... / .- / ... - .. -.-. -.- / --- ..-. / -.. -.-- -. .- -- .. - . --..-- / -... ..- - / .. - / --. . - ... / - .... . / .--- --- -... / -.. --- -. . --..-- / .- -. -.. / .. - / .-- .- ... / - .... . / ... - .- -. -.. .- .-. -.. / -- . - .... --- -.. / --- ..-. / - .-. .- -. ... -- .. ... ... .. --- -. / ..-. --- .-. / .- -... --- ..- - / - .... .. .-. - -.-- / ... --- .-.. .. -.. / -.-- . .- .-. ... / -.--. ..-. .- -- --- ..- ... .-.. -.-- / ..- ... . -.. / - --- / .. ... ... ..- . / - .... . / -.. .. ... - .-. . ... ... / -.-. .- .-.. .-.. ... / ..-. .-. --- -- / - .... . / ... .. -. -.- .. -. --. / - .. - .- -. .. -.-. -.--.- .-.-.- / .-- .... .- - .----. ... / ... --- -- . .-- .... .- - / .- -- .- --.. .. -. --. / .. ... / .... --- .-- / .-.. --- -. --. / .. - / - --- --- -.- / ..-. --- .-. / .--. . --- .--. .-.. . / - --- / -.. .. ... -.-. --- ...- . .-. --..-- / --. .. ...- . -. / .... --- .-- / ... .. -- .--. .-.. . / .. - / .. ... .-.-.- / .-- .... .- - .----. ... / . ...- . -. / -- --- .-. . / .- -- .- --.. .. -. --. / .. ... / .... --- .-- / -.-. .-.. --- ... . / - --- / -- .- --. .. -.-. / .. - / .. ... --..-- / --. .. ...- . -. / .... --- .-- / ... .. -- .--. .-.. . / .. - / .. ... .-.-.- / .. -. ...- .. ... .. -... .-.. . / -- . ... ... .- --. . ... / .. -. / - .... . / ... -.- -.-- --..-- / -- .- -.. . / .--. --- ... ... .. -... .-.. . / - .... .-. --- ..- --. .... / - .... . / ... .--. . -.-. .. ..-. .. -.-. / .- .-. .-. .- -. --. . -- . -. - / --- ..-. / .--. .-. .. -- .. - .. ...- . / . .-.. . -- . -. - ... / -... .- ... . -.. / --- -. / .- .-. -.-. .- -. . / -.- -. --- .-- .-.. . -.. --. . --..-- / .- -. -.. / -.. --- -.-. ..- -- . -. - . -.. / - .... .-. --- ..- --. .... / ... - .-. .- -. --. . / -.. .. .- --. .-. .- -- ... .-.-.- / .. -. / - . .-. -- ... / --- ..-. / - .... . / ... .... . . .-. / -. ..- -- -... . .-. / --- ..-. / --- -... .--- . -.-. - ... / .- -.. -.. . -.. --..-- / - .... .. ... / .. ... / --- -. . / --- ..-. / - .... . / .-.. .- .-. --. . ... - / ..- .--. -.. .- - . ... / .. -. / - .... . / .... .. ... - --- .-. -.-- / --- ..-. / - .... . / --. .- -- . / ... --- / ..-. .- .-. .-.-.- / .. / --. ..- . ... ... / - .... .- - / -- .- -.- . ... / .-. .- -.. .. --- / .--. .-. . - - -.-- / -.-. --- -- .--. .-.. .. -.-. .- - . -.. --..-- / .-. . .-.. .- - .. ...- . .-.. -.-- / ... .--. . .- -.- .. -. --. .-.-.- / .... --- .-- . ...- . .-. --..-- / .-- .... . -. / .. / - --- .-.. -.. / -- -.-- / .---- ..... / -.-- . .- .-. / --- .-.. -.. / ... --- -. / -- . --.. / .- -... --- ..- - / - .... . / -.-. --- -. - . -. - / .. -. / - .... . / ..- .--. -.. .- - . --..-- / .- -. -.. / - .... . / .-- .- -.-- / - .... .- - / .. - / .-- --- ..- .-.. -.. / .-- --- .-. -.- --..-- / .... . / ... .- .. -.. --..-- / .-..-. -.-- . - / .- -. --- - .... . .-. / -... .. --. / ..- .--. -.. .- - . / ..-. ..- .-.. .-.. / --- ..-. / ..- ... . .-.. . ... ... / - . -.-. .... -. --- .-.. --- --. -.-- -.-.-- / .... --- .-- / .- .-. . / .--. . --- .--. .-.. . / . ...- . -. / --. --- .. -. --. / - --- / -- .- -.- . / ..- ... . / --- ..-. / - .... .. ... --..-- / -... -.-- / .-.. . .- .-. -. .. -. --. / -- --- .-. ... . / -.-. --- -.. . ..--.. .-..-. / -... ..- - / --. .. ...- . -. / - .... . / -.-. --- ..- .-. ... . / --- ..-. / .... ..- -- .- -. / .... .. ... - --- .-. -.-- --..-- / .. - / ... . . -- ... / ... - .-. .- -. --. . / - --- / -- . / - --- / -.-. .- .-.. .-.. / - . .-.. . --. .-. .- .--. .... -.-- / ..- ... . .-.. . ... ... .-.-.- / .-- .... .- - / .... .- - .... / --. --- -.. / .-- .-. --- ..- --. .... - ..--.. / .--- .- ... --- -.

First, a few important fixes that you all should be aware of.

There was a bug in temperature weighting on mothers.  It was supposed to make ideal-temp mothers more likely to have a baby, but it was broken and not working.  That has been fixed now.  Furthermore, a Yum multiplier factor has been added to this weighting.  If you have a large Yum multiplier (from eating a chain of unique foods), you will also be more likely to have a baby.  If you're warm in addition to being on a yummy diet, you will be even more likely to have a baby.

And the way that Eve spawn locations were remembered---when Eve died of old age---was buggy.  Thus, the surprise appearance of Eves near villages.  This has been fixed.  But even when it's working correctly, it's meant to only function on low-pop servers, and not as a way-of-life for reviving collapsed villages, so that has been fixed as well (your last-Eve-death location will only be used for your next Eve spawn if there are fewer than four fertile females on the server currently).  This fix is even more important in light of this week's update, which I will describe in detail below.

In last week's update, I talked about how there will be no magic in the game.  What I meant to say is that there will be no non-inherent magic.

Some things about the game are inescapably magic.  Reincarnation---a reality for any commercially viable game---is the prime example of this.

But the map itself, and the servers, and how they get set up, and how they get updated, and how they get cleared, is another example.  I'm doing all this stuff behind the scenes to keep things updated and working.  I'm making choices.  I'm adding things.  I'm in control of the parameters that control when and how certain parts of the map go back to their natural state.

And the map is huge---unnaturally huge.  36,000x larger than the surface area of the earth.  Walking from one edge to the other in the game would take you 34 years of real-life time.  Walking around to visually see the entire map would take you 14 billion years.

It's a big map.  Mind-mindbogglingly so.  Yet I can change the entire thing with the push of a button, like when I add a new biome, or wipe it back to its natural state in the blink of an eye.  How can something so big be changed so fast?  Through procedural generation and the properties of computer file systems (where deleting data of length N is a constant time operation on N).  It's not magic, really.

But when we try to square these possibilities with a simulation of the real world, the end results are nothing short of miraculous.

And what does that make me, the guy pushing the buttons behind the scenes?

There's an amazing idea lurking in this game, and credit for the idea goes to Edmund McMillen.  When I visited him a few years ago, in between petting his hairless cat and having him kick my ass in a Magic draft, I told him about the game I was working on.  In a game that starts back at zero as a premise, a question arises:  how did we get to zero in the first place?  And what if, Edmund suggested, players were in control of taking everything back to zero?  What if, at the top of the tech tree, the most difficult-to-craft item was The Button?

It seems that, after all is said and done in this game, after all my updates are out, and the game stops evolving due to developer input, this just has to be the way that it will work.  Otherwise the game will stagnate.  Edmund was right.

But what about along the way?  In the arms race of player progress in the face of my weekly updates, players always win.

So, the idea of an along-the-way apocalypse arose.  What if The Button was a moving target?  Some item at the top of the current tech tree that represented the current endgame?

The problem here is that players can get to the top of the tech tree ridiculously fast.

This means that the apocalyptic item can't be technological.  It needs to be magic in some way.

Long ago, shortly after the game's release in early 2018, I tried something like this.  A monolith in the desert that you could use to conduct a kind of absurd ritual using a bit of material that was high-level tech at the time.  This experiment was an utter failure, as the first apocalypse was triggered four hours after the update, and subsequent apocalypses were triggered hourly after that until I gave up and disabled the whole system.

I left that failed experiment behind, without thinking about it any further.  Players can get through the tech tree---and craft any imaginable thing---way too fast.  This even planted seeds of doubt in my mind about Edmund's Button, even at the end of the update process, once the tech tree was gigantic.

Still, I really liked the shared collective event that had occurred.  People who were playing that fateful day will never forget that flash of white...

In the mean time, other ideas surfaced, like the bell tower, which involved slowing down player progress toward a goal and ensuring trans-generational cooperation.  A bell tower takes 18 hours to build.  In order to build it, your village has to survive that long.

This takes a page from the Clock of the Long Now.

The insight this week was that these two ideas can be combined.  An apocalypse, for the time being, is a magical, not technological event.  So there's a ritual.  What if it was a very slow ritual?  What if people had plenty of opportunity---and warning---to interrupt the ritual before completion?

That was always the idea with Edmund's Button anyway---that people would be fighting to stop it along the way.

So, I give you a new and improved apocalypse.  It has:

--Rare, unsustainable ingredients that you cannot procure while working completely alone.

--A map-wiping wave that is limited to one server only (no more chance of an Easy Apocalypse on a vacant server causing wipes on the populated servers).

--A map-wiping wave that you can live through and come out the other side.

--World-wide warnings as the ritual gets closer and closer to completion.

--The ritual itself is very fragile and easy to set back along the way.

--The entire ritual, if uninterrupted, takes 24 hours to complete.

And, for the time being at least, it's magic.

Sheesh, an internal combustion engine has a lot of moving parts.  I should know, because I just drew them all.  There was so much detail that I had to lay out the whole thing in CAD software first, as printed a tracing guide.  It's all still hand-drawn.  Call it Computer Aided Human Drawing.

So I drew it, and now it's up to you to put the damn thing together.

The internal combustion engine was actually a major sticking-point in the design of the game:  how do we get over the very steep hump that leads into industrialization?  Like I mentioned in a previous update, the actual history here is far from clear.  We went from very crude machines that were mostly made out of wood and powered by animals, water, or humans to finely crafted clockwork contraptions that could literally pump like well oiled machines.  My guess is that it was a process of micro-refinements over about five hundred years.

So, I kinda just winged it here, assuming that if we had something spinning fast enough, that would be enough to bootstrap the whole thing via the magic of the lathe.  And here we are, a week later, with a pretty accurate model of a four-stroke, two-cylinder diesel engine, complete with all major parts.

If you're interested in more details about how this works, this video explains the working of a single cylinder:

https://www.youtube.com/watch?v=fTAUq6G9apg

And this weirdly-narrated video explains how the camshaft ties the whole thing together in terms of timing:

https://www.youtube.com/watch?v=DZt5xU44IfQ

My gosh, humans are clever!

The major thought experiment in this game is this:

"It took us 4000 years to advance from stone-aged tech to the iPhone the first time around.  If we had to start over from scratch, naked in the wilderness, with nothing but rocks and sticks, but we retained all knowledge, how long would it take the second time?"

The more closely I study this stuff, the more baffled I am about how we ever did it in the first place.  How long would it take the second time?  My current best guess:  Forever.

As in, never.

This update paves the way for the forthcoming industrial revolution.  After banging my head against the actuality of human history in this area (much of which is, strangely, shrouded in uncertainty) for four straight hours on Monday, I realized that bootstrapping is hard.  This is the point where the true mystery of human civilization comes to a head:  how do you make a lathe without a lathe?

We've come a long way so far, and bootstrapped a whole bunch of things in this game.  But these are all things that I myself know how to make from scratch, in principle.  This is the kind of stuff that is made on the Primitive Technology YouTube channel.  If it looks hand-made, it can probably be made by hand, right?

But how do you bootstrap metal machines?  I'm starting to suspect that this is something that no one now living actually knows how to do.  So it's my job to figure it out, or at least make up a reasonable approximation.  I was thinking that we'd be skipping right over the age of steam---that steam was just an unnecessary side-branch on in the inevitable path toward internal combustion.  But now I think it's actually about tolerances.  Internal combustion requires extremely tight tolerances, and metal gaskets, because there's fire right there in the cylinder.  Steam engines can be made with much lower tolerances, and leather or rubber gaskets, because the fire is kept outside the cylinder.  I.e., crude steam machines can be made without the process of machining.  And with steam machines, we can make the machines with which we can actually "machine" parts with tighter tolerances.  Steam lathe, here we come.

And speaking of steam lathes, this video of a 1900's era steam-powered machine shop is pretty amazing.  But none of those belt-driven machines contain parts that were made by a blacksmith:

https://www.youtube.com/watch?v=9WXHNBMLZZM

But there's no lathe in there yet.  What can you do with this Newcomen Atmospheric Engine?  Pump water!  Why would you want to do that?  You'll find out soon enough.

This update also dramatically improves the mouse interface when dealing with stacks of things that should also be moveable as a stack (stacks of plates, for example).  Left click grabs the whole stack, and right click removes an item from the stack.  In other words, the stack now behaves similar to a container when you left or right click on it.

New wild wounds (or sickness) no longer replace your current wound.  You can't heal from a snake bite by contracting yellow fever or getting hog cut.