Coding the Earth with agents: Part One

The overview effect is a transcendental experience of awe around the Earth and our place in the universe, most famously experienced by astronauts in space. I’ve also felt it when looking up at the stars at a dark sky site, when observing a solar eclipse, or sometimes just when pondering the rise of civilization after the receding of the glaciers.

I wanted to explore that further, and also have a way to share this experience with others, through an interactive way of exploring the planet and its place in space. There’s nothing particularly unique about this; Space Engine would probably be the most straightforward way. I’ve experienced some moving videos based on it, such as this one about finding Earth in the universe (and an earlier version by another creator, too).

However, I also wanted to experiment with vibe-coding this: how good have the latest models become? Could I give them some prompts and come out with a beautiful simulation of the Earth on the other end? I wanted to try.

The game engine

While calling this a “game” would be a stretch, my idea was to have a simulation where you could explore the Earth and Moon, move around them, see their place in the solar system, and perhaps even walk around them in AR.

I decided to vibe-code this in C — namely, the more “modern” variant,¹ which lets you do things like this:

sg_pipeline_desc earth_pipeline = {
    .layout = { 
        .attrs = {
            [ATTR_vs_pos].format = SG_VERTEXFORMAT_FLOAT3,
        } 
    },
    .shader = earth_shader,
    .index_type = SG_INDEXTYPE_UINT32,
    .depth = {
        .compare = SG_COMPAREFUNC_LESS_EQUAL, 
        .write_enabled = true 
    },
    .cull_mode = SG_CULLMODE_BACK,
    .face_winding = SG_FACEWINDING_CCW,
    .label = "earth-pipeline",
};

Why a raw engine in C, and not something higher level or “nicer”? To be honest, I’m doing this for fun, and even if I’m not writing all of the code myself, I find it more fun to do things this way. It also forces the LLMs to keep the code simpler (most of the time), and keeps things closer to the metal, so I can explore different concepts, such as memory arenas.

I asked the LLMs to base this upon earlier work that I did in a prototype game of air hockey, and I also drew upon the following influences:

• Data-oriented design, first brought to my awareness by Mike Acton, for teaching me more about how the underlying hardware works.
• Handmade Hero, by Casey Muratori, for writing code with minimal dependencies and keeping things simple.
• Sokol, by André Weissflog, for making C feel nice, modern, and idiomatic.

It feels perhaps a bit blasphemous to take an “indie” approach here when the idea of AI writing software is still somewhat contentious for some, especially for those of us who have built our identities around software dev. I get that concern, and for now, the way I try to approach it is as another tool in our toolbox, while also trying to be aware of the pitfalls.

I also drew upon concepts like hexagonal architecture, which is just one expression of the idea that we should try to keep things organized nicely in layers, so the code doesn’t become one giant mess of spaghetti.

The agent framework

To do this, I started with a basic harness I had been experimenting with, based off OpenAI’s harness engineering framework that they published back in February. The ideas behind it:

• Keep a minimal AGENTS.md file.
• Store architectural and planning docs in Markdown files.
• The code and related scaffolding are built and managed primarily by agents.

There’s a lot more to it, but at the core, it’s supposed to help your favourite agent framework work more successfully inside your project. I also drew upon learnings from Anthropic’s harness design for long-running apps and The Shape of Things to Come by Steve Yegge (both parts 1 and 2), along with a lot of other reading on the topic.

Building upon the experience I gained while using agents in my work life, I settled on a system of three different agents:

1. Fable, the “boss”, would be the primary agent, responsible for architecture, overall planning, and verifying the work done.
2. Codex would be the primary executor, but I also gave Fable the choice of using Opus or Sonnet, and of passing in a model effort, based on its judgement.
3. A model different from the executor would do the first review of the work, so if Codex did the implementation, then Opus would do the review.

It’s easy to get into pedantic, nit-picky loops between the implementor and the reviewer, and the problem with this is that reactive fixes to reviewer comments can often seriously degrade the architecture, or add a bunch of useless tests that only assert or duplicate what the code is already doing.

To try and mitigate this, I have Fable handle all review comments critically before passing them back down to the executor.

Pitfalls

For the first pass, I wanted to get a view of the Earth in space, with the ability to spin it and turn it, like a globe. To keep it simple, I decided to just go with a basic texture from NASA’s Blue Marble Next Generation. For this, the AIs had to:

• Implement the meshes, texture shaders, and camera.
• Handle touch and pointer input for rotating the globe around.

It sounds simple enough, and for someone experienced, it probably would be, but I don’t do this type of work in my day job. That made it harder for me to direct and review the work that the AIs came up with. Some of the problems that we ran into:

• We kept running into a problem with a texture seam at the North Pole that even Fable kept blaming on the renderer. I also checked with Sol (GPT 5.6, the top model currently available in Codex), and both agents were absolutely convinced that they had fixed the problem, verified through unit tests.

  After some back & forth, I realized that the issue was with the texture itself, and that the renderer was fine. The AI agents couldn’t see this, because in a sense, they can’t actually see: even images get turned into a stream of tokens for them.²

• The input kept getting messed up. The globe would spin the wrong way, not enough, too much, or would get into janky glitches and jump around all over the place. No amount of asking the agents to add tests was able to fix this.

  I thought I might have to resort to reading up on some trigonometry and doing it manually, but I ended up collaborating with the agents on a JS prototype and fixing all of the issues there. With that second basis of comparison, the agents were able to finally get it right.

• I would burn through tokens rather quickly. Even with Fable working as the boss and Codex as the executor, I would hit my limits fast, and sometimes I would ask an Opus instance to become a substitute boss while waiting for the limits to reset.

At some point, I was thinking that even with Fable, the agents weren’t quite ready to take on graphics-heavy work like this. Would it end up being better to just do everything myself from scratch (or wait for the models to get better)?

The models can write code that is more verbose and not as architecturally sound as what a domain expert would do. They are prone to getting lost in minutiae and missing the forest for the trees. There’s also the bigger risk that I may not necessarily understand everything that the agents are doing, and the more I outsource to them, the less I’ll be able to understand.

However, while I have some knowledge and enjoy doing this as a hobby, I’m not a domain expert nor a math expert, so it’s really helpful when the model can write the code to project a flat map onto the sphere,³ in a way that at least looks correct and as good as anything I would have done manually:

The lighting and rendering are still flat, so the next step will be to try and add much nicer lighting. I’m also hopeful that this could be a good way to use the AI to improve learning, by seeing what it built and then studying the solution, reading best practices, and putting it together.

Next steps

So far, this project has been inspiring: taken in the best light, agents are going to make it easier to turn ideas into something tangible and make computing more accessible to more of us, as time goes on.

I’d like to continue working on this idea to see how far it can go. Next up is improving the rendering, and then adding more of the planets (including Pluto — in spite of designation changes, you’re still an honorary planet to me), and I think it would be really neat if we could walk around them in AR, to really get a feel for how vast space is.

I’ll take these agents as far as we can go, and hopefully will have more to share in future updates. Until then!

Footnotes: