Marcos Ashton Iglesias

What I learned writing thirty PRs against a framework I had used for years.

2026-05-01T00:00:00Z

Why did I end up doing this?

I started contributing to Keras because the PyTorch backend was not doing what I expected. I had a model that trained fine on TensorFlow and produced wrong gradients on PyTorch, and I wanted to understand why. The polite version of what happened next is that I filed an issue. The honest version is that I started reading the source instead.

That was thirty merged PRs ago. I have shipped fixes across recurrent layers, attention, linear algebra, ONNX export, conv shape inference, and the CTC decoder. I have also had several PRs closed, which taught me more than the merged ones.

What is Keras 3?

The short version is that Keras is a Python API for building neural networks. The longer version is that Keras 3 is the same API on top of five different numerical backends: JAX, TensorFlow, PyTorch, NumPy, and OpenVINO. You write a model once, you can train it on any of them, and the layers are supposed to behave identically.

That last part is where the work is.

If you write a custom op like keras.ops.pinv, it has to lower correctly to JAX, to PyTorch, to TensorFlow, to NumPy, and ideally to OpenVINO too. Each backend has different conventions for dtype promotion, broadcasting, error messages, and dispatch. Each one has subtly different LAPACK bindings. PyTorch’s default least squares driver interprets the rcond argument differently from NumPy’s. JAX’s experimental cuDNN binding only exposes LSTM and not GRU. The Keras layer has to paper over all of that without leaking the seams to the user.

This is what I underestimated when I started. I thought the work would be machine learning. It is mostly plumbing.

What I did not expect

I expected the hard part of contributing to be the algorithms. The first time I sat down to add a CTC beam search decoder for the PyTorch backend, I assumed the difficult work would be the search itself, the log space score accumulation, the blank token handling. It was not. The hard part was that torch.unique is slower than I expected, and the bottleneck of the entire decoder turned out to be a deduplication step that happens on every timestep. I rewrote it to use a hash of each path instead of a sort, and the runtime dropped by a factor of three.

That kind of thing kept happening. The most useful PR I have written for Keras was probably the fix to the PyTorch backend’s lstsq. The bug was that the test was flaky on torch only when an explicit rcond was passed. The reason was that torch defaults to a QR based driver called gelsy, and gelsy’s interpretation of rcond is not the same as NumPy’s SVD based gelsd. On a near rank deficient input, the result could differ from NumPy by an order of magnitude. Pinning the driver to gelsd when rcond is supplied fixed it. The whole patch was three lines plus a comment explaining why.

I had to go read LAPACK documentation to figure that out. I had not expected to read LAPACK documentation.

The review process

The Keras team’s senior reviewer leaves substantive feedback on almost every external PR, and over the course of thirty merges I started to see his patterns. Do not validate inputs in the backend independent ops layer. Trust the backend to raise its own errors. Do not write helper functions that wrap a single backend call. Imports go at the top of the file unless you are working around a circular dependency. Do not put validation only in compute_output_spec, because that method does not run in many real call paths.

None of these rules are wrong. All of them are obvious in retrospect. None of them are obvious if you are coming from a codebase where validation lives in the wrapper.

I started keeping a list of his comments after the first few PRs got the same kinds of corrections. By the time I was on the twenty fifth PR, I was checking my own diff against the list before I pushed. The bounce rate dropped noticeably. I would like to note how streamlined and painless the contribution pipeline is in Keras compared to other OSS projects.

The part that made it click

There is a moment when you realise that the interesting work in a framework like Keras is not in the layers. It is at the seams. The seam between the JAX backend and the cuDNN binding, where you discover that the gate ordering for an LSTM has to be transposed before the call. The seam between the symbolic shape inference and the concrete dispatch, where a layer that works fine in eager mode breaks in tf.function because one path checks shapes and the other does not. The seam between two backends that handle dtype promotion differently, where a model that trains fine on one produces NaNs on the other.

Most of the bugs I have fixed are at seams. Most of the value of a multi backend framework is that someone else has already absorbed the cost of those seams. Most of the cost of maintaining one is that those seams keep moving.

That is not a clean lesson, and I am not going to pretend it is. What I have instead is a different appreciation for the people who maintain frameworks at this scale. Building a model is a few lines of Python. Making sure those lines do the same thing on five different runtimes, across a decade of accumulated edge cases, is a job.

If you want to look at the project, the Keras repository is at github.com/keras-team/keras. The contribution guide is good. The review bar is high but fair, and the reviewers actually read the code.

More coming, including a writeup on the JAX side once I have shipped a PR there.

FFmpeg is everywhere. Almost no one knows what it is.

2026-03-23T00:00:00Z

Why did I end up doing this?

I started working on FFmpeg because I wanted to qualify for Google Summer of Code. The task was to add FATE test coverage for a decoder that had none. I figured it would take a weekend.

It took considerably longer than a weekend. But what surprised me was not the difficulty. It was what I found when I started reading the codebase. FFmpeg is not a tool. It is infrastructure. The kind that sits underneath everything and gets noticed by nobody.

What is FFmpeg?

FFmpeg is a collection of libraries for encoding, decoding, muxing, demuxing, filtering, and streaming audio and video. It handles hundreds of codecs. It has been in active development since 2000. The command line tool is the part most people have touched, if they have touched it at all, usually to convert a video file with a command they copied from Stack Overflow and did not fully understand.

That description undersells it badly.

When you upload a video to YouTube, FFmpeg is involved in transcoding it. When you watch something on a streaming platform, the player almost certainly links against libavcodec or something built on top of it. VLC is built on it. Chrome uses it. Blender uses it. Kodi uses it. Most of the video infrastructure on the internet either is FFmpeg or depends on something that is. There is a famous line in the project’s documentation that says it plays “everything”. That is not quite true, but it is closer than it sounds.

The codebase has been accumulating since before most current contributors were in secondary school. There are codecs in there for formats that no longer have any hardware to play them. There are comments in languages the primary maintainer does not speak. The git history is a geological record.

What is FATE?

FFmpeg has a test suite called FATE, which stands for FFmpeg Automated Testing Environment. The way it works is simple. You run a decode or encode operation on a known input, compute a checksum of the output, and compare it against a stored reference. If the checksum matches, the behaviour is correct. If it does not, something changed.

The problem is coverage. FATE has thousands of tests, but they are not evenly distributed. Some codecs have dense coverage. Others have none. A decoder can be in the codebase for years without a single test to verify it behaves correctly after a refactor.

That is what I was working on. Finding decoders with no FATE coverage, locating or generating suitable sample files, writing the test entries, and submitting them for review.

What I did not expect

I expected the hard part to be the codec internals. It was not. The hard part was the infrastructure around the tests themselves.

FATE tests are not checked into the repository. The sample files live on a separate server at samples.ffmpeg.org, accessed via rsync. Each test entry references a file by path and expected checksum. Writing the test means finding a valid sample, verifying the decoder produces consistent output across platforms, computing the reference checksums, and formatting everything correctly before sending a patch.

The review process is similarly careful. My first patch came back with formatting corrections. My second came back with a question about whether the sample was the smallest file that exercised the relevant code path. Both pieces of feedback were correct. The bar is high because the test suite runs on a lot of machines and a bad entry is expensive to fix later.

What I kept noticing was how much of the work was not the code. It was the archaeology. Understanding why a codec works the way it does often means reading a spec from the 1990s, tracing back through commit history, or cross-referencing with a format that no longer has a Wikipedia page. The codebase is a record of every weird decision the video industry made over twenty-five years, preserved in C.

The part that made it click

There is a moment when you realise that software this old and this widely used is still being actively improved by volunteers. Not maintained. Improved. New codecs, new optimisations, new hardware interfaces, new test coverage for things that have been untested for a decade.

Most open source projects at this scale become ossified. The contributors burn out, the bus factor drops to one or two, and the project enters a slow maintenance phase until someone forks it. FFmpeg has not done that. The mailing list is active. The commit rate is high. Michael Niedermayer has reviewed more patches than I have written lines of code in my life.

I do not have a clean lesson to wrap this up with. The project is too large and too old for a neat conclusion. What I have instead is a patch in the tree and a better understanding of what it means for software to be load-bearing. Most of the code that matters is code you never think about. FFmpeg is the clearest example of that I have found.

If you want to look at it, the FFmpeg repository is at git.ffmpeg.org. The FATE documentation is sparse but functional. The mailing list archives are surprisingly readable.

Lattice - The gap between planing and implementing

2026-03-13T00:00:00Z

Why did I end up doing this?

I have been thinking about fluid dynamics for a while, mostly because I wanted to understand why some CFD software costs as much as a car. The answer, I suspected, was not the physics. The physics is settled. The answer was the engineering around it, the pipeline from equations to something a person can actually interact with.

So I built Lattice. It is a GPU-accelerated wind tunnel that simulates airflow around 3D objects and computes drag coefficients in real time. You give it a model, it gives you a Cd readout and a visualisation. The code is on GitHub at github.com/MarcosAsh/3dFluidDynamicsInC. There is a live version at fluid-sim-frontend.vercel.app.

What is a Lattice Boltzmann simulation?

Most CFD solvers work by discretising the Navier-Stokes equations directly. That works, but it is hard to parallelise and hard to handle complex geometry with. Lattice Boltzmann takes a different approach. Instead of tracking the macroscopic fluid as a continuous field, you track the statistical behaviour of imaginary particles bouncing between nodes on a grid. At each timestep, particles collide and stream to neighbouring nodes according to a simple local rule. The macroscopic quantities you care about, velocity and pressure, emerge from the averages.

The reason this matters for GPU computing is that the collision and streaming steps are local. Each node only needs to know about its immediate neighbours. That is exactly the kind of workload that maps cleanly onto thousands of shader invocations running in parallel. A Navier-Stokes solver has global pressure solves that are hard to vectorise. A D3Q19 LBM grid is embarrassingly parallel at every step.

What did I build?

The simulation runs entirely on the GPU via OpenGL 4.3 compute shaders. The LBM solver runs the collision and streaming steps on the grid. A separate shader advects hundreds of thousands of particles through the resulting velocity field and detects collisions against arbitrary OBJ meshes. The drag coefficient is computed from the momentum transfer at each timestep and displayed in real time.

For desktop use, the binary runs with SDL2, a camera you can orbit, and keyboard shortcuts for switching between seven visualisation modes including velocity magnitude, vorticity, and turbulence regions. That part was straightforward.

The interesting part was making it usable without asking someone to compile a C binary.

The cloud pipeline

I wanted the project to have a web frontend where you could configure a simulation, hit run, and get a rendered video back without installing anything. The way I ended up doing this was to deploy the C binary as a Modal worker: a Python script that spins up a cloud GPU container, runs the simulation headlessly using EGL rather than SDL2, encodes the output to video, and uploads it to S3. The Next.js frontend sends a POST request with the simulation parameters and eventually gets back a presigned URL.

What surprised me was how much of the work in that pipeline was not the simulation itself. The simulation is fast. A trained renderer on modern hardware can push the LBM timesteps and the particle system at a rate that feels instant. The cost is in the container cold start, the video encoding, the S3 upload, and the round trip latency. Most of the time a user spends waiting has nothing to do with the physics.

That is a version of the same lesson I keep running into. In the Saddle project, the slow path was never the arithmetic, it was JAX’s compilation overhead and the WASM runtime boundary. Here, the slow path is never the GPU kernel, it is everything around it. The interesting engineering is almost always in the plumbing.

What I would do differently

The current architecture spins up a fresh container for each render request. That is fine for occasional use but expensive and slow for anything interactive. The right approach would be a persistent warm container with a stateful session model, so the GPU stays hot between requests and only the parameter delta gets sent over the wire. I have not built that yet because the current version is good enough to demonstrate the project, but it is the obvious next step.

The other thing I want to add is a stochastic turbulence mode that makes the particle injection noisy in a physically plausible way. Right now the turbulence visualisation is derived from the velocity field but the particles themselves are deterministic. Making them stochastic would make the vorticity regions much more legible.

The part that made it click for me

The visualisation mode I find most useful is not velocity magnitude or vorticity. It is flow progress, a rainbow gradient by X position that shows how far downstream a particle has travelled. When you put an Ahmed body in the tunnel and switch to flow progress, you can see the wake cavity behind the car immediately. The region where flow stalls and recirculates is obvious in a way that a pressure plot is not. That is the moment where the project stopped feeling like a benchmark and started feeling like a tool.

If you want to try it, the live demo is at fluid-sim-frontend.vercel.app. Upload an OBJ, pick a wind speed, and watch the Cd readout settle. The code is at github.com/MarcosAsh/3dFluidDynamicsInC.

More coming, including the persistent session model and the stochastic turbulence mode. I will write about those when they are done.

How do optimisers perform in different languages?

2026-03-12T00:00:00Z

Why did I end up doing this?

I have recently started working on Keras, mostly because I have been having some problems with the PyTorch backend. I delved into the world of open source for the first time. What I found was that the best way to understand why something is slow is to build a smaller version of it yourself.

What is an optimiser?

An optimiser is an algorithm that tries to find the lowest point on a loss surface. In machine learning, that surface represents how wrong your model is. The optimiser’s job is to navigate it, step by step, until it finds somewhere flat.

The reason there are so many of them is that navigating a high-dimensional surface is hard, and different surfaces reward different strategies. SGD with momentum barrels through valleys but struggles when the terrain is uneven. Adam adapts its step size for each parameter individually, which makes it faster in practice but more expensive per step. L-BFGS uses curvature information to take smarter steps but falls apart at scale. Each one is a different bet on what the loss surface is probably shaped like. Nobody has won yet, which is why the list keeps growing.

What did I make?

That is what Saddle is. I wanted to see optimisers actually behave, not just read about their convergence properties in a paper. So I picked five loss surfaces, implemented six optimisers, and wired them up to a visualiser where you can watch two of them race in real time.
But the more interesting question turned out to be underneath the visualisation: what does it actually cost to run an Adam update step? I implemented the same equations four ways, in C, JAX, WebAssembly, and plain Python, and measured them head to head. The results were not what I expected.

The server-side result was not surprising. C ran 10,000 Adam steps in 138 milliseconds. JAX took 1746 milliseconds for the same thing, 12.6 times slower. JAX’s JIT compilation overhead is real, and at this scale, on a simple 2D update step, it never gets the chance to pay for itself. If you are running a toy optimiser on a toy surface, C wins easily.

The client-side result was the one that made me think. I expected WASM to be faster than JavaScript. That is the whole point of compiling C to WASM. Instead JavaScript won, by a factor of two. WASM finished in 4 milliseconds, JavaScript in 2.

It turns out modern JavaScript engines are extraordinarily good at optimising tight numerical loops. V8 will JIT compile a hot loop and produce machine code that is competitive with, and in this case faster than, the WASM runtime overhead. WASM wins in other scenarios, particularly with heavy memory operations and larger workloads, but for a small repeated arithmetic kernel in the browser, the JS engine does not give it room to breathe.

The lesson I took from both results is the same one the Keras codebase kept teaching me: abstractions have costs that only show up at the edges. JAX is a remarkable piece of software. So is V8. But neither of them is free, and knowing when you are paying the price is most of the work.

If you want to see it yourself, the live demo is at saddle-fawn.vercel.app. Drop two optimisers on Himmelblau, watch them find different minima, and run the benchmarks in your own browser. The code is on GitHub at github.com/MarcosAsh/Saddle. There is more coming, including a stochastic gradient mode and a saddle point escape demo that I think makes the whole thing click. I will write about those when they are done.

P.S. I hope you liked it! This was my first article on Substack.