When I design a data-intensive system, as Kleppmann put forth, I like to pay extra attention to three ideas:

• Reliability
• Scalability
• Maintainability

1. Reliability

What makes a system reliable

Before even answering that question, I want to paraphrase what Kleppmann says about reliability.

A system that works even in the face of adversity is a reliable system. We cannot make a system 100% reliable (yes, practically speaking).

If I were to explain it slightly differently, one could say the following about a reliable system:

A reliable system can prevent things that could go wrong (faults) like your database or an upstream service not responding from becoming a catastrophic failure (taking your whole system down).

You can also call such systems fault tolerant and/or resilient.

How does one make their system reliable

Good error handling. Plain and simple. Your system should tolerate things that can go wrong using your best judgement (remember, anything that can go wrong WILL go wrong). These faults should never be allowed to take your whole system down with them.

What causes faults

A litany of things, actually.

Software errors caused by human error (yea, unavoidable when working with humans), your AI agent can slop bugs into your system, cascading bugs from an upstream service could metastasize in your system. Hardware errors like disk failures (when you’re building on prem), your cloud infra provider can pull the plug on your replicas for various reasons (they tend to prioritise elasticity over full availability in some cases), k8s pods can go down by themselves due to stuff we put in our code, etc etc.

How can one gain confidence in their system’s reliability

Netflix’s Chaos Monkey is a popular tool people use to battle-test their systems for reliability. Do thorough testing - unit tests, end-to-end integration tests that play out all known error scenarios (can’t stress this enough).

When building the systems, think about assumptions very carefully, and document them so they get seen by the systems that interact with yours. This will help avoid unpleasant bugs from arising.

You will thank yourselves when you have good telemetry for your systems (during incidents).

Story time: Reliability

In 2023 (?) we had this system that pushed files from one AWS account to another (customers’ accounts) at scale. A proof of concept was built for it. Someone flipped the flags ON for this system accidentally, and hell broke loose. As you can imagine, the PoC did not have any error handling. A simple fault like a timeout (among other things) over a period of time took the whole thing down. More to the joke, there wasn’t enough telemetry on the service. So, we couldn’t do much about it other than guess what was wrong.

---

2. Scalability

Almost all software systems grow over time. They grow in terms of the volume of data they process, the complexity of that data, etc. While it is impractical and borderline idiotic to build a system with 3 users at Google’s scale, it often proves to offer awesome returns when it’s built in a way that’s easy to re-arch the same system for scale (think: meaningful abstractions).

How to build scalable systems

We can do it in these steps (more or less):

1. Identify the Load Parameters for your system.

These are requests/sec in your application server, ratio of reads vs writes in your storage platform, number of jobs processed per time unit in your background job processing system, etc.

[!IMPORTANT] Step 1 is very very important. Not identifying the correct load params can cause all the optimisations you try to make to have no effect at all.

2. Describe performance or how your system handles that load.

Ask yourselves these questions:

1. How does an increase in the load params affect performance when the system resources (CPU, mem, n/w, etc) stay constant?
2. How much do you need to increase resources to keep performance unchanged when load increases?

To answer these questions, we need metrics. Measure them in percentiles: p50, p95, p99, p999 as they tell us the distribution.

3. Define a scaling strategy

Based on your answers to the questions, usually, scaling up to a more powerful machine (vertical scaling) and scaling out to a larger number of less powerful machines (horizontal scaling) are the two natural choices. A good design usually combines both approaches, and more as the variables in the system demand. As your systems become more sophisticated or complex, you’ll hit a wall in scaling. That’s when you’ll need to re-think how you do things, and re-arch your system for that scale.

When you’re at this point, give emphasis on what your system does: reads, writes, volume of data, complexity of that data, time requirements, access patterns, and the like.

3. Maintainability

There’s an old saying that goes - the code you write is not for machines, it actually is for humans. This may not be 100% true today as almost 100% of our code is being written and also read by AI.

However, I firmly believe that, maintainability of your code/system is more important than ever today as codebases worked on by these AI agents can quickly become that “big ball of mud”. Having good rule systems in place for your AI agents around styling in your code goes a long way.

The complexity of your system can be efficiently managed by using abstractions. Hide complexity behind a facade, so the onlooker looks at what your system does, not how it does it. This can be done at individual repo level, and at a distributed systems level, where you abstract data models behind APIs.

Avoid, at all costs, tight coupling of modules, hacks for performance gains, and tolerating unclean code from AI.

In essence, the system you build should be useful. In order for it to be called useful, it should fulfil its functional (what it should do functionally) and non-functional (security, reliability, compliance, scalability, etc) requirements.

With these things consciously in mind, and in practice we can still enjoy the art of software engineering in this day and age where our agents do the fun part (of coding).