22 February 2018
Salvatore Sanfilippo (antirez) · watch on YouTube ↗ · click any timestamp to jump the video
Machine-generated by an AI from the transcript and the top comments. Not my writing, and it may contain errors.
Thirty-four minutes of reading real Redis source in vim, with no slides. The premise is set out in the first minute; the design argument proper starts at 6:02 and the two best stretches are the buffering rationale from 20:02 and the fsync interaction at 26:06. This transcript was produced locally with faster-whisper rather than from YouTube's auto-captions, which mangle "Redis" throughout.
No command implementation anywhere in Redis calls anything like "log this to the AOF". Each one simply increments server.dirty to signal that it changed the dataset. Everything downstream — logging, replication, snapshot triggers — keys off that one number, so adding a command requires no knowledge of persistence at all.
call() in server.c executes every command without exception — scripting, MULTI/EXEC, ordinary clients. It reads the dirty counter before and after invoking the command's function pointer, and the difference tells it whether anything happened. One choke point is what makes the counter trick work.
PUBLISH needs to reach replicas, because their subscribers are waiting — but it must not bump the dirty counter, because that counter also drives snapshot scheduling and a busy pub/sub channel would trigger pointless saves. The resolution is a separate force-replication flag, which is what you end up needing once one counter serves two consumers.
The function named for feeding the append-only file does no I/O at all. It renders the command back into the Redis wire protocol and concatenates it onto a global buffer. Everything about disks happens somewhere else entirely.
The reason for that buffer: a single iteration of the event loop may read and execute commands from fifty clients. Issuing a write syscall per command would be enormously wasteful, so the protocol accumulates and is flushed once per full iteration — the persistence design is shaped by the concurrency model, not by anything about logging.
The sharpest detail in the talk. On Linux, calling write() on a file that has an fsync in flight does not land in the kernel buffer — it blocks until the sync completes. So under the every-second policy Redis checks whether a background fsync is still running and deliberately postpones its write rather than stalling the server.
Postponement is capped at roughly two seconds. Past that it goes ahead and accepts the block, and logs a message explaining that the previous second's fsync is still pending and the disk is too slow. Choosing to take the hit and then explaining why is a nice piece of operational honesty.
Under fsync-always the guarantee is that a client sees OK only after its write is durable. Honouring that per client would cap throughput at the disk's fsync rate. Instead the replies are withheld, a single fsync covers every client in the batch, and only then are the queued replies released — the guarantee preserved at a fraction of the cost. He closes by admitting this is genuinely hard to get right, given how it interacts with the event loop and command dispatch.
If a key is set to expire in an hour and the server stops, restarting thirty minutes later should leave thirty minutes remaining — not a fresh hour. So the log translates relative times into absolute Unix timestamps on the way in. A small thing nobody thinks about until it's wrong.
The dirty counter is incremented by the number of objects a command touched, because snapshot triggers care about how much changed. The append-only file only cares whether it moved at all. The same number answers two questions at different resolutions.
A counter that clients keep incrementing produces an endless run of increment operations in the log, all of which could be replaced by a single set of the final value. Tracking the file's size is what triggers that compaction.
A failed write under fsync-always is unrecoverable — the promise to clients has already been broken, so the server exits. Under weaker policies it sets an error flag and refuses new writes until the disk recovers, then carries on. The same failure is fatal or survivable depending on what was promised.
The log emits a database-selection command only when the target differs from the last one written, which is why one appears at the very start of a fresh file and rarely afterwards.
Unusually for a talk this good, there is almost no discussion underneath it — a couple of dozen comments, nearly all appreciation. Nothing to mine for counterarguments.
The most substantive suggestion is for a multi-hour recorded session implementing a real feature end to end, rather than a guided read of existing code.
An Italian commenter notes the algorithm decided it was finally time to recommend a seven-year-old video — a reminder that this material doesn't date the way framework tutorials do.
Auto-generated captions. Click any line or timestamp to seek; the current line highlights as the video plays.
0:01Hello, this is the first episode of what we will call Writing System Software following
0:15DHH example that did very interesting work in the other part of the stack basically.
0:26This is about system software, so we will see lower level things but the idea is exactly
0:33the same that to show in real source codes, not in small toy examples, how things work,
0:42why certain things were implemented in a given way.
0:47You know when software starts to become bigger then the very, very clean approaches start
0:55to fail, you have to face compromises in order to make things work and use little memory and
1:05for the code to be fast enough and all this stuff.
1:10Today we will talk about the Redis Append Only file and how it is implemented.
1:17The Append Only file is basically a feature of Redis that allows to create a file that
1:27will log every write operation performed by clients so that when the server will restart
1:34later it will read back all these operations and basically rebuild again in memory the
1:45same data set that hit before being stopped.
1:52So let's start Redis server with Append Only file enabled.
2:02In another terminal we can check the content of the file itself as Redis populates it.
2:14Let's open yet another terminal and execute some operation.
2:23Ok Redis basically executed a command and it's a client like any other client so we can see
2:36that in the Append Only file we have actually the Redis protocol generated in order to execute
2:46this command again when the server will be restarted.
2:49We can see that actually we executed set FooBar but there is also the select command because
2:58basically I could change the currently selected database so the select statement must be generated
3:08in order for the commands to target the right database and when the server is started the
3:14currently selected database is set to unknown so the first time the select statement will
3:21be created and I can just type another command and see the corresponding protocol to be created.
3:37This is just Redis protocol this means we have two arguments the first is 4 bytes argument
3:46the other is 9 bytes and so forth.
3:50Now if basically there is some debugging print depth because I was actually doing something
4:03about this codepad so let's compile again Redis for a second.
4:16So if I restart again the server the information inside the Append Only file will be processed
4:24again and I can see that we have back the values that we set in the previous session.
4:42Now how all this is implemented one interesting thing is that we don't actually have like
4:55the function calls in order to log inside the AOF inside every Redis command implementation.
5:04For example if I get the source code of the set command that's what we used as you can
5:18see here we will process the arguments and at some point we will perform the operation
5:26here it's implemented in terms of the set generic command because it has different entry points
5:34like set and exit expire and so forth there are different variants.
5:42So the set generic command implements the set operation with different operations and variants
5:50and here is the actual command implementation we set the key with the new value and then we
6:02perform this operation of incrementing this server dirty it means basically that this command
6:10performed some operation on the data set. So this is our only interface in order to
6:17to say to the Redis core that this command is of right command so it should be logged in in some way
6:29should be propagated and then we set the aspire and whatever and finally the command will send the reply
6:38to the client. As you can see there is nothing like I don't know log in AOF set and the arguments
6:48so how it's going to work internally? Well it's very simple to check because
6:57basically every command execution inside Redis is performed by a function inside server.c
7:11and this is the call function. Call is the core of the Redis execution of our command
7:20it has a bunch of flags because call is used in scripting or when you accumulate
7:28our transaction using multi and exec and everything that will at the end of the day execute a command
7:37will use call as interface so it must accommodate different needs and so there are many
7:44many flags but the function itself the prototype is very simple there is the client and there are the
7:55flags and what basically happens is okay let's use here as a whiteboard I have my client and then
8:08the protocol received by the client from the client is processed so that I will have the client
8:15argument vector and the client number of arguments processed inside the client structure
8:23that here in the call function it's called just c so I have c rb and c rc
8:38rb will be like a no set foo bar or something like that sorry too many notifications let's
8:50read telegram as well and rc will be three in this case so call knows what to call and
9:06it has a number of flags saying what it should do with this with this command exactly what other
9:13things to do other than executing the the command itself for example we can say that we want
9:24this command logged in the slow log and we want to populate statistics we want actually to write
9:32to the app and only file to replicate and so forth normally we just use command call
9:38full that means slow log start and propagate and let's say what what actually call will do
9:48so basically okay this is in order to send the command to clients having the monitor
9:56command enable in order to check what the other clients are doing then we basically initialize
10:03a bunch of things and this is the this is the core of calling a command in Redis what happens is
10:10that we basically set our dirty local variable to the current server dirty global variable
10:20and this is the same dirty that is implemented in the command implementation itself as you can see
10:29here we have server dirty so basically a global counter is the interface in order to see
10:38if the command produced or not right operation in the data set okay so basically we check the
10:47start time and we finally call the function pointer that's associated with this command
10:57c command is a function pointer that was resolved starting from the rv0 from the command so set is
11:07is inside the hash table of commands and it's resolved and if we are here in the call function
11:15it means that the command is already valid it exists the general form is valid like the
11:22minimum and maximum number of arguments and stuff like that then we calculate the time that
11:31the command took in order to be executed and at this point we can compute the difference
11:40in the dirty between the time before calling the function pointer implementing the command
11:49and the time and the the time after the command was called so what's actually this
11:58server dirty thing as we said it must be implemented when the command performed any
12:12operation but the amount we should increment it is proportional to the number of objects that the
12:21command affected so for example if i i don't know if i increment a key in redis then dirty
12:35will be incremented incremented by one but if i perform like a multi-set operation then dirty
12:44will be incremented by the number of focus effect so it should be like the mission of the number of
12:56changes that the data set received this is useful because the rtp persistence already says
13:05time and number of change trigger in order to understand if it's time to save a new snapshot
13:13so it uses dirty but from the point of view of the auf semantics the important thing is if dirty was
13:24incremented or not and we can see that okay we perform other things
13:36that are out of scope for our discussion here but the interesting thing is here at some point this
13:43command must be propagated it must be propagated in the auf file if enabled and it should also reach
13:54the slaves that are attached to our master and what happens here is that basically if dirty
14:03the propagate flags are set so that the command will be propagated to the auf and the replication
14:11sockets the slave sockets now i can have also flags that the commands can set in order to force the
14:24propagation of certain commands regardless of the fact that dirty was incremented or not
14:33for instance the publish command of pubsub it wants to be propagated to slaves
14:44because the slaves will can get subscribers to the same pubsub channels but at the same time
14:53we don't want to increment dirty in that case because we don't want the
15:00publish to affect the rdb saving semi-triggers so for example the publish command will set in the
15:08client flags client force replication and basically the interesting thing from our point of view
15:23is here if propagate flags is not equal to propagate known and then we actually want to
15:32propagate something and we call finally the propagate function that's the part that will
15:39really that the function that will really perform the propagation of this command so let's find where
15:46this function is okay propagate it's like two statements function
15:59uh feed up and only file and replication feed slaves so inside call we actually just have the logic
16:11in order to understand if we want to propagate this command to auf or replication but the actual
16:18the actual work of propagating is just these two function calls and we are interested in the first
16:27one feed up and only file and what it gets is the command that's a structure that describes
16:36the command that we executed the database id that the client was operating and writing to
16:47and the argument vector and the argument count and those are exactly the old client argument vectors
16:56and client argument count of the client that executed the command
17:06so we can check this function inside the file auf okay here it is feed up and only file
17:14so what here here is where basically this select command that we saw inside the auf file was generated
17:28if the current dictionary of the command executed is not equal to the latest selected database in
17:39the auf then we produce this protocol for selection of the database then there are special handling
17:48things about the expired command and set basically where there are relative times in commands we want
18:00to translate those into absolute times because if i set a key to have an expired of one hour
18:13and i stop the server and i then start the server again 30 minutes later i want virtually the time
18:21time continuing to pass so basically i register absolute Unix times inside the auf file so here
18:31there is a translation layer but basically at the end of the day what happens is
18:38cut up a only generic command inside this buffer and basically what means this means just to
18:46translate the argument vector into the argument vector into the redis protocol we can check
18:55this also well cut up and only generic command but the interesting thing here
19:03is that our function our feed up and only file function will just use a buffer it's not writing
19:16to disk it's writing nothing to disk we have this buffer at the end we produced our protocol inside
19:25the buffer and then what happens is basically that we concatenate this protocol that we produced
19:42inside a global buffer that's server.auf buffer so basically in this function we are just
19:52accumulating the protocol that we want to write to the disk inside a buffer so you may ask why
20:02don't write directly to our disk instead of having this buffer accumulating stuff
20:11well the reason is very simple so imagine redis having 50 clients to serve at the same time
20:20so basically things are like this
20:27why true because it's redis uses an event loop it's a an event driven program so you can imagine
20:39it as an event loop where things are like read from five descriptors wrote to five descriptors
20:55and here commands are executed so in the first two steps actually those are not
21:05two separated steps as i read from the five descriptors and i get protocols from the clients
21:14i execute such commands and later i will route this content to the five descriptors so the
21:26replies that were generated by the commands will actually read be received by the clients
21:33so if in a given cycle of the event loop i am reading
21:41protocols for 50 clients and i'm executing multiple commands for such clients it will be
21:49really really time consuming to call write system call again and again and again and again
21:56every time i process some client command so what i do instead is basically to accumulate
22:07all the replies all the up and only file protocol in a buffer
22:16while i while there are reliable clients that i can process
22:25yes and finally when i performed a full cycle a full iteration of the event loop i can finally
22:38write the aof buffer in the disk okay in this way basically i'm optimizing things a lot because
22:52write is a very slow system call moreover when i need to also perform fsing
23:06it's a much better reason in order to group together together the clients and the write to the disk
23:13as we will see later but for now for now let's see that normally fsing for example if it's
23:22configured through fsing every second it's not part of this game because basically there is a
23:30side thread performing this operation but in general otherwise we really want to write a single
23:38time for multiple clients so let's check the code about that and this is the function that
23:48performs this work okay flash up and only file is the function that's responsible of getting this
24:02buffer and actually writing the buffer to the disk however it has an argument that's called
24:12force because it's it's not always going to really do that work it depends let's see why sometimes it
24:24may not actually flash the buffer to the disk well to start if if the length of the accumulated
24:33buffer is zero will return as soon as possible there is no reason to continue to do anything
24:39um if a o f fsing is set the policies if the ready set fsing policy is set to to to be every second
24:52what we do is basically is to check if sing is already in progress here we call a interface
25:00that implements some i o trading inside red this but this will return basically three or fours
25:08depending if there is a pending right so if the policy is every second and we are not forced
25:22we see if we check if there is already a sing in progress because even if
25:31we called the thread in order to perform the background fsing some time ago it's possible that
25:40this fsing is still in progress for example for example if the disk is busy the fsing call can take
25:47very long time in order to run so here basically see if still
25:57we we if still just one second past or almost two seconds because this difference must be less than
26:06two we don't write still to disk because writing to the a o f file while there is an fsing in progress
26:17in the same file will basically block the write system call we will not be helped by the the
26:25kernel buffers but we will actually block waiting for the kernel to have seen before it accepts
26:32more writes at least in multiple versions of the linux kernel this is the behavior so we say okay
26:39still we can manage to postpone this writing to disk so we we do that however if if too much time
26:49passed we cannot return and we continue and even at the cost of blocking is we are going to persist
26:57but in that case we log this to the user so we say wait your aof fsing is not able to to basically
27:11to basically be accomplished one for each second so we are here after two seconds and we see that
27:22our previous second fsing call is still pending so your disk is too slow and we are going to
27:29to get some latency problem here and what happens and this is basically the code path that's
27:37reached when we really finally can write to disk at this point we perform the aof write
27:47it's just let's see but it's just write iterated in case a short write is returned
28:01okay so after the the write is executed we need to perform many checks the first thing that we do is to
28:10call the latency hooks in radis that will log latency events in different parts of the code so
28:20that radis is able to tell you via the latency using the latency doctor command what's going
28:32on with your server why why it's slow and at this point basically we check for short writes
28:41and in case we have some write error we basically set an error condition
28:49so that the server will stop not accepting more write commands
28:54however if
28:59if basically the fsing policy is set to always this writing to disk and getting an error like
29:10disk is full or something like that is an random radis cannot recover from and at this point it
29:16exits but instead if it's a recoverable condition because fsing is not set to to always we can just
29:30set this error flag and just throw errors to new writes till the situation is recovered again
29:37like that is at this point at a future point in time space again and we can continue
29:44okay so basically that's well here there are many other things like we have to track the current size of the
29:54auf file because we want to trigger a rewrite of basically it's log convection when the file gets
30:04too big because like there are operations like increment counters so if people keep incrementing
30:13a counter the auf file gets bigger and bigger and bigger but this could be rewritten as set
30:22counter that's the same operation but compressed
30:30a lot in a single operation instead of multiple increments and support so we need to trigger
30:39based on the space of the auf okay
30:49last thing is group commits so basically when auf is set to fsing always we have to guarantee
30:59a given safety property if client sends a write command like set foo bar it will receive the
31:14okay reply only after that it was a successful fsing performance to disk data must be committed
31:27to disk before the client either gets an okay reply the green light so it says basically okay
31:33you can stay safe your your client your data is persisted however to fsing for every client
31:43performing an operation will will make readies bound to the number of fsing's that the disk can
31:52perform every every second that will be extremely slow so returning to the example that we did
32:05before if i have 10 clients and they are doing like set foo bar client one client two is doing
32:16increment counter client three is doing xadd my stream foo bar temperature
32:29temperature uh 19.5 degrees and so forth what about waiting to send the replies
32:45to such clients then perform the fsing a single fsing for all the 10 clients and at this point
32:57we can allow the replies that the the commands generated to actually be sent to the socket
33:04so readies attempts to to perform this however to make sure that this is guaranteed
33:15it's a bit complex because there are multiple interactions between the event loop the way
33:23the commands are dispatched and and so forth that's the end of the first episode and see you
33:30the next time with a new topic bye