These days it is extremely easy to create a simple web app. I just prompted GPT-OSS with my requirements and got what I wanted in a few iterations. Since the word list is huge, instead of directly reading and generating the standalone html file, it is better to prompt the LLM to generate a script, which takes the word list as an input and generates the html file.
For a PWA, in addition to the html, you also need a service worker that specifies how to cache assets, a manifest file with the app’s metadata, and some icon files. I just took some existing generic app icon files from the internet, but it wouldn’t be hard to generate it with local AI either.
Very quickly I got all the files ready: I have index.html, service-worker.js, manifest.json, icon-512.png and icon-192.png in a folder locally.
To deploy a PWA to my phone, there is a step that is usually a little tricky to figure out, which is that I need to serve the web app with https. But luckily this is a very easy problem to solve these days with Tailscale.
You need to first sign up for an account, then run a Tailscale client on both the development machine and your phone and add both devices to the same tailnet. All of this is free, although slightly annoyingly this step relies on using third party services like Google and Tailscale for authentication. But at least it is free and none of the data goes through their servers.
Here is one extra step that took me a while to figure out: now, you should go to
the Tailscale admin console and change your machine’s name to a name unique to
the app you’re trying to deploy. This allows you to deploy multiple apps using
the same method, otherwise chrome on Android thinks they’re all one app and
refuses to install it multiple times. So let’s say you renamed the machine to
passphrase.
Then, you can find some way to serve the website locally, like running python3
-m http.server in the folder with the static assets. With that running (say on
port 8000), you can then use sudo tailscale serve --https 8001 8000 to serve
the same content to port 8001 but with SSL certs magically figured out, without
chrome complaining that your cert is broken in some way. Having dealt with certs
before, I am extremely grateful that this is such a simple and seamless step.
That command would give you a url that you can open in chrome on Android. Now chrome will prompt you to install the PWA. Install it and that’s it! As far as I can tell you can just kill the commands on your machine, delete all the files if you want, and the app will work just fine offline on your phone. It just looks and works like any other app.
As a final clean up step, you probably want to change the machine name in Tailscale back to the original name. If you’d rather not deal with Tailscale anymore, you can also get rid of all your clients and delete your account, but I’ve found Tailscale to be quite useful for things like this.
That weird step of renaming the machine just to deploy an app can be eliminated now with Tailscale Service! Incredible. I’m going to deploy so many apps.
]]>As far as I can tell, this relevant xkcd popularized the use of a passphrase as a means of generating passwords. Unfortunately, 44 bits of entropy is now considered insufficient. From a cursory search, proton recommends >100 bits of entropy for a “strong” password. Using the xkcd method, we’d need at least 9 english words, which is a lot of letters to type!
One might ask: why do we need easy to type passwords if I’m already using a password manager with random password generation and autofill? That’s great, but there are still going to be some passwords to be manually typed on a regular basis, e.g. computer login or the master password of the password manager.
Is it possible to have a high entropy password that isn’t crazy long?
I don’t know whom to credit, but the idea behind pronounceable gibberish is to increase the entropy per “word” by not actually using words, but using a string of letters that look somewhat like a word instead. So, like, “batim”, “phrux” and whatnot. Still a lot easier to memorize and type than random letters like “bFxls”, but much higher entropy per letter than using real English words.
Ideally, I’d like to have a word list of words no longer than 5 letters, containing at least 2^20 words (around 1 million), then I can plug it into KeePassXC to generate 80 bit passwords for normal passwords and 100 bit passwords for the master password. I looked around online but couldn’t really find anything like this.
I spent two days trying to craft a gibberish generator using GPT-OSS-120B, using techniques like alternating consonants and vowels, and stretching the definitions of each as much as I can. In the end I mostly failed - I couldn’t get past 500k words without starting to generate words that are unpronounceable.
Then I came up with the following algorithm, which was simple and good enough that I got what I wanted in 20 mins. The idea is to generate all possible 5 letter strings, run them through some scoring function, then take the words with the highest score. The scoring is a bit more involved but mostly GPT-OSS took care of it: take some large text corpus as input (e.g. a large list of uncommon english words), generate some log-likelihood of trigrams (i.e. how likely it is for a given three letter combination to appear in a word), then compute the log likelihood of the candidate accordingly. I’m honestly not sure how it worked in full - I didn’t bother to read the code. Since it’s just for my own use, I didn’t do anything like filter out bad words or filter out common passwords.
Here is the code if you’d like to run it yourself:
"""
generate_pseudowords.py
Usage
-----
python generate_pseudowords.py <corpus.txt> [output.txt]
"""
import sys
import math
import itertools
import heapq
from collections import defaultdict, Counter
ALPHABET = "abcdefghijklmnopqrstuvwxyz"
START = "<"
END = ">"
N_TOP = 2 ** 20
SMOOTHING_ALPHA = 0.001
MAX_LEN = 5
def clean_word(word: str) -> str:
"""Return a lower‑cased word containing only alphabetic chars."""
return ''.join(ch for ch in word.lower() if ch in ALPHABET)
def build_trigram_counts(corpus_path: str):
"""
Scan the corpus and count every trigram, including start/end markers.
Returns:
trigram_counts: dict{(a,b,c): int}
bigram_counts : dict{(a,b): int} (needed for conditional probs)
"""
trigram_counts = Counter()
bigram_counts = Counter()
with open(corpus_path, encoding='utf-8') as f:
for line in f:
# split on whitespace – Wikipedia already tokenises fairly well
for raw in line.split():
w = clean_word(raw)
if not w: # skip empty tokens
continue
# pad with start/end markers
padded = START + w + END
# slide a window of three characters
for i in range(len(padded) - 2):
a, b, c = padded[i], padded[i+1], padded[i+2]
trigram_counts[(a,b,c)] += 1
bigram_counts[(a,b)] += 1 # count of the context
return trigram_counts, bigram_counts
def trigram_logprob(trigram, trigram_counts, bigram_counts, vocab_size, alpha):
"""Return log P(c | a,b) with add‑α smoothing."""
a,b,c = trigram
num = trigram_counts.get((a,b,c), 0) + alpha
den = bigram_counts.get((a,b), 0) + alpha * vocab_size
# Guard against log(0) – denominator is never zero because of smoothing
return math.log(num / den)
def score_word(word, trigram_counts, bigram_counts, vocab_size, alpha):
"""
Compute the log‑probability score of a word (without start/end markers).
The word must already be cleaned (lower‑case, a‑z only).
"""
padded = START + word + END
score = 0.0
for i in range(len(padded) - 2):
tri = (padded[i], padded[i+1], padded[i+2])
score += trigram_logprob(tri, trigram_counts, bigram_counts,
vocab_size, alpha)
return score
def generate_all_candidates():
"""Yield every possible string of length 1‑5 over ALPHABET."""
for length in range(1, MAX_LEN+1):
for tup in itertools.product(ALPHABET, repeat=length):
yield ''.join(tup)
def main(corpus_path: str, out_path: str = "pseudowords.txt"):
print("[*] Building trigram counts from corpus …")
trigram_counts, bigram_counts = build_trigram_counts(corpus_path)
# Vocabulary for the trigram model = all symbols that can appear:
vocab = set(ALPHABET) | {START, END}
V = len(vocab)
print(f" #unique trigrams : {len(trigram_counts)}")
print(f" vocab size : {V}")
heap = []
total = 0
print("[*] Enumerating candidates and keeping the best ones …")
for cand in generate_all_candidates():
total += 1
sc = score_word(cand, trigram_counts, bigram_counts, V, SMOOTHING_ALPHA)
if len(heap) < N_TOP:
heapq.heappush(heap, (sc, cand))
else:
# heap[0] is the worst (smallest) score in the current top set
if sc > heap[0][0]:
heapq.heapreplace(heap, (sc, cand))
# occasional progress report
if total % 1_000_000 == 0:
print(f" processed {total:,} candidates … "
f"heap size = {len(heap)} (worst score = {heap[0][0]:.2f})")
print(f"[+] Finished scoring {total:,} candidates.")
print(f"[+] Keeping the top {N_TOP:,} words.")
print(f"[*] Writing results to '{out_path}' …")
top_words = heapq.nlargest(N_TOP, heap) # each entry = (score, word)
with open(out_path, "w", encoding="utf-8") as out:
for score, word in top_words:
out.write(f"{word}\n")
print("[+] Done!")
if __name__ == "__main__":
if len(sys.argv) < 2 or len(sys.argv) > 3:
print(__doc__)
sys.exit(1)
corpus_file = sys.argv[1]
output_file = sys.argv[2] if len(sys.argv) == 3 else "pseudowords.txt"
main(corpus_file, output_file)
Bit reversal is where we take an int (say, 64 bits) and reverse it, e.g. reversing 8 bits may look like 00110101 -> 10101100. There is no CPU instruction to do so, but it can be implemented relatively efficiently in a few ways, e.g. by masking + shifting, bswap (instruction for swapping bytes), lookup tables etc.
Concurrent data structures are data structures that are safe to use in shared-memory parallelism. For performance, modern computers don’t guarantee strong memory consistency, and programs that need to access the same memory locations need to explicitly do so, e.g. via mutexes and atomic operations.
This section covers the paper An Efficient Algorithm for Concurrent Priority Queue Heaps. The problem is: let’s say we want to have a heap (supports inserting an element with priority and removing the element with highest priority) and let a bunch of threads access it. The most naive thing to do is probably an array-based binary heap that is put behind a lock. The ith node has children 2i+1 and 2i+2, parent has higher priority than children, inserts happen at the bottom and bubble up to the root, removals swap the last element to the root and bubbles down, etc. Any thread that attempts to read/write needs to take the lock, and only one thread can proceed at a time.
What if we put a lock at each node? Before you swap two nodes in a bubble-up process, you’d only need to lock those two nodes. That way, you wouldn’t need to lock the entire data structure, giving other threads a chance to proceed.
Unfortunately this results in high contention. If two threads are inserting at the same time, one will be at node i, and the other will be at node i+1. Chances are they have the same parent, so they immediately start contending for the same lock. Even if they have different parents, their parents’ parents are probably the same, and so on.
This paper’s main insight is: what if after inserting at node i, we don’t insert at node i+1, but somewhere far away in the tree? Imagine the following binary tree:
(0)
/ \
(1) (2)
/ \ / \
3 4 5 6
Only nodes 0-2 are occupied. Inserting in the order (3, 4, 5, 6) will likely cause high contention. But if we insert in the order (3, 5, 4, 6), then we might avoid some of the most common lock contentions, since neighboring pairs don’t share the same parent.
More generally, when a low bit of the heap size changes, you want to choose a different sub-tree to insert; when a high bit changes, we have inserted a lot of nodes, so it’s fine to choose a closer sub-tree.
What this ends up looking like is: the high bits of the node at which to insert depends on the low bits of the heap size. We could literally implement that by reversing the order of bits after the leading 1 bit. So the insert order would look like:
0, 1; 10, 11; 100, 110, 101, 111; 1000, 1100, 1010, 1110, 1001, 1101, 1011, 1111…
This idea is simple but clever. Unfortunately this paper didn’t quite stand the test of time, most importantly there’s still global contention to mutate the heap size and to pop the heap at the root. Skip-list based designs are now considered a better choice.
This section covers the paper Split-Ordered Lists: Lock-Free Extensible Hash Tables.
Compared to priority queues, hash tables are undoubtedly more ubiquitous. Generally they can be open/closed - open meaning each bucket contains a resizable container (e.g. linked list) and closed meaning all data lives in the array, and collisions are handled by probing.
A significant challenge in designing a concurrent hash table is the resizing operation. Typically, resizes are done by doubling the size of the array and moving all old data over to the new one. During this migration, no readers/writers can proceed.
This paper asks: what if we don’t move the data into buckets, but move the buckets to point to the data in the right way? If we’re only creating a new index of the same underlying data, then the old index is still valid and can be used in the meanwhile.
More concretely, we want to put all the items in a linked list, so that all the hashes that are equal mod 2^k (for any k) are consecutive in the list. This way, we can maintain a hash table by having an array where the ith bucket points to the first node where hash mod 2^k = i.
It turns out you get this property by ordering the nodes by the reverse of the hash. Thinking about it more, this isn’t that surprising - at first, you want all the nodes with the hashes having 0 as the last bit to be in front of the rest; then, within those, you want all the 0 as the second last bit to be in front…
As a quick example, say we have 8 items with hashes 0-7. Sorted by this order, we have:
000 100 010 110 001 101 011 111
^ ^ | 2 buckets (hash mod 2)
^ ^ ^ ^ | 4 buckets (hash mod 4)
^ ^ ^ ^ ^ ^ ^ ^ | 8 buckets (hash mod 8)
Notice how across different number of buckets, the chains of each bucket is still valid, even though the underlying linked list never changed.
From what I can tell, this design is still not obsolete in 2025, even as new improvements have appeared and computers evolved.
]]>If you’re a total beginner to cyber security like myself, Capture The Flag events are something like puzzles that often involve extracting obscured information, writing bespoke interactive scripts or hacking designated processes to eventually get access to a string, i.e. the flag. It’s a bit like puzzle hunts in that challenges can take many forms, and the difficulty is mostly figuring out what the rules are, rather than following them.
Below, I’ll go through each category. I won’t detail the solutions, only describe general knowledge needed to solve the challenges.
There are 2 flavors of exploits here - client side and server side.
On the client side, I find the chrome dev tools (source code, console, debugger) more than sufficient to understand what the program does or read data out of memory.
On the server side, there are various types of injections. The relevant ones are
php and server-side template injection here. TimeKORP’s source code was given
which makes it pretty easy, as long as you know how to cat a file. Labyrinth
Linguist’s source code was given as an encrypted zip. This encrypted zip could
be decrypted, but note that the cracker
only works for uncompressed plaintext of at least 12 bytes (there is exactly one
file in the zip that matches the conditions). You actually need to figure this
out for a later challenge. But even if you didn’t, you can still see what
entries are in the zip and therefore what SSTI payloads are likely to work. If
you have Burp Suite Professional Edition, your life is a lot easier as it just
tells you the vulnerability. From there it was still difficult to get remote
code execution because none of the top google results for the payloads worked.
The hint here is that in addition to getClassLoader, there is another function
that achieves similar functionality that gives you the java runtime for RCE.
Similar to Labyrinth Linguist, you’re given an encrypted file for Phreaky. Refer to the above to decrypt the zip (john the ripper wasn’t the right tool). Then Wireshark can be used to reveal the next step, and NetworkMiner can get the job done.
Just install ghidra. It’ll be useful later.
LLMs make these coding problems very easy. I did struggle with Stop Drop and Roll a little bit until I changed the approach to use a regex.
This is like a Caesar cipher (not sure if there’s a name for this variant).
You just have to figure out which softwares to use to open these files. I used KiCad Gerber Viewer and Saleae Logic-2.
Ghidra was very helpful for these, as well as pwntools. For Labyrinth, you need to learn something called Return Oriented Programming. Embarrassingly I had to peek at a solution for Void, but I just didn’t read the linked website closely enough. This code snippet basically worked after filling in the details shown by ghidra.
There are quite a few libraries, tools and concepts to go through even in this “beginner-friendly” CTF. But adding these to my personal toolbox has been quite rewarding.
As a metapoint, it can feel like cheating to get hint by searching for the challenge name online. But all of this is for fun anyway, so if you’re going to have more fun knowing what to do than getting stuck, then that’s more important.
]]>You’re on a game show. The host shows you three doors, and you win a car iff you open the right door. After you name your choice (door 1), the host opens door 2, showing that it’s not the right one. Now you’re given the choice to switch to door 3. Should you do it?
Person A says it doesn’t matter, since 1 and 3 are still equally likely to be the right door. Why would eliminating door 2 make that untrue?
Person B says we should switch. Before the host opened door 2, door 1 has 1/3 chance to be the right one, i.e. there is 2/3 chance that you’re wrong. If you were wrong, switching would make you right, so door 3 now has 2/3 chance of being right!
This apparent paradox has caused heated debates throughout the years.
It turns out that what you should do depends on what the host was thinking when they opened door 2.
Let us reframe the game as a two-player zero-sum game. The contestant picks a door (name it 1), then the host either opens door 1 or another door (name it 2), showing that it’s not the correct door. Now the contestant can pick one of the two unopened doors.
The contestant’s strategy can be characterized with a single probability p. If
the host opens door 1, obviously the contestant can only guess among the other
two doors, getting it right 1/2 of the time. If the host opens door 2, the
contestant can switch with probability p.
The host’s strategy can also be characterized with a single probability q. If
door 1 is correct, then the host doesn’t have any choice but to open one of the
remaining two doors at random. Otherwise, the host can open door 1 with
probability q, forcing the contestant to guess.
The contestant’s probability of winning would then be:
P(door 1 is right) * P(contestant doesn't switch)
+ P(door 1 is wrong) * P(host opens door 1) * P(contestant guesses right)
+ P(door 1 is wrong) * P(host opens door 2 or 3) * P(constestant switches)
= 1/3 * (1 + p + q - 2pq)
If the host wants to minimize this probability by changing q, then it depends
the sign of 1-2p. If p < 1/2, the host would want q to be minimized,
otherwise maximized. The contestant faces an analogous situation: if q < 1/2,
p should be maximized, otherwise minimized. In other words, if p is small,
q should be small, which should make p large, which should make q large…
The only equilibrium is when p = q = 1/2, which makes the probability of
winning 1/2. This intuitively makes sense: either player is just randomly
guessing, and the result is the same as flipping a coin.
In the actual game show, the host never opens the contestant’s chosen door, i.e.
q = 0. By the above analysis, p should be 1, resulting in a winning
probability of 2/3. One way to think about this classic result is that the host
is playing a bad strategy and getting exploited by the contestant.
But as a thought experiment, what if you’re the first ever contestant, and you don’t know what the host is thinking when you see door 2 being opened?
q = 0 then switching is 2/3 of a prize;q = 1 then switching is 1/3;How do we stay sane when playing games when the rules are unknown?
]]>There are certain tasks that I want to do, but there aren’t clear deadlines, for example working on a hobby project, reading a book, or learning how to play a new song. I might write those down in my notes, thinking to myself: I’ll get to them when I get some free time.
But when I finally have some free time, I start to find excuses. I’m a bit tired right now. Maybe in an hour. Why not tomorrow?
The thing is, if at time t, I have the chance to do a thing, but I delay that to t+1, then by induction I am never going to do it. In other words, if I’m ever going to do it, it might as well be now.
It is always now or never.
]]>In this post I’ll share my favorite Sudoku trick, called the Unique Rectangle. It is really clever and comes up in actual puzzles. I’ve been solving the New York Times hard Sudoku every night for years, and I’ve used this trick on maybe 10-20% of them.
Imagine you’re solving a Sudoku and you’ve penciled in:
1 2 3
+===+===+===+
A |12 |12 | ? |
+---+---+---+
B | ? | ? | ? |
+---+---+---+
C | ? | ? | ? |
+===+===+===+
D |12 |123| ? |
+---+---+---+
You’ve eliminated all possibilities except 1 and 2 for grids A1, A2 and D1, and you know D2 can only be 1, 2 or 3.
The Unique Rectangle rule says that D2 must be 3.
“Why?” You object. “There are only 3 rules in Sudoku - all 9 3x3 blocks, 9 rows and 9 columns must all contain numbers 1-9. If D2 is 1 or 2, we can still fill in A1, A2 and D1 without violating the rules!”
You are right that the normal constraints of Sudoku aren’t enough. This trick relies on a leap of faith: we assume that the puzzle has a unique solution. This is true for all valid Sudokus – if there are more than one solutions, then the puzzle will not be solvable using logic, and is therefore invalid. (Using the fact that the puzzle is solvable to solve the puzzle might feel like cheating, but I have no personal issues with it.)
In the above example, if D2 isn’t 3, we have 2 possibilities:
(I) 1 2 3 (II) 1 2 3
+===+===+===+ +===+===+===+
A | 1 | 2 | ? | A | 2 | 1 | ? |
+---+---+---+ +---+---+---+
... ...
+===+===+===+ +===+===+===+
D | 2 | 1 | ? | D | 1 | 2 | ? |
+---+---+---+ +---+---+---+
The only constraints that can be used to solve these 4 grids are columns 1 and 2, rows A and D and the 3x3 blocks, but in all these constraints, (I) and (II) are indistinguishable, and we are left with an unsolvable puzzle. So D2 has to be 3.
More generally, the Unique Rectangle rule states that if you have 4 grids in 2 3x3 blocks forming a rectangle, and 3 of them have only 2 choices, then the fourth grid cannot be either of those choices.
There are some extensions to this trick. One is chaining which I have also used:
1 2 3
+===+===+===+
A |12 |12 | ? |
+---+---+---+
...
+===+===+===+
D |23 |23 | ? |
+---+---+---+
...
+===+===+===+
G |13 |134| ? | G2 cannot be 1 or 3
+---+---+---+
Another one that I recently figured out during solves:
1 2 3
+===+===+===+
A |12 |12 | ? |
+---+---+---+
B | ? | ? | ? |
+---+---+---+
C | ? | ? | ? |
+===+===+===+
D |123|123|34 |
+---+---+---+
Either D1 or D2 must be 3, so we know D3 must be 4.
Here is a website with other advanced Sudoku techniques. I have not found the other tricks to be as interesting or as applicable to NYT puzzles.
This sort of “if there is a solution, it must be this” thinking comes up in other occasions as well. Here are a few that I can immediately think of.
In minesweeper, it is very common to run into unsolvable boards, but the same logic can still apply. In this example:
1 2 3 4
+===+===+===+===+ ...
A | | | 1 | 0 |
+---+---+---+---+
B | | | 2 | 1 |
+---+---+---+---+
C | 1 | 2 | |>| 1 |
+---+---+---+---+
D | 0 | 1 | 1 | 1 |
+---+---+---+---+
...
There are three possibilities:
(I) (II) (III)
+===+===+ ... +===+===+ ... +===+===+ ...
| X | | | | X | | | |
+---+---+ +---+---+ +---+---+
| | X | | X | | | | X |
+---+---+ +---+---+ +---+---+
... ... ...
When you finish the rest of the board, you can see how many mines are left. If it’s 1, you know it must be (III); but you can’t tell between (I) and (II). Either way, clicking on A2 or B1 is never wrong, so you can do it without solving the rest of the board. (In minesweeper, if you must guess, it is better to guess earlier so that you don’t waste time in an unsolvable game).
Another example is Alice at the Convention of Logicians. Everything on this wiki page is worth a read, so I won’t bother explaining the puzzle here.
I’ve also found that this line of thinking is useful in solving math puzzles in general, or even problems that aren’t necessarily designed to be solvable. It is like a less blasphemous form of Pascal’s wager – if you’re right, then great, otherwise it doesn’t matter.
]]>There are three steps in this game. First, both players play rock paper scissors with one hand. Second, both players play rock paper scissors with the other hand. Third, both players take back one hand, and the winner is determined by comparing the remaining hands using normal rock paper scissors rules.
Let’s run through a quick example. Say players A/B play ✊/🖐, then 🖐/✌. Now since A can only pick between ✊ and 🖐, if B keeps 🖐 and retracts ✌, B will never lose. If A anticipates B to play 🖐 and also plays 🖐, then they will tie.
But if B anticipates that, B can sometimes pick ✌ which beats 🖐. But if A anticipates that, maybe A will also sometimes play ✊… In true rock paper scissors spirit, there always seems to be a better strategy.
But of course, all games have optimal strategies, when we allow strategies to incorporate randomness. The optimal play is:
Below, let’s go through an outline of the math involved. First, we have to define what both players are maximizing.
It might not be immediately obvious that it is nontrivial to define the goal of the game - of course you want to win instead of lose! But this is not always the case. For example, some people might really want to avoid losing, instead of trying too hard to win.
Rock paper scissors is commonly played as a way to fairly decide a binary outcome between two people (e.g. who gets to pick the restaurant). It is reasonable to apply the same to this game, meaning players will repeat the game until a winner is determined, allowing no ties. Playing optimally then means maximizing the probability of winning in a repeated game.
Since the game is fair, in the event of a tie, your chance of winning is still 1/2. So, we’re maximizing P(win)+P(tie)*1/2, which is the same as minimizing of P(lose)+P(tie)*1/2. Equivalently we can put those together and maximize P(win)-P(lose) for each game. In other words, we can pretend the loser pays $x to the winner, and maximize the expected value of winnings.
First we can analyze step 3 - both players have to pick one out of two given choices.
Let’s run through the boring cases quickly. Boring case 1: either player has the same choice for both hands (dumb). Boring case 2: both players have the same two choices (the optimal pick is obvious).
Now to analyze the remaining case where both players have different choices. Let’s say A has ✊, 🖐 and B has 🖐, ✌ to pick from. Let’s say the winner of the game wins $3, the loser loses $3. We have four outcomes after both players take back one hand:
| A \ B | 🖐 | ✌ |
|---|---|---|
| ✊ | -3 \ 3 | 3 \ -3 |
| 🖐 | 0 \ 0 | -3 \ 3 |
Since this is a zero sum game, to maximize your winning, you want to make your opponent’s best option as bad as possible. This happens when both of their options are equally bad.
When A (B) plays 🖐 with 2/3 probability, B (A) gets $1 (-1) in expectation no matter which hand they take back. Hence, the optimal strategy for step 3 is to pick the option that can lead to a tie 2/3 of the time (🖐 in this case).
Now that we’ve established step 3’s strategy, let’s do the same to step 2. Say A picked ✊ and B picked 🖐, and both have to pick their second hand. Since we already worked out the expected value for all possibilities at step 3, we can again tabulate the expected value of all scenarios for A and B here.
| A \ B | 🖐 , ✊ | 🖐 , ✌ |
| ✊ , ✌ | -1 \ 1 | 1 \ -1 |
| ✊ , 🖐 | 0 \ 0 | -1 \ 1 |
If you paid attention, you’ll realize that this is just the previous table from step 3 but with winnings scaled down by 1/3. This means that at step 2, we’re playing the exact same game! So the optimal strategy must also be the same - we play the option that leads to a tie with probability 2/3.
As a bonus fact, we can calculate the probability of a tie. If both players end up with the same two choices after step 2, the game must end in a tie; otherwise there’s still a 2/3*2/3 chance of a tie. This yields 1/3 + 2/3 * (4/9 + 5/9 * 4/9) = 193/243 = 79.4% chance of a tie.
This concludes the analysis of the game. In Cantonese, both players would both say something like “Xxx, Xxx, xxxxXxx” (inscrutable Chinese) where the uppercase letters indicate when the steps happen. I wonder how this gameplay can be translated into English exactly.
]]>We have a system that basically subscribes to a whole bunch of data, does a bunch of computations on them, and publishes output in real time. The computations are split into tasks identifiable by unique names. For both CPU & memory reasons, the system spawns up to a few dozens of workers (Linux processes) across a bunch of computers, and assigns each task to one process by the hash of its name, modulus the number of workers.
For redundancy and latency, we run a few replicas of the whole thing across multiple data centers, all doing roughly the same computations.
One day, during a routine system upgrade, all replicas crashed one after another. This got us into panic mode.
To be clear, this is bad - this is what we specifically tried to prevent by running replicas, and stagger their upgrade schedule.
To find out the root cause, the first thing as always is to inspect the logs. There was a single error message saying which worker crashed first, and the exception that crashed it. The exception suggested that one of the values that came from a data subscription was too large and caused a buffer overflow.
OK, that’s something, but we have hundreds of thousands of data subscriptions, so we need more cleverness to narrow it down so we can find the problematic data.
Well, we know the worker number is X out of N total workers from the logs. If we get a list of all data scriptions and their task names (a data subscription is also considered a task), then we can compute the hash of each name and see which ones are running on worker X. This will narrow it down by a factor of N, which is on the order of 50. Which is not nearly enough.
But it happens that due to whatever reason, some replicas run with a different number of workers. That means we can gather a bunch of Xi and Ni pairs, and narrow down the set of suspected tasks further.
If you have a few equations of the form A mod Ni = Xi, you can merge them together to get A mod N* = X* where N* is the LCM of all Ni using the Chinese Remainder Theorem. The larger we can make N*, the fewer tasks will satisfy the equation, and the better chance we have to pin down exactly the one task we’re looking for.
So we gathered 4 pairs of X and N, and ended up shrinking the number of suspected tasks by a factor of a few thousand, leaving us with only a few dozen options. Poring over the task names one by one, we finally found the one subscription that caused the crashes.
The bug itself is fairly simple, but the mechanism in which it crashed all replicas is a bit subtle.
There was a recent code change that changes the behavior when the system gets erroneous values from data subscriptions. In the past, when a worker gets an error, it just passes the error value to downstream computations. The code change was to append metadata to these errors to help track down where they came from.
This change seems innocent enough, but the issue manifests when the system is configured to subscribe to data published by itself. Let’s say such a self loop exists in a task. When this task first computes, it subscribes to data that hasn’t been published yet, so it will result in an error. In the old code, this error will then be fed in to the task again, but the output will not change. However, in the new code, each round of feedback leads to a bigger error value due to the additional metadata, eventually overflowing buffers and crashing the process and also clients consuming the value.
This also explains why despite rolling out the new executable to only a subset of replicas, all of them crashed. When subscribing to a data, you have to specify some sort of url, and this url points to one of the replicas. In other words, only one replica has the self loop, and other replicas are consumers of the output of the loop. So when the loop is completed in the roll process, all replicas will crash upon receiving the large value, regardless of whether they contain the code change.
This incident didn’t end up causing too much headache because it was fixable with a rollback. Either way, it’s a good exercise to think through it to learn the maximum amount of lessons out of it.
Pinning down the issue this time required some amount of luck. In particular, we had replicas running with different number of workers. This did not have to be the case, and it even seems undesirable to have different enviroments. One change we could make here is to change the hashing scheme of task names to worker. We could hash the task name and the replica ID together (i.e. use the ID to salt the hash). This way, we don’t have to rely on the numbers of workers being coprime with each other to narrow down tasks using worker IDs.
This incident is not the first time that snowballing error values caused hiccups. I’ve also heard of stories where parsing and appending to error values lead to accidentally quadratic time complexity. Perhaps we should be a bit careful when dealing with error values, because they can sometimes be unexpectedly large. (I’m not sure how much this makes sense in various programming languages; some languages might not have the concept of a error value object that can be manipulated at runtime.)
Another thing that is less clear is that perhaps we could just outright ban self loops, as these are probably just footguns. But this might or might be reasonable depending on the actual situation.
One might also be tempted to think that instead of relying on clever filtering based on hashes, we should just improve the error message in the logs to show exactly what caused the crash. I think practically this would not have helped in this case. Sometimes you just don’t know where the system could crash - if we had anticipated it, we would’ve fixed it. Wrapping every single part of code with error tagging just seems excessive and infeasible.
In the end, I didn’t think anyone made a mistake in the process, and there was not much we could’ve done to avoid it. Testing couldn’t have caught it because the loop would only exist in production, due to all testing systems also subscribing to the url that points to production; this bug was hard to anticipate in code review; and careful deployment wouldn’t have prevented it.
Sometimes, incidents are just a cost of business, even if they happen in production.
]]>let sign__old f =
assert (Float.is_finite f);
if String.is_prefix (Float.to_string f) ~prefix:"-"
then `Negative
else `Nonnegative
Naturally, I replaced it with the following:
let sign__new f =
assert (Float.is_finite f);
if Float.is_negative f then `Negative else `Nonnegative
A few days later, this caused an issue in prod. How is it possible? These two functions are obviously equivalent, right?
It turns out there is exactly one edge case for which these two functions behave differently: -0., aka negative float zero.
In the IEEE floating point standard, numbers are represented as sign and magnitude. This means it is technically possible to have both a positive and a negative zero. While these two values are numerically equal, both are treated as valid floats, and they behave differently when passed into different functions.
In this case, sign__new sees negative zero as `Nonnegative, because it is not numerically smaller than zero, despite having a negative sign. On the other hand, Float.to_string (-0.) produces "-0.", so sign__old thinks it is `Negative.
I think it’s likely uncommon that the existence of negative zero leads to bugs in code, because typically programs see these two values as having the same behavior. In my case, the code is constructing an AST that represents the float value. The old code produces a unary negation applied to positive zero immediate, while the new code proces a negative zero immediate. This change caused an exception in prod because the code generation and parsing process no longer round trips.
The first time I learned about negative zero, I thought it was a misfeature. But it actually makes sense, considering infinities are also signed. With all four values representable, we can have 1/-inf = -0, 1/-0 = -inf and so on, which is nice.
With the bug identified, the fix is easy: use Float.ieee_negative instead of Float.is_negative.
If you enjoyed this post, try another: Precisely Compare Ints and Floats
]]>