Make the binary say yes

A reverse engineering challenge gives you one file and one condition. Produce the input the program accepts. No source. No comments. No specification. The Jane Street reverse engineering problem is that condition made explicit: a compiled artifact, a pass or fail gate, and nothing describing how the gate decides. That starting state is not academic. It is the exact position a red team operator occupies against a target with no documentation.

The challenge measures one capability. Can you recover intended behavior from observable behavior when the only authority on what the program does is the program itself. A binary does not lie about its logic. It executes it. Every accept and reject path is present in the file, reachable, and deterministic on hardware you control. The task is not invention. It is recovery.

Red teaming operates on the same authority. You are rarely handed source. You are handed an endpoint, a firmware image, a client binary, an API that returns yes or no. Your job is to determine what the system does, not what its documentation claims it does. The gap between those two statements is where access lives. The Jane Street problem is a bounded instance of that gap. Solve it and you have practiced the core motion of the work: treating a closed system as readable because it must run to be useful.

The assumption most people carry into a binary is that compilation is a one way door. Source goes in, an opaque blob comes out, and the logic is sealed. Under that belief the executable is a black box and the only move is to guess inputs until one works. That belief is why brute force is the first instinct, and why most people stall within the first hour. The file is treated as a wall instead of a record.

The second assumption is that solving the challenge requires understanding every instruction in the file. Beginners open the binary and read from the entry point, tracing linearly, trying to hold the entire program in their head. A stripped binary of any size defeats that approach immediately. The instinct to comprehend everything is the instinct that guarantees you finish nothing. Effort gets spent on code that never touches the decision.

The same assumption governs how products ship. A vendor compiles a client, removes the symbols, and treats the resulting binary as a protected boundary. Closed source is filed as a security property. Obscurity is recorded as a control. Leadership signs off believing that because the logic is not published, the logic is not available. That is the assumption the Jane Street problem exists to break, and it is the assumption a red team is paid to break in production.

Once you stop reading and start locating, the binary stops being opaque. Compilation strips names, not behavior. The routine that decides accept or reject is a comparison, and a comparison has a fixed location in the file. Ghidra or IDA recovers the control flow graph and surfaces the validation routine directly. You do not read the whole program. You find the single branch that gates success and you work backward from it. The problem collapses from thousands of instructions to the handful that decide the outcome.

Static recovery gives you the shape. Dynamic analysis confirms it. A debugger runs the binary under your control, sets a breakpoint on the comparison, and exposes the exact register and memory state at the moment of decision. You watch what the program expects rather than guessing at it. When the check is algebraic rather than a fixed string, you hand the constraint to a solver. Symbolic execution with a tool like angr, or a constraint solver like z3, walks the branch conditions and returns an input that satisfies the accept path. The input space stops being a search and becomes a solution.

This is why the vendor assumption fails, and why it fails the same way every time. If a system executes on hardware the analyst controls, its logic is recoverable, because execution is disclosure. Obfuscation raises the cost of recovery. It does not change the property. The Jane Street problem demonstrates the property in miniature, on a file built to be solved. Red teaming demonstrates the same property against systems built and shipped on the belief that a stripped binary keeps its secrets. It does not. A closed binary is not a closed system. It is a readable one that has not been read yet.

The control that failed is obscurity recorded as enforcement. A stripped binary removes symbol names. It does not remove instructions. A processor cannot execute a name. It executes opcodes, and every opcode is decodable because the hardware decodes it to run. The distance between unreadable to a human at first glance and unexecutable is the entire failure. Stripping raised the reading cost. It never touched the execution requirement. The logic the vendor believed was sealed was present in the file the whole time, in the only form that matters, the form the processor accepts.

The decision boundary is fixed in place. Accept or reject is a branch. A branch has a condition, an address, and two edges. The condition is evaluated against input state held in registers and memory, on hardware the analyst controls. A breakpoint on that branch exposes the operands at the moment of the comparison. Renaming, dead code, and control flow flattening do not remove the branch, because removing the branch removes the check, and removing the check defeats the product. The one routine the vendor most needs to keep is the one routine the analyst most needs to find. Both requirements point at the same address.

Obfuscation multiplies instructions. It does not change the class of the problem. Packing, virtualization, and flattening each raise recovery cost by adding steps the analyst must reverse. Every added step is itself executed by the processor, which means every added step is itself observable. A defense built by adding more for the processor to run is a defense that hands the observer more to watch. The cost curve moves. The property does not. Recovery stays possible because execution stays mandatory.

The pattern is a boundary placed on the wrong side. A control is only a control at a boundary the defender enforces. When the deciding check executes on hardware the adversary owns, the adversary is not attacking the check from outside. The adversary is the runtime. Every hardcoded key in a shipped client, every license gate in a desktop application, every authorization decision made inside a mobile binary sits on the attacker’s side of the line and is evaluated by the attacker’s processor.

The examples differ in packaging and resolve to the same mechanism. A firmware image is extracted, loaded, and executed under the analyst’s control. A client that decides whether a user is entitled runs the entitlement branch on the analyst’s processor. Same branch, same operands, same recoverability as the challenge file. The Jane Street artifact and a shipped product differ in intent, not in property. One was built to be solved. The other was built to be trusted. The property underneath both is identical, because both must run to be useful and both run somewhere an analyst can control.

The corollary is direct. A secret shipped to the client is a secret handed to the adversary. A key embedded in a binary is a key you delivered. Once logic crosses onto hardware you do not control, its confidentiality is not confirmed and its integrity is not enforceable by you. Any trust that depends on the client not being read is trust extended to the party most motivated to read it.

What must now be true is a single placement rule. If a decision matters, it executes where you control the runtime. That is your server, your enforcement point, the boundary you own. The client renders and requests. It does not decide. Any accept or reject that gates access is evaluated on your side of the line or it is not enforced. Identity is the boundary, and the boundary cannot be held on a machine you handed to the adversary.

Closed source as a security property is a finding, not a strategy. If a threat model depends on the binary not being read, it depends on the adversary declining to do the exact thing the adversary is there to do. Treat obfuscation as delay and budget it as time, not prevention. Assume the client is fully known to whoever holds it, and design so that full knowledge of the client changes nothing about what they can access.

The Jane Street challenge and a red team engagement reduce to one statement. A system that runs is a system that discloses. Recovery is not a question of whether. It is a question of cost and time, and both are finite for anyone who controls the hardware. Build so that disclosure of the client costs the adversary nothing they did not already hold, and place every decision that matters behind a boundary they cannot execute. Everything else is noise.