Production software: Be wary of assertions

As the Wikipedia puts it (emphasis mine):

An assertion is a predicate (a true–false statement) placed in a program to indicate that the developer thinks that the predicate is always true at that place. If an assertion evaluates to false at run-time, an assertion failure results, which typically causes execution to abort.

On the programmer’s side, assertions are invaluable in writing readable code: they provide a mechanism with which developers can explicitly state their thoughts and expectations in the form of code instead of comments.

However, on the deployer and sysadmin’s side, misplaced assertions can easily turn into a massive, possibly distributed outage. What should be a controlled failure in the face of invalid input or unpredicted state, an assertion can easily exacerbate those two problems by making the deployed application go “poof” without notice.

The basic rule of thumb is: never ever use assertions to validate user input. By user input we understand any data that comes from outside the program, including a configuration or input file, the parameters embedded in an URL query or the arguments provided in an RPC. User input always needs some form of sanitization and such checks must be done in a way that allows returning controlled error codes to the caller. Assertions don’t allow this kind of control so they are not suitable. In fact, assertions present two problems in this case: if assertions are enabled, the program will crash when facing invalid input; and if assertions are disabled, the program will perform no validation of the input! Both situations are undesirable.

“But not all assertions are about user input!” I hear you say. True. But, first of all, ask yourself again if the assertion floating your mind really is not about user input… sure it’s not? Really? OK, then your assertion might still be a problem because it can still be triggered. Remember: any assertion failure will cause the whole process to exit abruptly.

Why are all these things scary, though? Because somebody, intentionally or unintentionally, could cause all your servers to crash at once. Suppose you have, by mistake, used an assertion to ensure that a given integer in an input parameter of an RPC call is within some bounds. If the attacker can send an RPC with an integer out of bunds to all of your servers, the servers will immediately crash as soon as they start performing input data validation. And note that this user need not talk to all of your servers directly: maybe he is just talking to a frontend server which later just distributes the “query of death” to a large subset of backend services. What should have happened here, however, is that the server should have spotted the invalid data and returned an error to the attacker.

So to what extent should you go in removing assertions? As usual, it depends… but I’ve witnessed the developers of some major products audit the serving path of the product’s whole stack in search for assertions and replaced them all with explicit error checks. Maybe it’s the time to do the same on your end. And don’t be shy if you are “just” the sysadmin and not the developer: it’s probably a good time to educate your peers!