De Morgan's Laws

Concepts covered: Boolean Operators

Pipelines are full of negated conditions: skip a batch unless it is loaded and non-empty, alert unless a job succeeded or was cancelled. De Morgan's 2 laws, named for the 19th-century logician Augustus De Morgan, move a not inside parentheses. That usually turns a condition you have to decode into one you can read left to right.

The 2 laws

• not (a and b) is equivalent to not a or not b
• not (a or b) is equivalent to not a and not b

The operator inside flips between and and or, and each operand gets its own not. Because not binds tighter than and and or, not a or not b already means (not a) or (not b). With 2 inputs there are only 4 cases, and the table checks all of them:

Negating a single comparison

Once a not sits on a single comparison, replace the pair with the opposite operator. When 2 numbers are ordered, exactly one of x < y and x >= y is True, so >= is the opposite of <. Writing > instead is a common slip:

The last row is De Morgan in disguise. A chained comparison is an and of 2 comparisons, so negating it yields an or of their opposites.

skip = not (row_count > 0 and status == "loaded")
skip = row_count <= 0 or status != "loaded"

Short-circuiting survives the rewrite. In both lines, status is compared only when row_count is above 0.

The != with or trap

The bug De Morgan exposes most often reads naturally in English. Someone wants payments that are neither succeeded nor refunded and writes status != "succeeded" or status != "refunded". Every status differs from at least 1 of those 2 strings, so the condition is True for every input, "succeeded" included.

status = "succeeded"
print(status != "succeeded" or status != "refunded")
print(status != "succeeded" and status != "refunded")
print(not (status == "succeeded" or status == "refunded"))

Output:

True
False
False

The intent was not (status == "succeeded" or status == "refunded"). Pushing the not inside turns the or into and, which gives the second line. Treat a run of != tests joined by or as a likely bug until you have checked it.

Where the comparison flip breaks

The rules for and and or, like the rule for not, hold for every truth value. The comparison flip carries an extra assumption: that the 2 sides are ordered. A float NaN is ordered against nothing, so any comparison with it using <, <=, >, >= or == returns False.

latency = float("nan")
print(not (latency < 500))
print(latency >= 500)

Output:

True
False

A filter written as not (latency < 500) keeps rows whose latency is NaN, and the rewrite latency >= 500 drops them without warning. When missing measurements arrive as NaN, decide which side they belong on and test for them first. latency != latency is True only for NaN, since NaN compares unequal even to itself.

A retry rule reads if not (retries < 3 and status == "failed"):. Which condition is equivalent for an integer retries and a string status?

• retries > 3 or status != "failed"
• retries >= 3 and status != "failed"
• retries >= 3 or status != "failed"
• retries < 3 or status != "failed"

A rewrite that agrees with the original on 1 input proves little. retries > 3 and retries >= 3, each taken alone, disagree at 3 and nowhere else between 0 and 5. Test a rewrite where each comparison flips: at equal values, and 1 step to either side.

About this lesson

This section is part of the Control Flow: Advanced lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.