Why is a "defect" a "defect"?

Why is a "defect" determined to be a "defect" in visual inspection?
How can a "defect" be determined as a "defect" even if neither the binarization level nor the dimensional tolerance is known?
One reason is "because it is different from usual.
It is impossible for one to be a good product and the other a defective product, even though they look the same as other good products. Of course, the part that cannot be seen visually is a different matter.
The other reason is "because the pattern is different from what is expected.
For example, a black spot on a white paper is a defect because there is a black spot even though it is guessed to be pure white. Another is that if the pattern is a repeating pattern but some parts of it are disrupted, that is a defect.
However, this way of thinking may lead to a misjudgment. The presence of both black dots and pattern disturbances may be "positive" and their absence may be "false.
The difference between the logic commonly used in image inspection and that used in visual inspection is so large that it is the main reason why image processing is so difficult. I often hear people give up when things don't work as expected in image inspection, saying that "the human eye is amazing," but I think there is often a problem with the fundamental way of thinking.