The test you almost never perform

When writing tests for your systems you write a long list of test for what should happen when using a function or class. If you did any less, people would say the code was “untested” or badly written.
To compensate for this you perhaps use TDD to make sure you don’t write anything you don’t use – good job. However TDD will not do the trick. TDD is not magic water and everything is now fully tested.

Why do I say this? I haven’t even graduated as an ICT Engineer yet – how could I know?

In my summer vacation I had decided to not only sit and code but do a bit of reading… of fiction. I stumbled across this book, Harry Potter and the Methods of Rationality by the author Less Wrong aka. Eliezer Yudkowsky. Long story very short – Harry is super smart and faces challenges of the mind and not courage.

So – Snippet from chapter 8 between Harry and Hermione, Harry has given Hermione a test and figure out how a certain spell he “invented” works (software function). She looks at it, test it ones and reach a conclusion. But since her conclusion is not correct the following takes place:

The boy’s expression grew more intense. “This is a game based on a famous experiment called the 2-4-6 task, and this is how it works. I have a rule – known to me, but not to you – which fits some triplets of three numbers, but not others. 2-4-6 is one example of a triplet which fits the rule. In fact… let me write down the rule, just so you know it’s a fixed rule, and fold it up and give it to you. Please don’t look, since I infer from earlier that you can read upside-down.”

The boy said “paper” and “mechanical pencil” to his pouch, and she shut her eyes tightly while he wrote.

“There,” said the boy, and he was holding a tightly folded piece of paper. “Put this in your pocket,” and she did.

“Now the way this game works,” said the boy, “is that you give me a triplet of three numbers, and I’ll tell you ‘Yes’ if the three numbers are an instance of the rule, and ‘No’ if they’re not. I am Nature, the rule is one of my laws, and you are investigating me. You already know that 2-4-6 gets a ‘Yes’. When you’ve performed all the further experimental tests you want – asked me as many triplets as you feel necessary – you stop and guess the rule, and then you can unfold the sheet of paper and see how you did. Do you understand the game?”

“Of course I do,” said Hermione.

“Go.”

“4-6-8″ said Hermione.

“Yes,” said the boy.

“10-12-14″, said Hermione.

“Yes,” said the boy.

Hermione tried to cast her mind a little further afield, since it seemed like she’d already done all the testing she needed, and yet it couldn’t be that easy, could it?

“1-3-5.”

“Yes.”

“Minus 3, minus 1, plus 1.”

“Yes.”

Hermione couldn’t think of anything else to do. “The rule is that the numbers have to increase by two each time.”

“Now suppose I tell you,” said the boy, “that this test is harder than it looks, and that only 20% of grownups get it right.”

Hermione frowned. What had she missed? Then, suddenly, she thought of a test she still needed to do.

“2-5-8!” she said triumphantly.

“Yes.”

“10-20-30!”

“Yes.”

“The real answer is that the numbers have to go up by the same amount each time. It doesn’t have to be 2.”

“Very well,” said the boy, “take the paper out and see how you did.”

Hermione took the paper out of her pocket and unfolded it.

Three real numbers in increasing order, lowest to highest.

Hermione’s jaw dropped. She had the distinct feeling of something terribly unfair having been done to her, that the boy was a dirty rotten cheating liar, but when she cast her mind back she couldn’t think of any wrong responses that he’d given.

“What you’ve just discovered is called ‘positive bias’,” said the boy. “You had a rule in your mind, and you kept on thinking of triplets that should make the rule say ‘Yes’. But you didn’t try to test any triplets that should make the rule say ‘No’. In fact you didn’t get a single ‘No’, so ‘any three numbers’ could have just as easily been the rule. It’s sort of like how people imagine experiments that could confirm their hypotheses instead of trying to imagine experiments that could falsify them – that’s not quite exactly the same mistake but it’s close. You have to learn to look on the negative side of things, stare into the darkness. When this experiment is performed, only 20% of grownups get the answer right.”

First of all this is the best book I have ever read and can be found here: fanfiction.com.

Second – What this snippet told us is that you can write a function to do something for you, give it a name and two parameters, make it perform calculations and return a result. No more no less. You can try different numbers, but positive and negative and all of your tests are green flagged. And still, one day on the production server, when you no longer work on the project the char will overflow the limit of 256:

///Summery
///Add two values below 256
///Returns integer
int MyFunction(char first, char second)
{
    int result = first + second;
    return result;
}

int main()
{
    int a = MyFunction(240, 100);
    cout << a << endl; //a == 84
    getchar();
    return 0;
}

Unfortunately this is a poor example because you would have’d done your boundary tests which quickly would reveal something is wrong but this is not hard to overlook – Perhaps Ariane 5 rings a bell?

Point be, as Harry explains, that YOU think your function will do what you want it to do.
You do testing – everything is fine. But you only have half the truth! You think you tested your code but not until you got a ‘NO’ are you done testing it.

P.S.: for those who wonder why the result is 84 here’s the explanation:
When you add two variables you add second to first which is 240+100. That is now stored in the first char. But chars is 8-bit (2^8 = 256 – 1) and when 255 is reached it overflow and drops to 0 again. 256-240 = 16, 100-16 = 84.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>