>> Because it does not actually matter. If an Int gets in my array, it’s because I screwed up and likely had very poor testing around the scenario to begin with.
>The benefits of static typing is that you don't need testing of things like that. The compiler guarantees safety, allowing you to avoid writing test case that are mundane and boring, such as checking that you don't put an Int into a String array.
This is a canard. You hardly ever (within an epsilon of "never") write tests for types. You write tests for values that you expect, which the type-system doesn't guarantee. Those values have types, so those get tested as a side effect without any additional effort.
What about when you are handling external data? Don't you test what will happen when unexpected types are received? How do you test the handling when a JSON key you were expecting to be an array turns out to be a string or an integer or vice versa?
Do you handle nil's correctly in every place they might occur. These are all things that can be covered by a powerful type system (such as Swift's unless you overuse "!"[0]).
[0] Anything other than use for a member that is initialised during the init method is a code smell in my view.
Well, types are essentially sets of values. So if you can capture the values that you expect in a type, then the type system _does_ give you a guarantee.
OK, I see your point. Of course, the type system will not substitute this kind of tests. But it allows you to check the whole range of values. This is helpful if you e.g. wanted to handle overflows, wanted to limit the operands to positive numbers, etc.
Not only will the type-system not substitute these types of tests, but also the reverse: you just don't write tests for types, as they are subsumed by the value tests, which is why I object to the canard of "the type system saves you from having to write trivial tests for types".
And yes, a type-system can do certain types of "forall" analyses that are difficult or impossible with tests, but that's a different topic.
Generally with tests, you're less concerned about having some particular examples succeed, than you are with finding if there are possible values that fail.
I know that's just an example and that you probably don't actually write tests like that in your day job, but it's a terrible unit test. For example, it fails to tell apart addition from the constant function 7.
Even if you add more data points, you're still not testing addition.
In a way, "testing for specific values" is very misleading. Consider that it'd be the wrong if you were actually trying to prove something. Now, tests aren't proofs, but we still should write them to be as strong as it's practical.
Strong static typing with property testing (a la quickcheck) whenever possible are preferrable in my opinion.
> it fails to tell apart addition from the constant function 7
It's not supposed to do that.
Have you heard of TDD? In a TDD/XP setting, the constant function 7 would be the appropriate implementation for making that test pass, because it is the simplest thing that could possibly work. Then you add another test, let's say add(40,2) EXPECT(42).
Now you could extend your add() function to do case analysis, and maybe in a first step you even do that. But then you refactor towards better code, and you replace the case analysis with actual addition.
For addition the steps are, of course, a bit contrived, because it is "obvious" what is supposed to happen. For production code the technique works really well in keeping the solution as simple as possible but no simpler. You probably wouldn't believe all the "obviously needed" code I haven't written because of it.
Another interesting benefit is that it splits coding into two distinct activities: (1) making the test pass as stupidly as you can and only then (2) making the code good by refactoring an existing solution.
Doing a good job of (2) is much, much easier when you are transforming a solution known to work and safeguarded by tests.
Also having just two test to induce addition may seem a bit sparse, and it is(!), but in my practical experience with TDD, I have been utterly surprised by how few concrete examples have been sufficient to safely cover potentially large spaces. Much fewer than I would have suspected or believed possible.
I understand that this kind of testing often works (though I'm less enthused with TDD as a design technique, which is unfortunately what TDD proponents emphasize, and what you seem to be describing here). I'm saying it is too limited. "But software is written and tested this way", you can say. But buggy software is written every day, even with testing, which is why we should strive to improve our testing tools & processes.
To be clear I'm not saying "abandon TDD" (ok, maybe I'm not rooting for TDD in particular, but I'm definitely not saying "abandon unit testing"). What I am saying is "complement it with static typing and more advanced testing techniques, such as property testing."
Finally:
>> it fails to tell apart addition from the constant function 7
> It's not supposed to do that.
Well, in a way it is. Your tests must attempt to detect flawed implementations, even though they cannot prove correctness. I'm sure you can think of cases where an algorithm sometimes seems to return the correct result, but fails in some cases.
Well, you're obviously not familiar with TDD, just with silly straw man arguments against it. No, the 4+3=7 test is not supposed force creation of actual addition, the additional tests + design principles do.
And I am not saying "but software is written and tested this way". Software is written and (often not) tested in many ways. I am saying that in my practical experience software that is written this way is both much simpler and much more robust than people not familiar with these techniques such as yourselves imagine. Or maybe can imagine.
As to architecture, I strongly recommend Henrik Gegenryd's PhD Thesis: "How Designers Work"[1].
By the way, please don't confuse "easy" with "simple" like the second blog post you reference.
Here are some interesting comments from the above:
- Bloch: tests are inadequate as documentation.
- Norvig: I like TDD but it's inadequate to discover unknown algorithms.
(The infamous Sudoku Solver debacle is a particularly painful example of Norvig's claim, and in particular I think Ron Jeffries' attempt at doing TDD was embarrassing. Like the blog says, if I were a TDD proponent, "I'd be pretty strongly tempted to throw Jeffries under the bus")
Very few if any of the people mentioned outright dismiss TDD, but they do point out its limitations, which to me are mostly about TDD as a design process.
If we go back to TDD as testing, my initial objections apply: it's useful, but it's not enough. More advanced and formal tools, and static typing, are of great help here.
Even if you disagree with everything else, you must at least agree with this: computers are about automation. Automating as much as we can, including testing, is a good thing. Writing tests is itself something that can -- in some areas -- be automated, in which case it should be preferred over hand-writing those tests.
> For example, it fails to tell apart addition from the constant function 7.
Right. But how would you test for the addition function without exhaustive O(N^2) search over the whole input space? (You need to test for commutativity; if you're going to test for associativity, it grows to O(N^3)).
My comment was in the context of the "unit tests vs type systems" debate. It was meant to illustrate my opinion that "more general" is better than "specific values" when testing, as much as is practically possible.
I'm aware that addition is a toy example, but suppose we want to test our implementation:
Except for very simple verification, to exclude obviously broken implementations, I'd rule out testing specific values such as 3+4=7. And, like you said, performing an exhaustive exploration of all values is out of the question.
So I'd try property testing instead. Relevant properties in this case are associativity, commutativity, etc.
As an example, I'd try writing properties such as:
for all X, Y: add(X, Y) = add(Y, X)
I'm not arguing that everything can be tested like this, or that the properties are always easy to formulate; but when they are, I think this is the superior approach.
The tools that I'm aware of that can do this (such as QuickCheck or ScalaCheck) come from statically typed languages, though I don't see why they couldn't be used with dynamically typed languages.
The point is that this starts to look a lot closer to static typing and "testing generalities", philosophically, than what proponents of dynamic typing + unit testing propose. Testing generalities is better than testing specifics, because it's at least a step closer to a proof of correctness.
> The tools that I'm aware of that can do this (such as QuickCheck or ScalaCheck) come from statically typed languages, though I don't see why they couldn't be used with dynamically typed languages.
The tools use type information to determine the universe to draw test values from and the mechanism used to do it. You can actually do something very similar for dynamically typed languages, but if you don't have queryable type annotations for parameters, etc., you have to have more verbose test specifications that provide the scope of testing.
E.g., in a statically-typed language (or a dynamically-typed one with optional type annotations), if your add function is defined as something equivalent to:
double add(double x, double y)
{
...
}
then your test system can use that information and a value generator function for doubles to generate the appropriate test data to validate the property.
OTOH, in a dynamically typed langage where you just have something like
def add(x, y)
{
...
}
Without some additional specification, the test framework doesn't know how to generate the X and Y values for the test you propose.
There are degrees of static typing. Swift tries to get close to Haskell, but without the full set of tools to do so without losing things along the way. This is sort of the worst of both worlds.
In the meantime Strongtalk already demonstrated that you can write a fairly rich optional type system to reap the compile time benefits of static typing while retaining the runtime power of message passing. I think that would have been more fruitful.
Yes, there are degrees of static typing (hence, some people unfortunately disregard great type systems because they are only familiar with Java's cumbersome one).
The thing is, as soon as you realize that unit testing (of the kind proposed by 3+4=7) alone is not enough, or even adequate, and that you must use more advanced tools, and that more advanced tools can and should be automated whenever possible... you will probably start considering the possibility that static typing was a good idea after all. Especially if you were considering using type annotations in your dynamically typed language because you needed to extra info in order to use more advanced testing tools.
Static typing is an incredibly useful automated verification that the compiler performs for you, freeing you to write more interesting tests. It won't catch all mistakes, but no-one is arguing that any single technique will. The argument is about the relative merits of "unit testing alone is enough" vs "testing + automated testing with type checking is way better".
Since we're talking about Swift generics here, you have to realize that the point is not whether generics / static typing can be useful, but if the type of generics and static typing as enforced by Swift is good enough.
The problem here is that Swift has incomplete generics. In several aspects they are significantly worse than even Java's(!)
Together with its brand static typing and type inference that doesn't always work, you are really working against the compiler to get things to work. Not because you get the types wrong, but because you have to work around the incompleteness of the generics implementation!
Swift is not Haskell and should not be mistaken for it. The problem with Swift's generics is one of incompleteness, inconsistencies and problematic trade-offs to maintain ObjC compatibility.
Why do you say it is the worst of all worlds? To me it is a very good compromise and about as good a situation as you can get while maintaining C/Objective-C compatibility. Don't forget there are performance benefits to being less dynamic (not using runtime message passing) and you can opt in to that runtime mode if you want by making classes @objc.
Well, in practice the broken generics stop large amounts of reasonable uses of generics, the lack of co/contravariance force you into hideous workarounds everywhere.
As for the promised runtime performance improvements they have yet to surface and worse: the language is still plagued by extremely uneven performance, even for optimized builds. I could go on at great length citing issues that will be hard to fix within the next year or so. Unfortunately. This is what my fairly hard won (15k loc of Swift code) experience with the language has revealed so far.
I haven't done that volume but I have written a few thousand lines including this branch of a GCD wrapper library which uses generics to allow return values to be passed between the closures running on the different threads in a typesafe way: https://github.com/josephlord/Async.legacy/blob/argumentsAnd...
It was a nightmare getting it all building right (generic methods on generic classes took me a bit of time to sort out) but I think it works well now.
As for performance noting the amount of Swift you've done (and your activity on the DevForums) I expect you know all this but I've done quite a bit on speeding someone else's code up. There are certainly plenty of ways to accidentally slow down code:
I regard Swift's inconsistent runtime performance and over-reliance on compiler optimizations as a major problem with the current state of the language.
It does not help that some slowdowns are due to bugs, and some due to questionable implementation details that require knowledge of how the runtime operates.
Of course, knowing about the runtime is necessary for all micro-optimization regardless of language, but Swift currently requires it almost all of the time.
The difference between Swift and ObjC/C is glaringly obvious.
Your GCD wrapper examples use only the simplest form of generics, with no constraints at all. Magnify the problems you had to get this to work by x10.000 and you get how easy it is to do anything complex with Swift's generics. It's a maze of missing features and bugs.
I think it's very understandable that Owens got fed up with it.
There are actually many poor claims you made in your posts about "good tests".
> I'm aware that addition is a toy example, but suppose we want to test our implementation:
> Except for very simple verification, to exclude obviously broken implementations, I'd rule out testing specific values such as 3+4=7. And, like you said, performing an exhaustive exploration of all values is out of the question.
> So I'd try property testing instead. Relevant properties in this case are associativity, commutativity, etc.
> As an example, I'd try writing properties such as:
> for all X, Y: add(X, Y) = add(Y, X)
These properties can also be satisfied by implementations of add() that:
- return a constant value
- return the smallest number of (x, y)
- return the largest number of (x, y)
The cases you threw out as an "obviously broken implementation" are required to actually validate that functionality of the method. The functionality of the method is also one of the properties of the method.
You can write it in a more generic way than simply: assert(7, add(3, 4)). However, those tests are _also_ required. Without them, you never actually test that the `add()` function does what it's supposed to: add two numbers together.
Regardless of type system, you also have to worry about underflows and overflows - another property of the functionality of the method.
> You are right, without additional information property testing would be less useful. Which is yet another reason to favor static typing in my opinion.
Static typing doesn't help you constrain sets of inputs; it may not be valid that your method accepts all ranges of integers. You could have a method `addbase2(int x, int y)` that is to be used only when x and y are powers of two because of an optimization you perform in that method. Static typing doesn't help you generate the correct input set for x and y.
The only thing that static typing provides, in regards to test cases, is this:
def add(x, y)
assert x is int
assert y in int
return x + y
// test cases
assertIsThrown(add("foo", "bar"))
That was the test case you had.
Regardless, the point of the article was not about static typing being bad. There is value it. However, there is also value in not being so rigid in your type system that things don't work well.
add((short)0, (long)1) // compiler error if you have an extremely rigid type system
Generic systems typically swing the pendulum far to the right requiring an extremely rigid type system. That always causes pain. The question you have to ask, is the ROI worth it. For some, it is. For others, it's not.
Note I never claimed unit testing should be disregarded (I practice it and recognize its benefits), or that static types catch all errors, or that add(x,y) was anything but a toy example.
Please note I didn't throw out add(3,4)==7, but instead pointed out it's terribly inadequate as a test. Additional testing tools must be employed; unit testing alone of this kind is not enough.
With property testing you're still not proving correctness. Tests cannot prove correctness. But it's a step in the right direction. Sure, maybe you have a function "add" that is associative, commutative, and has a neutral element, and it's still not integer addition. I'd argue your confidence in such a function will be a lot higher than if you had simply unit tested a few border cases. You can still do that in addition to property testing, anyway.
> The only thing that static typing provides, in regards to test cases, is this [example]
This assertion is wrong. Static typing done well provides a lot of things "for free", such as restricting incorrect behavior. For example, if you write generic methods you can rule out entire classes of misbehavior. It's not that you "type check" that you are not using something that is not an int (as in your example), but that you simply forbid entire groups of operations at compile time!
Here's another toy example to illustrate the point: what values can a function with the following signature return?
f :: [a] -> a
(For the purposes of this question, you can read that as "a function that takes a list of type a and returns a value of type a).
Now, repeat the exercise with a dynamically typed language. What values can the following function return? (If you want, for the purposes of this question, assume it returns an atomic value and not a collection).
dynamic_f(a_list)
This has an obvious implication on the effort you must make when testing either function.
> Please note I didn't throw out add(3,4)==7, but instead pointed out it's terribly inadequate as a test. Additional testing tools must be employed; unit testing alone of this kind is not enough.
I don't follow what you are saying here at all... It's an API test with no external integration points, what other kind of tests besides unit tests would you have? Your `add(x, y) = add(y, x)` are still unit tests.
Also, you said that you would "rule out testing specific values such as 3+4=7". You have to test specific values; the contract of the function is:
f(x, y) = z
Where z is the mathematical sum of x and y.
Specific value testing is the only way you can verify that claim for a given set of inputs.
>> The only thing that static typing provides, in regards to test cases, is this [example]
> This assertion is wrong. Static typing done well provides a lot of things "for free", such as restricting incorrect behavior.
This was taken out of context; this was in reference your to "generator" of test values for X and Y. The type signature alone is inadequate to generate test values.
Regardless,
f :: [a] -> a
Is no harder to test in a dynamic language.
let r = f([...])
assert(r, correct_value)
assert(r.type, correct_type)
It's up to the contract of the function to determine what, if any, validation needs to be done on the input. This is true regardless of type system. The only question is this: do you also check the type.
Again, type is only _one_ of the constraints that get applied to parameters. In the add function example, the other constraints are:
1. x <= TYPE_MAX (for free in a statically typed system)
2. x <= TYPE_MAX - y
3. y <= TYPE_MAX (for free in a statically typed system)
4. y <= TYPE_MAX - x
Plus the similar for TYPE_MIN. In the `addbase2` example, additional constraints are:
1. x power of 2
2. y power of 2
In this example, we still need to add verification for two-thirds of the constraints.
On the flip-side, with generics, especially with the type of generics we see in Swift (using the `f :: [a] -> a` example), you'll probably need to model constraints of the collection, the element type of the collection, and the type of indexer that is being used if you wish to make your function actually work.
And then your implementation only works for those that rigidly adhere to the type conformance, where as the dynamic one can work for any type that conforms to the protocol, whether loosely or explicitly.
This flexibility is very powerful, is not hard to code safely around, and requires significant less code gymnastics before you can even get your code compiling.
I think you are underestimating the power of statically typed generics when using a language with a decent type system.
In your example:
let r = f([...])
assert(r, correct_value)
assert(r.type, correct_type)
This doesn't test everything we need to know. For example, the following function passes your asserts (in pseudocode):
f(a_list):
if (a_list instanceof List[Int]):
return 0
else
... other stuff ...
whereas the original, statically typed version of the function with signature
f :: [a] -> a
cannot ever do that. This is a profound insight. It cannot return zero "in the case of a list of ints". It doesn't know anything about its input if you don't tell it. And you shouldn't tell it, either, unless you have a very specific reason to do so.
Also, in programming language with decent static typing (that is, not Java or C++; I wouldn't know about Swift to comment), there is a huge additional difference between the two functions:
I can promise you my function doesn't write to disk, doesn't output to the screen, etc. You cannot promise the same with your function. Your function might work when it has access to the disk, as in your test environment, but fail on production where it does not. Ok, so you inspect the code to make sure your function (or any function it calls) don't do I/O. But I don't have to do this, because the type system tells me my function is side-effect free.
So now you have some pretty powerful assurances in favor of my statically typed function:
- It doesn't perform side effects. I don't know about the dynamic function.
- It doesn't produce any value out of thin air; it must work with the list I passed it, because it doesn't know anything else. It doesn't know how to create new values.
- As a consequence of the above, there are fewer possible implementations of my function than of yours, excluding no-ops.
This is a kind of testing "for free" that you don't have with dynamically typed languages. And it is pretty powerful.
Yes, you can cover a lot of cases with unit tests in a dynamic language, but why not let the computer do the boring work for you? It's what computers are there for. Focus on the interesting test cases instead.
> I can promise you my function doesn't write to disk, doesn't output to the screen, etc. You cannot promise the same with your function.
WHAT?!
Your type signatures have absolutely no assurances with regards to side effects. They cannot even make a claim that the function is thread safe, let alone that it doesn't write to disk our output to the screen.
I'm baffled at why you think that is true:
int foo(int bar):
// network call here
// write a log to disk here
// change a global value here
return happy_int
And you are woefully mistaken about about this claim as well: "it must work with the list I passed it, because it doesn't know anything else."
Many languages that actually have good generic type systems allow for type specialization. That means that I can provide different implementations for different types. So in the contrived example of you doing something completely different with my list of ints in the dynamic version is completely possible in many statically typed languages too.
> Your type signatures have absolutely no assurances with regards to side effects. [example]
This might be true for Swift (which I suspect it is), but it's not true in the wider ecosystem of statically typed languages. A language with a type system with allows controlling side effects, such as Haskell, does indeed make such assurances. In Haskell, for a function to make network calls or output to disk, it must live within the IO monad (which can be seen by its type!). I'm discussing Haskell here because I'm more familiar with it, but there are alternative mechanisms in other languages.
Compare:
f :: [a] -> a -- I promise you this function doesn't do any I/O
with
f :: [a] -> IO a -- this function may do I/O
This is a powerful assurance right there! Of course, Haskell programs as a whole must live in the IO monad (a program without any kind of I/O is useless). But you're encouraged to write as much as the program as you can as pure functions, which can then be tested (unit tested or whatever you prefer) with the very useful knowledge that they cannot do I/O.
Next, generics systems. Languages with OOP and generics, such as Scala and, I suspect, Swift, let you do all sorts of naughty things within generic functions.
But doing generic programming like in Haskell is way safer in this regard. No, you are not allowed specialize the type in f :: [a] -> a. Doing so would be unsafe.
So let's go back to my claim: the above function cannot do anything else but produce a value out of the list I passed it. It doesn't know how to produce something else out of thin air. It cannot "inspect" the value of its type; it has no unsafe "instanceOf" operator. It cannot even apply any operation to the values of the list (except of course list operations), since I didn't declare the generic type had any. This is a very powerful assurance that a dynamic language cannot make, and one that simplifies the tests I have to write.
Because of this property, you are encouraged to write, whenever possible, functions that are as generic as possible. Sometimes you can't, but then you'll specify as little as possible, such as:
sumAll :: Num a => [a] -> a -- "a" is a number with operations +, -, etc.
And then you'll have additional operations available for your type a, but not as many as if you were writing this with a dynamically typed language with no assurances at all!
Once you realize this, you'll start thinking of generic programming as a tool that constrains the kind of errors you can make (because you have less choices to make, so to speak). And this has a huge impact on testing!
> So in the contrived example of you doing something completely different with my list of ints in the dynamic version is completely possible in many statically typed languages too.
Of course, many statically typed languages are no better than dynamic languages in this regard. I was talking about decent type systems. Not sure where you'll place Swift, though.
Even with languages which allow instanceOf checks, such as Java and I'm willing to bet most OO languages, the practice is frowned upon and wouldn't pass a code review. Unless, of course, there was no other way to solve the problem, but this really would limit the usefulness of generic programming.
>The benefits of static typing is that you don't need testing of things like that. The compiler guarantees safety, allowing you to avoid writing test case that are mundane and boring, such as checking that you don't put an Int into a String array.
This is a canard. You hardly ever (within an epsilon of "never") write tests for types. You write tests for values that you expect, which the type-system doesn't guarantee. Those values have types, so those get tested as a side effect without any additional effort.