Why do dynamic languages make it more difficult to maintain large codebases? [duplicate]

Question

This question already has answers here:

Is there a correlation between the scale of the project and the strictness of the language? (5 answers)

Closed 6 years ago.

Large codebases are more difficult to maintain when they are written in dynamic languages. At least that's what Yevgeniy Brikman, lead developer bringing the Play Framework to LinkedIn says in a video presentation recorded at JaxConf 2013 (minute 44).

Why does he say this? What are the reasons?

Answer 1

dynamic languages make for harder to maintain large codebases

Caveat: I have not watched the presentation.

I have been on the design committees for JavaScript (a very dynamic language), C# (a mostly static language) and Visual Basic (which is both static and dynamic), so I have a number of thoughts on this subject; too many to easily fit into an answer here.

Let me begin by saying that it is hard to maintain a large codebase, period. Big code is hard to write no matter what tools you have at your disposal. Your question does not imply that maintaining a large codebase in a statically-typed language is "easy"; rather the question presupposes merely that it is an even harder problem to maintain a large codebase in a dynamic language than in a static language. That said, there are reasons why the effort expended in maintaining a large codebase in a dynamic language is somewhat larger than the effort expended for statically typed languages. I'll explore a few of those in this post.

But we are getting ahead of ourselves. We should clearly define what we mean by a "dynamic" language; by "dynamic" language I mean the opposite of a "static" language.

A "statically-typed" language is a language designed to facilitate automatic correctness checking by a tool that has access to only the source code, not the running state of the program. The facts that are deduced by the tool are called "types". The language designers produce a set of rules about what makes a program "type safe", and the tool seeks to prove that the program follows those rules; if it does not then it produces a type error.

A "dynamically-typed" language by contrast is one not designed to facilitate this kind of checking. The meaning of the data stored in any particular location can only be easily determined by inspection while the program is running.

(We could also make a distinction between dynamically scoped and lexically scoped languages, but let's not go there for the purposes of this discussion. A dynamically typed language need not be dynamically scoped and a statically typed language need not be lexically scoped, but there is often a correlation between the two.)

So now that we have our terms straight let's talk about large codebases. Large codebases tend to have some common characteristics:

• They are too large for any one person to understand every detail.
• They are often worked on by large teams whose personnel changes over time.
• They are often worked on for a long time, with multiple versions.

All these characteristics present impediments to understanding the code, and therefore present impediments to correctly changing the code. In short: time is money; making correct changes to a large codebase is expensive due to the nature of these impediments to understanding.

Since budgets are finite and we want to do as much as we can with the resources we have, the maintainers of large codebases seek to lower the cost of making correct changes by mitigating these impediments. Some of the ways that large teams mitigate these impediments are:

• Modularization: Code is factored into "modules" of some sort where each module has a clear responsibility. The action of the code can be documented and understood without a user having to understand its implementation details.
• Encapsulation: Modules make a distinction between their "public" surface area and their "private" implementation details so that the latter can be improved without affecting the correctness of the program as a whole.
• Re-use: When a problem is solved correctly once, it is solved for all time; the solution can be re-used in the creation of new solutions. Techniques such as making a library of utility functions, or making functionality in a base class that can be extended by a derived class, or architectures that encourage composition, are all techniques for code re-use. Again, the point is to lower costs.
• Annotation: Code is annotated to describe the valid values that might go into a variable, for instance.
• Automatic detection of errors: A team working on a large program is wise to build a device which determines early when a programming error has been made and tells you about it so that it can be fixed quickly, before the error is compounded with more errors. Techniques such as writing a test suite, or running a static analyzer fall into this category.

A statically typed language is an example of the latter; you get in the compiler itself a device which looks for type errors and informs you of them before you check the broken code change into the repository. A manifestly typed language requires that storage locations be annotated with facts about what can go into them.

So for that reason alone, dynamically typed languages make it harder to maintain a large codebase, because the work that is done by the compiler "for free" is now work that you must do in the form of writing test suites. If you want to annotate the meaning of your variables, you must come up with a system for doing so, and if a new team member accidentally violates it, that must be caught in code review, not by the compiler.

Now here is the key point I have been building up to: there is a strong correlation between a language being dynamically typed and a language also lacking all the other facilities that make lowering the cost of maintaining a large codebase easier, and that is the key reason why it is more difficult to maintain a large codebase in a dynamic language. And similarly there is a correlation between a language being statically typed and having facilities that make programming in the larger easier.

Let's take JavaScript for example. (I worked on the original versions of JScript at Microsoft from 1996 through 2001.) The by-design purpose of JavaScript was to make the monkey dance when you moused over it. Scripts were often a single line. We considered ten line scripts to be pretty normal, hundred line scripts to be huge, and thousand line scripts were unheard of. The language was absolutely not designed for programming in the large, and our implementation decisions, performance targets, and so on, were based on that assumption.

Since JavaScript was specifically designed for programs where one person could see the whole thing on a single page, JavaScript is not only dynamically typed, but it also lacks a great many other facilities that are commonly used when programming in the large:

• There is no modularization system; there are no classes, interfaces, or even namespaces. These elements are in other languages to help organize large codebases.
• The inheritance system -- prototype inheritance -- is both weak and poorly understood. It is by no means obvious how to correctly build prototypes for deep hierarchies (a captain is a kind of pirate, a pirate is a kind of person, a person is a kind of thing...) in out-of-the-box JavaScript.
• There is no encapsulation whatsoever; every property of every object is yielded up to the for-in construct, and is modifiable at will by any part of the program.
• There is no way to annotate any restriction on storage; any variable may hold any value.

But it's not just the lack of facilities that make programming in the large easier. There are also features that make it harder.

• JavaScript's error management system is designed with the assumption that the script is running on a web page, that failure is likely, that the cost of failure is low, and that the user who sees the failure is the person least able to fix it: the browser user, not the code's author. Therefore as many errors as possible fail silently and the program keeps trying to muddle on through. This is a reasonable characteristic given the goals of the language, but it surely makes programming in the larger harder because it increases the difficulty of writing test cases. If nothing ever fails it is harder to write tests that detect failure!

• Code can modify itself based on user input via facilities such as eval or adding new script blocks to the browser DOM dynamically. Any static analysis tool might not even know what code makes up the program!

• And so on.

Clearly it is possible to overcome these impediments and build a large program in JavaScript; many multiple-million-line JavaScript programs now exist. But the large teams who build those programs use tools and have discipline to overcome the impediments that JavaScript throws in your way:

• They write test cases for every identifier ever used in the program. In a world where misspellings are silently ignored, this is necessary. This is a cost.
• They write code in type-checked languages and compile that to JavaScript, such as TypeScript.
• They use frameworks that encourage programming in a style more amenable to analysis, more amenable to modularization, and less likely to produce common errors.
• They have good discipline about naming conventions, about division of responsibilities, about what the public surface of a given object is, and so on. Again, this is a cost; those tasks would be performed by a compiler in a typical statically-typed language.

In conclusion, it is not merely the dynamic nature of typing that increases the cost of maintaining a large codebase. That alone does increase costs, but that is far from the whole story. I could design you a language that was dynamically typed but also had namespaces, modules, inheritance, libraries, private members, and so on -- in fact, C# 4 is such a language -- and such a language would be both dynamic and highly suited for programming in the large.

Rather it is also everything else that is frequently missing from a dynamic language that increases costs in a large codebase. Dynamic languages which also include facilities for good testing, for modularization, reuse, encapsulation, and so on, can indeed decrease costs when programming in the large, but many frequently-used dynamic languages do not have these facilities built in. Someone has to build them, and that adds cost.

Answer 2

Because they deliberately abandon some of the tools that programming languages offer to assert things you know about the code.

The best-known and most obvious example of this is strict/strong/mandatory/explicit typing (note that the terminology is very much disputed, but most people agree that some languages are stricter than others). When used well, it acts as a permanent assertion about the kind of values you're expecting to occur in a particular place, which can make reasoning about the possible behaviour of a line, routine or module easier, simply because there are fewer possible cases. If you're only ever going to treat someone's name as a string, many coders are therefore willing to type a declaration, to not make exceptions to this rule, and to accept the occasional compilation error when they have made a slip of the finger (forgot quotes) or of the brain (forgot that this rating is not supposed to allow fractions).

Others think that this restricts their creative expressivity, slows down development and introduces work that the compiler should do (e.g. via type inference) or that isn't necessary at all(they'll just remember to stick to strings). One problem with this is that people are quite bad at predicting what kind of errors they will make: almost everybody overestimates their own ability, often grossly. More insidiously, the problem becomes gradually worse the larger your code base - most people can, in fact, remember that the customer name is a string, but add 78 other entities to the mix, all with IDs, some with names and some with serial 'numbers', some of which really are numeric (require computation to be done them) but others of which require letters to be stored, and after a while it can become pretty hard to remember whether the field you're reading is actually guaranteed to evaluate to an int or not.

Therefore, many decisions that suit a quick prototype project well work much less well in a huge production project - often without anyone noticing the tipping point. This is why there is no one-size-fits-all language, paradigm or framework (and why arguing which language is better are silly).

Answer 3

Why don't you ask the author of that presentation? It's his claim, after all, he should back it up.

There are plenty of very large, very complex, very successful projects developed in dynamic languages. And there are plenty of spectacular failures of projects written in statically typed languages (e.g. the FBI Virtual Case File).

It is probably true that projects written in dynamic languages tend to be smaller than projects written in statically typed languages, but that is a red herring: most projects written in statically typed languages tend to be written in languages like Java or C, which are not very expressive. Whereas most projects written in dynamic languages tend to be written in very expressive languages like Scheme, CommonLisp, Clojure, Smalltalk, Ruby, Python.

So, the reason why those projects are smaller is not because you can't write large projects in dynamic languages, it's because you don't need to write large projects in expressive languages … it simply takes much fewer lines of code, much less complexity to do the same thing in a more expressive language.

Projects written in Haskell, for example, also tend to be pretty small. Not because you can't write large systems in Haskell, but simply because you don't have to.

But let's at least take a look at what a static type system has to offer for writing large systems: a type system prevents you from writing certain programs. That's its job. You write a program, present it to the type checker, and the type checker says: "No, you can't write that, sorry." And in particular, type systems are designed in such a way that the type checker prevents you from writing "bad" programs. Programs that have errors. So, in that sense, yes, a static type system helps in developing large systems.

However, there is a problem: we have the Halting Problem, Rice's Theorem and many other Incompleteness Theorems which basically tell us one thing: it is impossible to write a type checker which can always determine whether a program is type-safe or not. There will always be an infinite number of programs for which the type checker can't decide whether they are type-safe or not. And there is only one sane thing to do for the type checker: reject these programs as not type-safe. And an infinite number of those programs will, in fact, not be type-safe. However, also an infinite number of those programs will be type-safe! And some of those will even be useful! So, the type checker has just prevented us from writing a useful, type-safe program, just because it cannot prove its type-safety.

IOW: the purpose of a type system is to limit expressiveness.

But, what if one of those rejected programs actually solves our problem in an elegant, easy to maintain manner? Then we cannot write that program.

I'd say it's basically a give-and-take: statically typed languages restrict you from writing bad programs at the expense of occasionally also preventing you from writing good programs. Dynamic languages don't prevent you from writing good programs at the expense of also not preventing you from writing bad programs.

The more important aspect for maintainability of large systems is expressiveness, simply because you don't need to create as large and complex a system in the first place.

Answer 4

Explicit static types are a universally-understood and guaranteed correct form of documentation which is not available in dynamic languages. If this is not compensated for, your dynamic code will simply be harder to read and understand.

Answer 5

Consider a large codebase including database bindings and a rich testsuite and let me highlight a few advantages of static languages over dynamic languages. (Some examples may be idiosyncratic and not apply to any static or dynamic language.)

The general idea—as others pointed that out—is that the type system is a “dimension” of your program which exposes some information to automated tools processing your program (compiler, code analysis tools, etc.). With a dynamic language, this information is basically stripped and therefore not available. With a static language, this information can be used to help writing correct programs.

When you fix a bug, you start with a program that looks good to your compiler but has faulty logic. When you fix the bug, you make an edit fixing locally the logic of your program (e.g. within a class) but breaking this logic at other places (e.g. classes collaborating with the previous one). Since a program written in a static language exposes much more information to the compiler¹ than a program written in a dynamic language, the compiler will help you to locate the other places where the logic breaks more than a compiler for a dynamic language will do. This is because a local modification will break the type correctness of the program at other places, thus forcing you to fix the type correctness globally before having a chance to run the program again.

A static language enforce type-correctness of a program, and you can assume that all type errors you encounter when working on the program would correspond to a runtime failure in an hypothetical translation of the program in a dynamic language, thus the former has less bugs than the latter. As a consequence, it requires less coverage tests, less unit tests and less bugfixes, in one word, it is easier to maintain.

Of course, there is a tradeoff: while it is possible to expose a lot of information in the type system and thus taking the chance to write reliable programs, it might be difficult to combine this with a flexible~API.

Here is a few examples of information that one can encode in the type system:

— Const correctness the compiler can guarantee that a value is passed “read-only” to a procedure.²

— Database schema the compiler can guarantee that code binding the program to a database corresponds to the database definition. This is very useful when this definition changes. (Maintainance!)

— System resources the compiler can guarantee that code using a system resourcce only does it when the resource is in the correct state. For instance, it is possible to encode the attribute close or open of a file in the type system.

¹ It is not useful to distinguish between a compiler and an interpreter here, if such a difference exists.

Answer 6

Because static typing enables better tooling, which improves the productivity of a programmer when he tries to understand, refactor or extend a large existing code base.

For instance, in a large program, we'll likely have several methods with the same name. We instance, we might have an add method that adds something to a set, another that adds two integers, another that deposits money into a bank account, ...). In small programs, such name collisions are unlikely to occur. In large programs worked on by several people, they occur naturally.

In a statically typed language, such methods can be distinguished by the types they operate on. In particular, a development environment can discover, for each method invocation expression, which method is being invoked, enabling it to show a tooltip with that method's documentation, find all call sites for a method, or to support refactorings (such as method inlining, method renaming, modifying the parameter list, ...).