Falling Short on Training Software Developers
I worry daily about the software we rely on in our daily lives. One need only look at the problems our airlines have had and the disruptions they have caused in our lives. I feel like there is a significant risk that we will be crushed under the weight of technical debt in this software. As an industry, we have got to start getting ahead of the mismatch between the complexity of the problems we are trying to solve and the approaches we are taking to manage its complexity. We need only look back to the Healthcare.gov debacle to see our failures on the grandest stage. Interestingly, our ability to deliver large and complex projects successfully still hovers around 5%, which is appalling.
It is 2017 and we still see many instances of teams and projects that, while they may be leveraging agile processes for project management, have no structure for translating those requirements into a software design. What’s more, we still see very few organizations effectively leveraging proven practices of test-driven design, automated integration testing, code reviews, and so on. I estimate that we have interacted with over 100 engineers from dozens of organizations over that last seven years and only a few (less than five) were working in environments where test-driven design or automated unit/integration testing was part of their development culture.
Why is this still happening in the face of all the benefits these structures and practices provide? I believe it is in large part due to how we train and develop software development professionals.
The Role of Formal Education
What role is formal education playing in this? Unlike my generation of software developers, most of the people we see entering the field today are coming from an educational background that is Computer Science-related (either a computer science/engineering degree or minor). I have said for years that the way we are training software developers (and engineers for that matter) in our universities is flawed.
I am not alone in my viewpoint. Thought leaders like David Parnas, Steve McConnell, and Fred Brooks (“A scientist builds in order to learn; an engineer learns in order to build.”) have all written about this. Research papers have been published based upon surveys showing the gap between what is taught in school and what is required in practice. We are asking educational institutions to assemble a single program of study that satisfies accreditation and simultaneously prepares people for either a career in academic research or a career in our industry. How is this any different than eliminating mechanical engineering as a degree program and asking a physics department to train people to build bridges, aircraft, robots, industrial machines, etc?
It seems absurd, right? But that is exactly what we are expecting the Computer Science programs of our universities to do. The reality is, these programs are more focused on in-depth coverage of the body of knowledge of computer science at the expense of the types of curriculum a true engineering program would have. Most programs I see have a single course on “software engineering” and no courses on best practices for software architecture or software design and development.
Sad, but true.
I am by no means indicting these programs for a failure to prepare students for the real world. The vast majority of kids coming out of college with these degrees find employment so the universities might not see that there is a problem. While I believe these students are underprepared, we as an industry need to demand a better product from our higher education system.
Until something changes, we will continue to see students enter the workforce with little or no understanding of what a mature software development team looks like, what the design criteria should be for decomposing a system into a coherent architecture and design, how to effectively evaluate design tradeoffs against a set of accepted design principles, and the benefit of following best practices for the design and development of their code.
If students are lucky, they will be entering the world as apprentices and hopefully land in an organization that has a mature and strong software design and development culture. Good luck. My experience is that these types of organizations are rare.
The Role of Professional Development
What role is professional development playing in training people once they get into the workforce? Tech conferences have historically been focused on introducing new technologies and frameworks. Occasionally there are presentations covering patterns and best practices, but I suspect most of the session presenters come from organizations that have a disciplined maturity to their development processes. A lot of what they are talking about almost assumes that the audience could take advantage of these frameworks, which are designed to enable the best practices and patterns they themselves were following.
I suspect what really happens is either the developer walks away from the conference frustrated that they would never be able to leverage the tools or the developer “implements” the tools and frameworks with less than optimal results. I have seen first-hand the results of teams implementing the WPF MVVM framework after working through the quintessential tutorial. They jump into using the framework without really understanding the MVVM pattern and are unable to make the nuanced decisions required to fully leverage the advertised benefits of developing a UI layer on top of this framework (separation of UI and business logic, testability of view models, etc). Our own experiences have made us wonder whether we would be better off with good ‘ol windows forms.
Bottom line: it seems like we as an industry are completely ignoring the need to coach up our development teams on these fundamental principles and practices.
What is the solution? I feel like we must address both the root cause of this problem (our university education programs) as well as addressing the way we are developing software developers already in the industry.
Fixing Our Education System
Changing the way we train software developers is not a new idea. Thought leaders in our industry such as David Parnas and Fred Brooks (especially his chapter discussing “Where Do Great Designers Come From?” in “The Design of Design”) have been arguing for this type of change for quite some time.
In 1999, Parnas wrote a paper proposing a curriculum designed to address many of the concerns I have outlined above. In it, he argues for separating the education of software engineers from computer scientists, allowing the software engineers to be trained in the style of the other engineering disciplines and emphasizing that being a software engineer is more than just being a good programmer.
Ever since I joined the advisory board for the University of Nebraska-Lincoln (UNL) College of Computer Science & Engineering, I have been advocating for a dedicated B.S. in Software Engineering that would be differentiated from not only the computer science program at UNL but similar programs throughout the country.
The idea for this began gaining traction a couple of years ago and eventually led to UNL formally launching their B.S. in Software Engineering last fall. While many aspects of this program are still being developed, it is encouraging to see industry best practices as part of the performance criteria for the students and inviting industry practitioners to talk on state of the practice.
I have often spoken with faculty about the need to establish behaviors and norms that are consistent with industry best practices. For example, making activities like test-driven design something the students understand and use naturally throughout their coursework. It is also very encouraging to see unique software engineering courses targeted at providing students the background and experience that will better prepare them to enter the workforce.
I believe the UNL software engineering program has the potential to be a model for an improved approach to preparing students for careers in software development.
Creating True Professional Development Opportunities
If everyone had a Robert Martin or Martin Fowler on their team, then we would have nothing to worry about. Unfortunately, many (if not most) teams do not have people with the depth of understanding and experience (or even the time) to train up their teams. We need to help these teams along their path to maturity by providing meaningful and effective professional development opportunities. Somehow, we “mere mortals” need to be equipped with tools, technologies, processes, and patterns that can help us be successful without being experts. Our goal should be that the journeyman software developer is able to effectively adopt these patterns and practices.
To be clear, I am not talking about traditional “code schools” as a solution. We can’t send people to code schools and expect them to be productive in a true software development team. Where will they be getting the background and experience on concepts like encapsulation? Information hiding? Loose coupling? SOLID principles?
These code schools can, at best, produce people who can augment a development team as a junior apprentice as they are certainly less prepared for the real world than those coming out of traditional computer science programs. These are the folks working on the “factory floor”. At least the computer science grads have some exposure and working knowledge of a broad set of concepts.
The answer to me is to build upon the foundation these computer science grads have gotten and provide the type of education that can help them make the connection between engineering concepts they may have learned and the benefits they provide. I’m imagining a program that provides hands-on experience in order to successfully apply these concepts in real-world scenarios, allows them to see how these patterns and practices will benefit them, demonstrates that software “rot” is not inevitable in every code base, and shows that “funability” is something every development team should experience.
This type of combined lecture/hands-on skill development is exactly what we are trying to build at Don’t Panic Labs through our Software Design & Development Clinics. These courses are focused on reinforcing concepts that software developers are familiar with and making them concrete and actionable. We show students how to apply these concepts, practices, and patterns in real-world scenarios of both new projects and, more importantly, legacy systems that have technical debt in the architecture, design, and code.
If we can instill people with the confidence to begin applying these practices and patterns on the maintenance of their legacy systems, then we will have achieved what we set out to do.
Wrapping it Up
What’s it going to take to make progress in this area? I know we at Don’t Panic Labs have attempted to do our part by engaging with our university (UNL) and developing professional development programs for engineers. Maybe this is how it will grow, through grassroots efforts.
My hope is we, as an industry, can get ahead of the curve before some catastrophic event occurs that results in a massive upheaval in our education system. That would be tragic. I worry that we have a lot of people who, while doing the best they can, are crippled by a lack of understanding or confidence to do what’s right and are making poor judgments that could have a significant impact on their companies’ ability to compete and innovate.
Let’s mobilize and bring forward those who have been left behind by the folks who have already adopted these best practices and patterns. Let’s raise the tide in a way that floats everyone’s boats. If we can do this, imagine where we could be as a community and a profession. I encourage you to share your thoughts below.
This post was originally published on the Don’t Panic Labs blog.
