On Agility and Architectural Engineering

Agile Concepts and Methods are All the Rage

Social media and market surveys indicate an increasing number of organizations reporting that they are applying agile practices (see e.g. [1]). Those practices are used by a wide range of organizations with different sizes, geographic locations, market roles and industry focus. At first, they were deployed on small- to medium-sized projects with reasonable time frames and co-located teams. However, there is a trend towards larger projects including distributed development teams and the adoption of agile practices on corporate level referred to as scaled agile. The implementation of agile practices is driven by the demand to accelerate the delivery of software solutions and enhance the ability to respond to changes. In addition, organizations using agile practices aim for increasing productivity, improving business and IT alignment and enhancing software quality.

Performance of Agile Projects

Despite the increasing adoption level of agile practices, a considerable number of projects still fall short of their expectations, i.e. they deviate in relation to dimensions like business value, cost and time control, productivity and technical quality (see e.g. [2]). The performance of an agile project should, of course, be rigorously measured as any other project. However, underachieving agile projects can be distinguished by a couple of outstanding symptoms:

• The teams spend most of their time fixing defects and the development of new capabilities/features/stories stagnates.
• The integration of components developed by different teams leads to a lot of incompatibility defects and requires a significant amount of rework in order achieve system qualities (i.e. performance) from an end-to-end perspective.
• Demands for refactoring surprisingly emerge and the overall effort for refactoring continuously swells up.
• The costs for testing raise and in order to supposely reduce the overall performance some testing activities are discarded.

All these symptoms lead to a continuous degradation of the project performance and particularly a decrease of the technical quality. The root causes of these developments are manifold but can be fundamentally classified as either nontechnical or technical. Nontechnical root causes encompass for example aspects of culture, business support and stakeholder management. Root causes classified as technical involve aspects like development processes, quality attribute requirements and software architecture.

Software Architecture Practices as a Critical Success Factor

Particularly the software architecture is from my experience a critical but often discarded critical success factor in agile projects. The root cause of all symptoms which are listed before can be strongly related to the negligence of software architecture practices. This is surprising to me because there are a lot of architectural concepts and methods available. Nevertheless, those architectural practices seem to be ignored by many agile projects.

Misconceptions About Software Architecture Practices

One reason for the negligence of software architecture practices could be that they are simply not known. In addition, the agile methods are rather silent on architectural concepts and methods (cp. [3]). They don’t elaborate for example on how to identify architecturally significant requirements, perform incremental architectural design and validate architectural features. Particularly, they don’t reference or reuse existing architectural concepts and methods. They are essentially monolithic, have a homegrown presentation and no common ground (see [4]). Another reason is from my perspective strongly related to some myths and misconceptions of the agile manifesto, particularly its eleventh principle (see [5]):

The best architectures, requirements, and designs emerge from self-organizing teams.

There are a lot of proven concepts and methods available which support the design, evaluation, documentation and implementation of software architectures. Therefore, architectures can be engineered. They don’t emerge. The state-of-the-art architectural practices might not be well known but they are significant and valuable for every project – independent of the chosen development methodology (agile, plan-driven or hybrid) (see [6]). Moreover, although it is often asserted, there is no tension between agile and architecture. The challenge is not whether to adopt agile or architectural practices but how to best integrate agile and architectural practices.

Architecture and Agility Practices Enable Each Other

For medium and large software development projects¹⁾, agility is enabled by architecture and vice versa (cp. [8]). The architectural practices benefit from the agile practices because shorter iterations enable the incremental design and implementation of the architecture and provide faster feedback loops. As the stakeholder’s demands evolve, functional and architectural requirements can be incrementally elicited. The dependencies between functional and architectural requirements can be proactively managed to ensure that all necessary architectural elements are available in upcoming iterations (see e.g. the zipper model) (see [8]). The architecture is developed over several iterations and continuously validated by the features and functions which leverage the architecture. The agile practices, in turn, benefit from the architectural practices because the architecture enables an agile project to define an implementation structure that structures the organization into teams considering the need to minimize the coupling between teams. The work assignment view is one fundamental architectural design perspective (see [9]). It captures the mapping between the implementation modules of a system and the people, teams and organizational units who are in charge for developing those modules. Therefore, the work assignment view supports the planning and managing of team allocations by assigning the responsibility for builds and explaining the structure of the project. Dedicated tactics enable a simultaneous focus on the architecture and agile development team by aligning the feature-based development and system decomposition in order to minimize coupling between the teams (see [7]). Furthermore, the architectural practices bring along proven concepts, methods, architectural tactics and pattern which accelerate the overall development process and ensure technical quality.

Closing the Knowledge Gap

The need for agility in a lot of software development projects is uncontested. However, particularly medium to large agile projects fall short of their expectations. I assert that the poor performance of those projects is connected with the negligence of software architecture practices. It doesn’t matter whether those architectural practices are underrepresented in agile methods or whether they are simple not known. There is a significant knowledge gap related to architectural concepts and methods, and how to integrate them with agile practices. Therefore, I kicked off a new endeavor to tackle this knowledge gap. The objective is to dispel the idea that there is a tension between agile and architectural practices and make the existing architectural knowledge transparent and applicable. The endeavor’s deliverables encompass a Slidedoc²⁾ document, a set of frameworks and tools, and accompanying blog posts and talks. All deliverables will be developed incrementally and are driven by the idea to inspire collaboration and knowledge sharing within the community of architects.

Perspectives on Integrating Agile and Architectural Practices

While immersing myself in the topic and endeavor, I’ve carved out several perspectives from which the integration of agile and architectural practices should be considered (see figure 1).

Agile and architectural practices are based on various concepts and methods. The concept and method perspective examines which and how those concepts and methods can be integrated. The organizational perspective considers how a program/project/team should organize in order to minimize the coupling between teams and thus accelerate the software delivery. Furthermore, this perspective deals with dedicated tactics that can be used to establish a simultaneous focus on the architecture and agile development teams by aligning the continuous delivery of capabilities/user stories/features with the incremental development of the software architecture. The architectural design perspective is concerned with the software architecture as a set of structures and how it can enable agility. This perspective covers the quality attributes, tactics and patterns that enhance the ability to respond to changes, increase productivity, etc. The perspective on technical debt discusses how technical debt can be proactively managed. In the domain of developing software-intensive systems, the technical debt concept involves design or implementation constructs that are expedient in the short term but set up a technical context that can make future change more costly or impossible [10]. The concept is likewise relevant in the context of agile, plan-driven or hybrid development methodologies. However, as agile methods aim at accelerating the delivery of software solutions, they exceedingly tend towards accumulating technical debt. I’m going to use those perspectives as a high-level structure for all deliverables, particularly the Slidedoc document.

Summary and Conclusion

The adoption level of agile practices is increasing but a considerable number of projects still fall short of their expectations. Especially the software architecture is a critical but often discarded critical success factor in agile projects. State-of-the-art architectural practices are available and provide value for every project – independent of the chosen development methodology (agile, plan-driven or hybrid). Particularly, there is no tension between agile and architectural practices. In fact, both enable each other and there are significant benefits of having them integrated. However, there is a significant knowledge gap related to architectural concepts and methods, and how to integrate them with agile practices. The aforementioned endeavor aims at closing this knowledge gap by making the existing architectural concepts and methods transparent and applicable, and providing further guidance on how to integrate agile and architectural practices.

Notes

1) The size of project is typically defined as a function of scope, team size and duration. A medium size project consists of more than approx. 20 people who are assigned to different teams and might be geographically distributed. The project duration typically extends beyond one year.
2) A Slidedoc is a document which was developed using presentation software and uses visuals and words to convey well-structured content without the attendance of a presenter. It is a medium that can be read easily and digested more quickly than either a document or a presentation. Nancy Duarte described the Slidedoc concept by using the concept itself: A Slidedoc in PowerPoint© format. See: Duarte, Nancy: Slidedocs – Spread ideas with effective visual documents.

References

[1] CollabNet VersionOne: 13th Annual State of the Agile Report. May 2019.
[2] Jørgensen, Magne: Relationships Between Project Size, Agile Practices, and Successful Software Development – Results and Analysis. In: IEEE Software, Volume 36, Issue 2, March-April 2019.
[3] Abrahamsson, P.; Babar, M.A.; Kruchten, P.: Agility and Architecture – Can They Coexist? In IEEE Software, Volume 27, Issue 2, March/April 2010.
[4] Jacobson, I.; Stimson, R.: Tear Down the Method Prisons! Set Free the Practices! In: acmqueue, Volume 16, Issue 5, September-October 2018.
[5] Beck, K.; Beedle, M.; van Bennekum, A.; Cockburn, A. et al.: Manifesto for Agile Software Development.
[6] Boehm, B.; Turner, R.: Balancing Agility and Discipline: A Guide for the Perplexed. Addison-Wesley, Boston et al. 2003.
[7] Nord, R.L.; Ozkaya, I.; Kruchten, P.: Agile in Distress: Architecture to the Rescue. In: Dingsøyr, T.; Moe, N.B.; Tonelli, R.; Counsell, S.; Gencel, C.; Petersen, K. (eds): Agile Methods. Large-Scale Development, Refactoring, Testing, and Estimation. XP 2014. Lecture Notes in Business Information Processing, Volume 199, Springer, 2014.
[8] Bellomo, S.; Kruchten, P.; Nord, R.L.; Ozkaya, I.: How to Agilely Architect an Agile Architecture. In: Cutter IT Journal of Information Technology Management, Volume 27, Number 2, February 2014.
[9] Bass, Len; Clements, Paul; Kazman, Rick: Software Architecture in Practice. 3rd Edition, Addison-Wesley, Boston et al. 2012.
[10] Kruchten, P.; Nord, R.L.; Ozkaya, I.: Managing Technical Debt: Reducing Friction in Software Development. Addison Wesley, Boston et al. 2019.