On Agility and Architectural Engineering (Revisited)

Agile Concepts and Methods are All the Rage

Social media and market surveys indicate an increasing number of organizations reporting that they are applying agile practices (see e.g. [1]). Those practices are used by a wide range of organizations with different sizes, geographic locations, market roles and industry focus. At first, they were deployed on small- to medium-sized projects1) with reasonable time frames and co-located teams. However, there is a trend towards larger projects including distributed development teams and the adoption of agile practices on corporate level referred to as scaled agile. The implementation of agile practices is driven by the demand to accelerate the delivery of software solutions and enhance the ability to respond to changes. In addition, organizations using agile practices aim for increasing productivity, improving business and IT alignment, and enhancing software quality.

Performance of Agile Projects

Despite the increasing adoption level of agile practices, a considerable number of projects still fall short of their expectations, that means they deviate in relation to dimensions like business value, cost and time control, productivity, and technical quality (see e.g. [2]). The performance of an agile project should, of course, be rigorously measured as any other project. However, underachieving agile projects can be distinguished by a couple of outstanding symptoms:

  • The teams spend most of their time fixing defects and the development of new capabilities/features/stories stagnates.
  • The integration of components developed by different teams leads to a lot of incompatibility defects and requires a significant amount of rework in order achieve overall qualities (such as performance, reliability) from an end-to-end perspective.
  • Demands for refactoring surprisingly emerge and the overall effort for refactoring continuously swells up.
  • The costs for testing raise and to supposedly reduce the overall performance some testing activities are discarded.

All these symptoms lead to a continuous degradation of the project performance and particularly a decrease of the technical quality. The root causes of these developments are manifold but can be fundamentally classified as either nontechnical or technical. Nontechnical root causes encompass for example aspects of culture, business support and stakeholder management. Root causes classified as technical involve aspects like development processes, quality attribute requirements and software architecture. In this post, the focus is on the nontechnical root causes, particularly the problems caused by poor software architecture practices.

Software Architecture Practices as a Critical Success Factor

Particularly the software architecture is from my experience a critical but often discarded critical success factor in agile projects. The root cause of all symptoms which are listed before can be strongly related to the negligence of software architecture practices. This is surprising to me because there are a lot of architectural concepts and methods available. Nevertheless, those architectural practices seem to be ignored by many agile projects.

Misconceptions About Software Architecture Practices

One reason for the negligence of software architecture practices could be that they are simply not known. In addition, the agile methods are rather silent on architectural concepts and methods (cp. [3]). They don’t elaborate for example on how to identify architecturally significant requirements, perform incremental architectural design and validate architectural features. Particularly, they don’t reference or reuse existing architectural concepts and methods. They are essentially monolithic, have a homegrown presentation and no common ground (see [4]). Another reason is from my perspective strongly related to some myths and misconceptions of the agile manifesto, particularly its eleventh principle (see [5]):

The best architectures, requirements, and designs emerge from self-organizing teams.

There are a lot of proven concepts and methods available which support the design, evaluation, documentation, and implementation of software architectures. Therefore, architectures can be engineered2). They don’t emerge. The most adequate architectures are consciously designed by skilled, talented, trained, and experience architects who apply structured and proven architectural practices (cp. [10]). The state-of-the-art architectural practices might not be well known but they are significant and valuable for every project – independent of the chosen development methodology (agile, plan-driven or hybrid) (see [7]). Moreover, although it is often asserted, there is no tension between agile and architecture. The challenge is not whether to adopt agile or architectural practices but how to best integrate agile and architectural practices.

Architecture and Agility Practices Enable Each Other

For medium and large software development projects3), agility is enabled by architecture and vice versa (cp. [9]). The architectural practices benefit from the agile practices because shorter iterations enable the incremental design and implementation of the architecture and provide faster feedback loops. As the stakeholder’s demands evolve, functional and architectural requirements can be incrementally elicited. The dependencies between functional and architectural requirements can be proactively managed to ensure that all necessary architectural elements are available in upcoming iterations (see e.g. the zipper model) (see [9]). The architecture is developed over several iterations and continuously validated by the features and functions which leverage the architecture.

The agile practices, in turn, benefit from the architectural practices because the architecture enables an agile project to define an implementation structure that arranges the organization into teams considering the need to minimize the coupling between teams. The work assignment view is one fundamental architectural design perspective (see [10]). It captures the mapping between the implementation modules of a system and the people, teams and organizational units who are in charge for developing those modules. Therefore, the work assignment view supports the planning and managing of team allocations by assigning the responsibility for builds and explaining the structure of the project. Dedicated tactics enable a simultaneous focus on the architecture and agile development team by aligning the feature-based development and system decomposition to minimize coupling between the teams (see [8]). Furthermore, the architectural practices bring along proven concepts, methods, architectural tactics, and pattern which accelerate the overall development process and ensure technical quality.

Closing the Knowledge Gap

The need for agility in a lot of software development projects is uncontested. However, particularly medium to large agile projects fall short of their expectations. I assert that the poor performance of those projects relates to the negligence of software architecture practices. It doesn’t matter whether those architectural practices are underrepresented in agile methods or whether they are simply not known. There is a significant knowledge gap related to architectural concepts and methods, and how to integrate them with agile practices.

My initial idea was to develop a Slidedoc4) document, a set of frameworks, tools, and accompanying blog posts and talks to close this knowledge gap. However, while I was immersing myself in Essence, I came up with the idea that leveraging Essence is a much better approach to grapple with the knowledge gap. Therefore, I discarded the idea of developing a Slidedoc document and explored Essence. Essence isn’t a new practice or method. It is an open standard by the Object Management Group (see [11]) that defines a common ground for describing, adopting, and improving software engineering practices and methods. The Essence specification establishes a common vocabulary and framework which among other things helps practitioners to learn, adopt, and compose practices/methods. If the existing practices/methods related to software architecture would be available based on Essence, the adoption and integration of architectural practices would be significantly simplified. The description of the existing architectural practices would enable people and organizations to use of the right architectural practices for the right software development endeavors and in the right contexts. Particularly, it would support the integration of architectural practices with other software engineering practices which are already available based on Essence.

I’ve elaborated on the idea of developing a library of architectural practices which encompasses existing practices/methods related to software architecture described based on Essence. According to the concept of Essence the practices in this library should be a succinct summary of the most important things you need to remember about a practice. However, the Essence-based practice descriptions aren’t supposed to replace the books, articles, reports, etc. in which the existing architectural practices are documented. They are providing an additional access layer to the practices and make their adoption and integration easier. Moreover, the practices in this library can be further enhanced with frameworks, tools, and templates which help to learn, adopt, and integrate the practices. I’ve joined forces with the team of Ivar Jacobson International because they provide feature-rich tools related to Essence and have a lot of experience in describing practices/methods based on Essence. The library of architectural practices will be developed incrementally. It is driven by the idea to inspire collaboration and knowledge sharing within the community of architects and Essence practitioners. Particularly, I plan to collaborate closely with the owners of architectural practices and respect their intellectual property. The working title of all deliverables is the Architecture Practice Library (APL). I’ve already started to work on this library and there are several architectural practices based on Essence on the way. The development of the library will be accompanied by blog posts, articles, talks, and more.

Summary and Conclusion

The adoption level of agile practices is increasing but a considerable number of projects still fall short of their expectations. Especially the software architecture is a critical but often discarded critical success factor in agile projects. State-of-the-art architectural practices are available and provide value for every project – independent of the chosen development methodology (agile, plan-driven or hybrid). Particularly, there is no tension between agile and architectural practices. In fact, both enable each other and there are significant benefits of having them integrated. However, there is a significant knowledge gap related to architectural concepts and methods, and how to integrate them with agile practices.

Essence defines a common ground for describing, adopting, and improving all practices and methods related to software engineering. The adoption and integration of architectural practices is significantly simpler if the practices are described using Essence. Particularly, a library of architectural practice based on Essence can close the existing knowledge gap. Therefore, I’ve kicked off the work on the Architecture Practice Library (APL) which aims at establishing a library of architectural practices based on Essence. This library will be developed incrementally and collaboratively respecting existing intellectual property. There are already some architectural practices in the pipeline and more blog posts, articles, and talks yet to come. Stay tuned.

Please note that this blog post is revised version of the post ‘On Agility and Architectural Engineering’ published on my website on December 30th, 2019.

Notes

1) This post uses the term project although the ideas and arguments are also valid for software products.

2) There are three fundamental approaches to software architecture which are identified and described by George Fairbanks in [6]: Teams can ignore, embrace, or hoist software architecture. Accordingly, George Fairbanks distinguishes between the three approaches architecture-indifferent design, architecture-focused design, and architecture hoisting. All approaches are reasonable and I endorse this classification. The key point is that a team deliberately selects one approach as part of its way of working and is aware of all consequences.

3) The size of project is typically defined as a function of scope, team size and duration. There are varying size categories. As a rule of thumb consider medium size project consists of more than approx. 20 people who are assigned to different teams and might be geographically distributed. The project duration typically extends beyond one year. 4) A Slidedoc is a document which was developed using presentation software and uses visuals and words to convey well-structured content without the attendance of a presenter. It is a medium that can be read easily and digested more quickly than either a document or a presentation. Nancy Duarte described the Slidedoc concept by using the concept itself: A Slidedoc in PowerPoint© format. See: Duarte, Nancy: Slidedocs – Spread ideas with effective visual documents.

References

[1] Digital.ai: 15th State of the Agile Report. May 2021.
[2] Jørgensen, Magne: Relationships Between Project Size, Agile Practices, and Successful Software Development – Results and Analysis. In: IEEE Software, Volume 36, Issue 2, March-April 2019.
[3] Abrahamsson, P.; Babar, M.A.; Kruchten, P.: Agility and Architecture – Can They Coexist? In IEEE Software, Volume 27, Issue 2, March/April 2010.
[4] Jacobson, I.; Stimson, R.: Tear Down the Method Prisons! Set Free the Practices! In: acmqueue, Volume 16, Issue 5, September-October 2018.
[5] Beck, K.; Beedle, M.; van Bennekum, A.; Cockburn, A. et al.: Manifesto for Agile Software Development.
[6] Fairbanks, G.: Just Enough Software Architecture: A Risk-Driven Approach. Marshall & Brainerd, September 2010.
[7] Boehm, B.; Turner, R.: Balancing Agility and Discipline: A Guide for the Perplexed. Addison-Wesley, Boston et al. 2003.
[8] Nord, R.L.; Ozkaya, I.; Kruchten, P.: Agile in Distress: Architecture to the Rescue. In: Dingsøyr, T.; Moe, N.B.; Tonelli, R.; Counsell, S.; Gencel, C.; Petersen, K. (eds): Agile Methods. Large-Scale Development, Refactoring, Testing, and Estimation. XP 2014. Lecture Notes in Business Information Processing, Volume 199, Springer, 2014.
[9] Bellomo, S.; Kruchten, P.; Nord, R.L.; Ozkaya, I.: How to Agilely Architect an Agile Architecture. In: Cutter IT Journal of Information Technology Management, Volume 27, Number 2, February 2014.
[10] Bass, Len; Clements, Paul; Kazman, Rick: Software Architecture in Practice. 4th Edition, Addison-Wesley, Boston et al. 2021.
[11] Object Management Group: Essence – Kernel and Language for Software Engineering Methods. Standard Document, Version 1.2, October 2018

Leave a Reply