Tag Archives: grounded systems modeling

Augmented Qualitative Analysis (AQA) and Large Language Models (LLMs)

Conducting experiments in the automated labelling of topics generated using the Augmented Qualitative Analysis (AQA) process outlined in an earlier post has resulted in some observations that have some bearing on the use of Large Language Models (LLMs) in Soft OR/PSM practice.

The starting point for AQA was the partial automation of some of the elements of qualitative analysis, which has resulted in the use of probabilistic topic models to code text in a corpus at the paragraph level and to produce maps of the interrelationship of concepts (qua topics). These maps of interrelationships can only be put forward for consideration as potentially causal links after the topics have been labelled. A map of topic X being linked to topic Y n times is only of statistical interest. We need meaning to be attached to the topics – ideally a process of consideration by a group that ‘owns’ the raw data – before we can produce a putative causal loop diagram (CLD).

To ground this technique in traditions of qualitative analysis would require the labelling of topics to proceed through an inductive process of inspecting the term lists and inspecting the text coded by the topics to build-up an understanding of what the topic means to the stakeholders (e.g., see Croidieu and Kim (2017)). This is a back-and-forth process that continues until all the topics have been labelled. The fact that the map can be updated with these topic labels in parallel provides an additional perspective on the understanding of the meaning of the topics. 

With the advent of LLMs it is possible to feed the term lists and the example text coded by a topic – and even the map of interrelationships – into a tool like ChatGPT with the purpose of generating topic labels. However, experiments in doing this have produced disappointing results, despite extensive efforts in refining prompts. From the perspective of a qualitative researcher, the coding seems to be too much in the text, too in-vivo. Despite attempts to get the LLM to draw on the breadth of its training data there seemed little evidence of the sort of theorising from the data that is a key feature of qualitative analysis (Hannigan et al., 2019).

This is clearly a new area and other researchers have conducted experiments on precisely this point of prompt engineering e.g., see Barua, Widmer and Hitzler (2024). However, there is still the sense that a LLM is operating as nothing more than a ‘stochastic parrot’ (Bender et al., 2021). Further, coupling the outputs from a probabilistic topic model to a LLM are unlikely to generate the sort of management insight that is discussed by Hannigan et al. (2019); although the putative causal maps are likely to make sense to the participants in a group, and are statistically justified. Ultimately, the use of LLMs in a process of problem structuring is only ever going to be limited. Problematising is a human activity, it requires a felt-sense of a situation being problematic for an intent to intervene to emerge. Asking a LLM to feel something is a wayward expectation.

The recommendation here, for any group working with a large and potentially growing corpus of documents and in need of a technique that supports rapid problematisation, is to work with two Group Support Systems (GSS). The first presents an interactive means of exploring the probabilistic topic model (e.g., using pyLDAvis the topic model for the 2012 Olympics data set discussed in a previous post can be explored here) combined with a means of investigating the text as coded by the topic model i.e., selecting text that is coded by a specific topic and inductively generating a topic label that has meaning to the group. In effect, replicating some of the features of a Computer Aided Qualitative Data Analysis Software (CAQDAS). The fully labelled model can then be taken into a strategy-making workshop supported by the second GSS, in this case Strategyfinder. 

The prospects of these two GSS merging into a single Problem Structuring Platform is the subject of my upcoming talk at OR66, see my previous post on AQA.

Barua, A., Widmer, C., & Hitzler, P. (2024). Concept Induction using LLMs: a user experiment for assessment. https://doi.org/10.48550/arXiv.2404.11875

Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021). On the dangers of stochastic parrots: Can language models be too big? FAccT 2021 – Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, https://doi.org/10.1145/3442188.3445922

Croidieu, G., & Kim, P. H. (2017). Labor of Love: Amateurs and Lay-expertise Legitimation in the Early U.S. Radio Field. Administrative Science Quarterly, 63(1), 1-42. https://doi.org/10.1177/0001839216686531

Hannigan, T. R., Haan, R. F. J., Vakili, K., Tchalian, H., Glaser, V. L., Wang, M. S., Kaplan, S., & Jennings, P. D. (2019). Topic modeling in management research: Rendering new theory from textual data. Academy of Management Annals, 13(2), 586-632. https://doi.org/10.5465/annals.2017.0099

Augmented Qualitative Analysis (AQA)

I first started working on the semi-automated construction of Causal Loop Diagrams (CLDs) as part of a process of qualitative analysis for my MBA project. This work was developed into a full paper and published in the European Journal of Operational Research in 2013.

Yearworth, M., & White, L. (2013). The Uses of Qualitative Data in Multimethodology: Developing Causal Loop Diagrams During the Coding Process. European Journal of Operational Research, 231(1), 151-161. https://doi.org/10.1016/j.ejor.2013.05.002 [post-print version]

Judging by the citations, the methodology described in this paper has been used to develop CLDs across a wide range of application areas.

At the time of writing, the methodology was supported by the use of conventional Computer Aided Qualitative Data Analysis Software (CAQDAS) and, specifically, the use of matrix queries to compute the number of times pairs of concepts (codes) are related by the fact that they co-code paragraphs of text in the sources. The resulting adjacency matrix could be interpreted as a preliminary CLD and an input to further analysis.

Since then, the emergence of probabilistic topic modelling based on the Latent Dirichlet allocation (LDA) means that it is now possible to automate the coding process for very large collections of documents (hundreds to thousands), comprising millions to hundreds of millions of words. The co-coding technique described in Yearworth and White (2013) can be similarly applied by feeding the corpus into the topic model and counting and thresholding the resultant classification of paragraphs by topic(s). This process still requires an inductive1 bridge – in deciding i) a meaningful number of topics (k) to be used for the topic model, ii) the labelling of the topics based on an exploration of the term lists and a re-reading of the text coded by each topic, and iii) a decision about the meaning of the links – can they be interpreted as causal? In practice the last two processes are bound together and interactive once a topic model of particular size has been chosen.

An example graph is shown below based on a corpus of documents assembled from archival material about the 2012 London Olympic Games. The corpus is relatively small for a machine learning technique, 170 documents and just over a million words. However, this is of the sort of volume of data that is starting to get beyond the abilities of a single qualitative researcher to analyse. I’ve called this territory a hinterland for qualitative analysis and hence the motivation for an augmented approach. The graph presented here has been coloured according to betweenness centrality.

The graph was automatically translated into the JSON format supported by the Strategyfinder platform and is shown as a potential causal map below. Once imported into Strategyfinder it is then a further process to discuss the meaning of the relationship between statements i.e., the nature of the links, whether they represent a causal relationship, and their directionality. An export filter to create MDL files suitable for import into Vensim also exists.


I discuss this technique further in Chapters 14 and 15 of my forthcoming book Problem Structuring : Methodology in Practice in the context of a deployment of the technique via a problem structuring platform as a type of Group Support System (GSS). This is also the subject of my upcoming talk at OR66.

Annual OR Conference OR66
Details of the my talk at OR66
  1. I’ve labelled this an inductive step because at this stage in a process of problem structuring this is what it feels like you’re doing. However, situated inside a wider problem structuring process loop that includes modelling and taking action then it could be considered as abductive reasoning. ↩︎

A Systems Reading List

I often get asked to recommend books on systems thinking, systemic problem structuring, and systems modelling – from general introductions to specialist texts. In this update I have reduced the list to a more manageable length and split it into two parts – essential and further reading. Note that I recommend the 1999 versions of ‘Systems thinking, systems practice‘ and ‘Soft Systems Methodology in Action’ since they both include Checkland’s excellent reflections on 30-years’ of Soft Systems Methodology (SSM). If you are learning about and using SSM then I think you also need to know something about Strategic Options Development and Analysis (SODA)/JourneyMaking and the Strategic Choice Approach (SCA).

Essential Reading

  • Ackermann, F., & Eden, C. (2011). Making strategy : mapping out strategic success (2nd ed) London: Sage.
  • Beer, S. (1985). Diagnosing the system. Chichester: John Wiley & Sons Ltd.
  • Checkland, P. (1999). Systems thinking, systems practice: Including a 30-year retrospective. Chichester: John Wiley & Sons Ltd.
  • Checkland, P., & Poulter, J. (2006). Learning for action : a short definitive account of soft systems methodology, and its use for practitioner, teachers and students. Chichester: John Wiley & Sons Ltd.
  • Checkland, P., & Scholes, J. (1999). Soft Systems Methodology in Action: Including a 30-year retrospective. Chichester: John Wiley & Sons Ltd.
  • Jackson, M.C. (2019). Critical Systems Thinking and the Management of Complexity. Chichester: Wiley-Blackwell.
  • Midgley, G. (2000). Systemic intervention : philosophy, methodology, and practice. New York: Kluwer Academic/Plenum.
  • Mingers, J., & Rosenhead, J. (eds) (2001). Rational analysis for a problematic world revisited : problem structuring methods for complexity, uncertainty and conflict (2nd ed). Chichester: John Wiley & Sons Ltd.
  • Pidd, M. (2004). Systems modelling : theory and practice. Chichester: Chichester: John Wiley & Sons Ltd.
  • Pidd, M. (2010). Tools for thinking : modelling in management science (3rd ed). Chichester: John Wiley & Sons Ltd.
  • Sterman, J.D. (2000). Business dynamics : systems thinking and modeling for a complex world. Boston, Mass.: Irwin McGraw-Hill.
  • Vennix, J. (1996). Group Model Building: Facilitating Team Learning Using System Dynamics. Chichester: John Wiley & Sons Ltd.

Further Reading

  • Ackoff, R.L., & Emery, F.E. (1972). On purposeful systems. London: Tavistock Publications.
  • Coyle, R.G. (2004). Practical strategy : structured tools and techniques. Harlow: Financial Times Prentice Hall.
  • Friend, J.K., & Hickling, A. (2005). Planning under pressure: the strategic choice approach (3rd ed). Oxford: Elsevier Butterworth-Heinemann.
  • Jackson, M.C. (2003). Systems thinking: creative holism for managers. Chichester: John Wiley & Sons Ltd.
  • Midgley, G., & Ochoa-Arias, A. (2004). Community operational research : OR and systems thinking for community development. New York ; London: Kluwer Academic/Plenum.
  • Morecroft, J.D.W. (2007). Strategic modelling and business dynamics : a feedback systems approach. Hoboken, N.J.: Wiley
  • Ramage, M., & Shipp, K. (2009). Systems Thinkers. London: Springer.
  • Richardson, G.P. (1991). Feedback thought in social science and systems theory. Philadelphia: University of Pennsylvania Press.
  • Senge, P.M. (1990). The Fifth Discipline: The Art and Practice of the Learning Organization. London: Random House.

Systems modelling in engineering

Systems modelling in engineering

The wider and more pervasive use of appropriate systems modelling techniques would have a beneficial impact on the way in which engineers deal with messy socio-technical problems. This class of problems is commonly defined by the following characteristics; i) difficulty on agreeing the problem, project objectives, or what constitutes success, ii) situations involving many interested parties with different worldviews, iii) many uncertainties and lack of reliable (or any) data, and iv) working across the boundary between human activity systems and engineered artefacts. All systems models attempt to conceptualise, via appropriate abstraction and specialised semantics, the behaviour of complex systems through the notion of interdependent system elements combining and interacting to account for the emergent behavioural phenomena we observe in the world.

Engineers have developed a multitude of approaches to systems modelling such as Causal Loop Diagrams (CLDs) and System Dynamics (SD), Discrete Event Modelling (DEM), Agent Based Modelling and simulation (ABM), and Interpretive Structural Modelling (ISM) and these are all included in my programme of research.   However, despite their extensive use, there still exists a number of research challenges that must be addressed for these systems modelling approaches to be more widely adopted in engineering practice as essential tools for dealing with messy problems. These systems modelling approaches as used in current engineering practice provide little or no account of how the process of modelling relates to the process of intervention (if any). This is in part due to the wider challenge to address the poor awareness and uptake of Problem Structuring Methods (PSMs) in engineering, the current inadequate way of integrating these more engineering-focussed systems modelling approaches into PSMs, and lack of understanding in how to deploy them appropriately in addressing messy problems in specific contexts. There is also the need to interpret the current state of the social-theoretic underpinning to systems modelling into a form that is appropriate for use in engineering. This need arises from the endemic atheoretical pragmatism that exists in engineering practice. The lack of methodology supported by suitable theory to counter this i) hinders the development of understanding why methods work or not, and also what it means for them to work, ii) acts as a barrier to communication between practitioners and disciplines, and iii) has ethical consequences, as pragmatic use of methods raises the problem of instrumentalism.

Addressing this methodological challenge is currently a central core of my work. I believe this research is transformational in that it integrates academically disparate areas of expertise in engineering, management, and social science, into a coherent articulation of systems modelling for engineers.