User-Friendly Documentation and Validation Processes

Part 2: Documentation Makes for a Good System. What Makes for a Good System of Documentation? A Vision with Documented Strategy and Tactics

Validity is the ultimate goal of any system. The concept of validity is relatively straightforward: it means that an instrument, or system, measures what it is purported to measure. Getting valid results in practice is a little more complicated. The validation process is greatly facilitated by, and perhaps even dependent upon, the availability of complete and user-friendly documentation. After all, an ineffective system of documentation can lead to ineffective implementation, which throws into question the validity of the entire system and its outputs. Well-designed or well-crafted user-friendly documentation is the means by which these systems can be quality controlled, validated, and replicated (for suggestions on how quality control processes should be implemented see D’Brot and Keng, 2018).

And while the guidance that high-utility resources and documentation provide is essential (see my previous post) as we think about large and complex systems like accountability, when it comes to documentation, we’ve often only built minimally viable products. 

What Do I Mean by the Term Minimally Viable Product? 

In general, a minimum viable product is one that has just enough features to be usable. For example, have you ever purchased a product or piece of furniture that came in a very flat box? Typically, you know you’re in for a treat that includes some non-descript directions walking you through several pictures and many small connector pieces. 

Like other task-specific activities, it does get easier with practice. And while perhaps easier with practice, only those with the greatest direction-following prowess will ever see the task as easy. I’d guess that most folks have experienced some frustration with this style of furniture. Despite the best intentions of those that produced those directions, it is likely that for that piece of furniture, these directions are the minimum viable product that is, on average, accessible to the typical customer.  

Building box furniture while dealing with less-than-helpful directions is a lot like thinking about documentation and validation processes in educational systems. Like furniture building, you can’t make progress without the right documentation or the access points to get to those documents. 

It is perhaps helpful, therefore, to reconsider our current practices in documenting educational assessment and accountability systems in order to meet the need for systems of documentation that go beyond minimum viable products. 

Moving Technical Documentation Beyond the Minimum Viable Product

Documentation is often crowned as king in an effort to build legacy, build process, and promote replicability. However, there is a difference between good documentation and documentation for documentation’s sake.

We often produce the minimum viable product due to cost, personnel, and time constraints, and, frankly, to be able to check off the Technical Documentation box in the list of project deliverables. But we cannot be satisfied with simply completing a product—especially if it has just enough features to be useful only to expert users. We should strive toward not only a viable set of documentation but high-utility documentation. This high-utility documentation can help tell a procedural story that enables the current (and next) user to make sense of their current context, as well as provide them with the next steps in the process that support intended outcomes. 

I argue that the documentation should meet the same criteria for quality as the system it is supporting. High-quality assessment systems should demonstrate the following characteristics: coherence, comprehensiveness, continuity, efficiency, and utility. These characteristics can be applied to the documentation of systems and processes that require validation. I describe these in relation to documentation and the validation of systems below, but readers are invited to review the criteria in their original context if desired (description of assessment characteristics adapted from NRC, 2001; Chattergoon & Marion, 2016; Marion, et al, 2019.)

1. Coherence: In system documentation, we have to determine and articulate the intended target, or intent, of each piece of documentation. That is, what role does documentation and its associated processes individually serve and how does it serve the larger validation and replication process? This applies both vertically and horizontally. 
2. Vertical coherence: In documentation and system processes, there must be vertical coherence that begins with a clear purpose at the top and aligns with each incrementally more detailed piece of documentation within a system. The more detailed documentation (e.g., the models used, analyses, decision points, data flows, and data aggregations) should then reflect the intent of the more general without inconsistencies or contradictions.   
3. Horizontal coherence: Horizontally coherent documentation and validation rely on a common vision, purpose, or outcome that spans across decisions, initiatives, outputs, and outcomes. Like vertical coherence, documentation must be free of inconsistencies or contradictions. 
4. Comprehensiveness: The purposes, uses, claims, and constraints of data- or decision-processes must be clearly articulated in order to develop the appropriate documentation. That documentation must also be inclusive enough to address decisions across the system to support reproduction, validation, and comparison along the way. 
5. Continuity: Documentation, like validation processes, needs to be continuous because it must reflect not only the processes associated with individual decisions but also the entirety of the pathway of a series of decisions, if applicable. Validation processes and the associated documentation must enable users to monitor validation progress within systems, which will then support the monitoring of progress across the system or across sub-systems. 
6. Efficiency: Efficiency refers to the elimination of redundant decisions, decision points, and validation steps. Note that redundancy differs from the idea of replication for validation (see D’Brot & Keng, 2018), which often requires the use of multiple processes to determine whether one can arrive at the same answer following a different set of steps.  
7. Utility: Finally, utility is about providing “the information necessary to support…multiple and often diverse purposes (Marion, et. al, 2019, p. 6).” Often, these diverse purposes can be a byproduct of the number and types of stakeholders who will be using the documentation, making decisions, and using the data from the system. The identification of these stakeholders will help clarify what sources of evidence are necessary to support decision-making to the relevant processes or data. Ultimately, the documentation and validation processes must be perceived as useful in order to be used. And that utility may differ depending on the presentation and structure of documentation depending on the person using them.  

But the Greatest of These is Utility

The first four criteria above are obvious needs when building high-quality documentation in support of validation processes. However, I would argue that the first four criteria will not be evident to a majority of users if there is not a focus on the Utility criterion throughout. The Utility criterion can help system designers—and those who must document the processes and decisions along the way—consider how to make the coherence, comprehensiveness, continuity, and efficiency criteria obvious. 

The attention paid to the Utility criterion likely will end up defining (or restricting) the type of user who has the capacity and capability to use the documentation available. The more useful your documentation, the broader your user base can be. The narrower the perceived usefulness of your documentation, the fewer the number of people who will actually use it. 

Recall the example of furniture building. There is a reason why services like Ikea and Amazon now offer in-home assembly; not everyone is a super-user with a wealth of experience building box furniture. But, the stakes are usually low enough that most of us will take a stab at it. I’m advocating for making the furniture building process (as well as the implementation and validation of educational assessment and accountability systems) a little more user-friendly by investing in that last push to prioritize utility, not just viability. 

The execution and development of high-quality documentation to support high-quality validation processes is complex but requires an intentional focus on ease of use to promote effective use.