EVALUATING CONTROLLED DOCUMENT MANAGEMENT SYSTEM SOFTWARE

Published by Silkwood Software, makers of Elyse.

Introduction

A controlled document management system — used to manage standards, procedures, engineering drawings, and similar controlled information — should support the fundamental behaviors required of controlled documents without extensive custom construction. This is important when evaluating general-purpose document management products. A product may provide extensive document management functionality while its underlying model imposes limitations that become problematic when managing controlled documents. These limitations may result in expensive customization, edge-case failures, inappropriate constraints on users, or behaviors that cannot be reliably corrected through configuration.

The tests in this document are intended to expose the above limitations. They are not a feature checklist; they test fundamental functionality and architectural invariants that arise from controlled document use cases, including requirements that flow from standards such as ISO 9001. References to ISO 9001 here are not intended to imply that ISO 9001 mandates the specific functionality being tested, but rather that passing a test criterion provides evidence of a robust control designed to meet the requirements of the standard.

The underlying principle is the hierarchy of effectiveness of controls. Where a controlled document invariant can be enforced by the system, relying on a separate procedural control rather than an enforced software constraint introduces an avoidable opportunity for failure. In this context, an invariant is a condition that must remain true regardless of how the system is operated through its supported interfaces.

The tests focus on observable behavior rather than prescribing implementation. The underlying proposition is simple: the system's model constrains the behavior it can reliably provide to end users. A controlled-document system should be evaluated on the behaviors its underlying model can guarantee, not simply on the features it can be configured to display. The tests therefore seek to establish whether controlled document concepts are native to the system's model rather than something constructed from general-purpose features.

How to Use The Tests

The tests are intended to be performed against a working product, preferably using a vendor-provided demonstration or evaluation environment.

The tests should be demonstrable before any financial commitment to the product is required. If a test cannot be demonstrated without a financial commitment being made and development work being carried out, then the product does not pass the test.

Ask for each test to be demonstrated using the product's standard supported functionality. Do not accept a statement that the behavior “could be developed”, “could be achieved with customization”, or “would normally be handled by procedure” as a passing result. If it cannot be demonstrated then it does not pass.

Record the result of each test using the A–D scoring system. Where a result depends on configuration, record the configuration required to achieve it. Where custom development is required, record the nature of the development and the additional controls on which the resulting behavior depends.

The objective is not to determine which product has the most features. It is to determine whether the product's underlying model can reliably represent and enforce the required controlled document behavior.

These tests, and the use case scenarios on which they are based, are considered generic foundational, controlled document system requirements — not unusual, bespoke, or organization-specific needs that would justify a highly customized implementation.

Each test represents a situation that a real user, administrator, auditor or document controller may encounter, together with plausible failure scenarios that illustrate why the underlying behavior matters. The scenarios are illustrative; the pass criteria define the actual requirement. The tests were designed from real, observed controlled document failure modes. They should be a fair, generic bar that can be applied to any product being considered for managing controlled documents.

A failure is not simply a missing feature. It indicates that the system's underlying model may be unable to fully represent or enforce an expected controlled document management system behavior, or that attempts to do so may introduce further anomalies. A failure indicates that the required constraint is not managed at the software level and hence must be managed elsewhere, such as at the less robust procedural level. A failure also does not necessarily indicate poor product design. The product may be very powerful and suitable, but just not for the specific purpose of managing the particular requirements of controlled documents.

The use case scenarios frequently mention migration. The objective of migration is not to bring disorganized legacy documents under control. For a mature quality system, that discipline already exists. The objective is to ensure that the software does not degrade it.

Differences in terminology are largely irrelevant, provided that the actual system behavior matches the test.

Architectural Acceptance Tests

Test 1. Create a Document ID

Principles

User-facing document identity exists independently of content.

User-facing document identification is owned and controlled by the organization, not the software.

The user-facing document identification in the system must exactly match that displayed on the document, including the identifier displayed on documents which did not originate from that particular system. That is, the system must support arbitrary existing identifiers.

Use Cases and Failure Scenarios

An organization imports an established procedure, a large number of historical scanned engineering drawings, a supplier drawing or a vendor manual. Users will search for the document using the identifier that is actually printed on the document and on documents that refer to it. They may not be able to find the document if it is not registered under that exact identifier. Users should not be dependent on special tribal knowledge.

Users preparing drafts of a set of multiple cross-referenced documents need to have the document identifiers of the complete set registered before they start so they can include the cross references as they create the drafts.

A user might commence drafting a document as a PPTX file and then want to change to DOCX format without needing to re-register the document and change the document identifier, noting also that the document identifier might already have been written into other documents as a cross reference.

An organization is migrating tens of thousands of legacy documents and first needs to establish and finalize their identities in the system before loading the associated files.

Action

  • Create a document ID: PROC-OPS_01/02
  • Create another document ID: VENDOR_XYZ.UM-678

Pass Criteria

Document IDs must be able to be created as an organization-defined string. The user-facing identifier displayed by the system and searchable by users must be the organization's authoritative identifier – the same as what is printed on the documents.

The system must not impose character or format restrictions that prevent the registration of existing organizational or external document IDs exactly as they appear on the source document.

A document ID must be able to be created without submitting a file. The document identity must be an object separate from files — not simply metadata attached to a file.

Test 2. Identity Discovery

Principle

Finding a document must be an object lookup task, not a generic search or navigation function. This principle is specifically about document ID lookup.

Use Cases and Failure Scenarios

A user is given a newly-registered document ID by a colleague and needs to be able to find the document immediately, without waiting for the system to process or index it. If they cannot find it then they may conclude any of a number of incorrect reasons why they cannot find it, such as that there was a mistake and the ID they were given was not the one registered.

The only information a user has regarding a particular document is the ID. They do not know anything about which project, library or folder contains it, or any special information encoded into the ID. If the system requires more knowledge than just the document ID then the user will not be able to find it.

Action

  • Immediately after creating the above document IDs, conduct a lookup, as an ordinary user with no elevated privileges, for document ID: VENDOR_XYZ.UM-678

Pass Criteria

Following successful completion of the document ID creation operation, the document ID must be immediately discoverable by an ordinary user through the system's document ID lookup facility. Discoverability must not depend on a subsequent, separately scheduled or asynchronous indexing, crawling, harvesting, or synchronization process that may take an indeterminate or unacceptably long length of time to complete.

The document ID must be discoverable independently of the existence of a file.

The document ID must also be discoverable directly without first navigating to a folder, library, workspace, project or other location associated with the document. That is, the user must not require any knowledge other than the document ID.

Test 3. Document Identity Uniqueness

Principle

Document identity uniqueness is a strictly enforced invariant. This includes case insensitivity.

Use Cases and Failure Scenarios

ISO 9001 requires appropriate identification and description of documented information. For a controlled-document system, the organization's document identity must therefore be reliably distinguishable. If two different documents can have the same user-facing identity then the system cannot reliably distinguish which document a user is referring to.

A user may not know if the system is case sensitive and not realize that if they retrieve proc-ops_01/02 instead of PROC-OPS_01/02 they will get a different document.

Action

  • Attempt to create a new identical document ID: PROC-OPS_01/02
  • Attempt to create a new document ID: proc-ops_01/02

Pass Criteria

Both attempts must fail. The system must natively prevent creation of duplicate document IDs, including duplication arising through case sensitivity. The uniqueness must be constrained by the system, not merely by a user interface warning. The uniqueness must be a guaranteed constraint that is consistent across all supported ways of modifying or retrieving the controlled information. If the constraint is not a system invariant and requires custom programming to implement then that is an indication that there is a risk that the constraint will fail under some circumstances.

Test 4. Create a Release/Version/Revision

Principle

Document release control is a concept separate from file version control.

Use Cases and Failure Scenarios

An organization transfers its existing controlled documents into the new system and must retain revision identifiers already printed on the historical documents.

Action

  • Create a release against document ID PROC-OPS_01/02 identified as Revision ‘A’.

Pass Criteria

The system must allow the organization to migrate existing documents such that the release identifiers in the software match that printed on the pre-existing documents, without requiring existing documents to be modified. A release must be able to be created which is linked to the document identifier, not simply metadata attached to an individual file.

Test 5. Independent Release Identity and Historical State

Principle

Document release control is a concept separate from file version control.

Use Cases and Failure Scenarios

A document may have several files associated with a particular controlled release, such as a native file and an issued PDF, without those files becoming competing document identities. A supporting file may also be required as part of a release set, such as a change narrative or calculation sheet.

An organization needs to bring large volumes of existing controlled documents under management without artificially forcing historical data through workflows designed for newly created documents.

Action

  • Link a blank dummy file to Rev A.
  • Create Rev B.
  • Link a different blank dummy file to Rev B.
  • Create Rev C.
  • Link a different blank dummy file to Rev C.
  • Create Rev D.
  • Link a different blank dummy file to Rev D.
  • Link another different blank dummy file to Rev D.
  • Set Rev C as the authoritative release (e.g. ‘Released’, ‘Current’) and Rev A and Rev B as Superseded.

Pass Criteria

The system must be able to separate document identity from file identity. Each release must be independently identifiable and addressable as a release of the document, and the files associated with each release must be independently identifiable and associated with that release. Specifically, a release must not merely represent the historical edit state of a single file.

The system must have a mechanism for distinguishing which release of a document is authoritative.

The system must be capable of migrating pre-existing data and achieving a valid final state (e.g. ‘Authoritative’ or ‘Superseded’) directly without forcing every file through a workflow system.

Test 6. No Duplicate Authoritative Release

Principle

There must be no more than one authoritative release of a document, either under steady-state conditions or during a transition. This test asks: Can the system's data model permit two authoritative releases?

Use Cases and Failure Scenarios

ISO 9001 requires controls to prevent unintended use of obsolete documented information. A system that presents multiple releases through the default search and retrieval system creates opportunity for that control to fail. Presenting an ordinary user, searching via a default search interface, with only the authoritative release ensures that this type of failure cannot occur.

Users in different parts of the organization must not encounter two releases both presented as the current authoritative document.

Action

  • Attempt to designate Rev B as authoritative in addition to Rev C.

Pass Criteria

The system must enforce that a document ID can have no more than one authoritative release. This constraint must exist through all supported routes, such as UI, bulk import, API, workflow or administrative interface. The above action must either fail, or the release state of Revision C must be automatically changed to a non-authoritative state so that the single authoritative release condition is maintained.

Test 7. Document ID Search

Principle

Preventing unintended use of obsolete documents is a key requirement of controlled document management.

Use Cases and Failure Scenarios

An ordinary user with no elevated privileges or rights to access obsolete information saves a link to a file. The associated document is subsequently revised and re-issued. The link to the now-obsolete revision still works. The user is still applying the obsolete revision that the link points to. This results in a negative audit finding because a control system failure has occurred.

An ordinary user performs a document search and is presented with a number of results, one of which is marked as the current authoritative release. However they accidentally click on the wrong record and then proceed to apply an obsolete document.

Action

  • As an ordinary user without elevated privileges or rights to access obsolete information, perform a document-ID-equals search for document ID PROC-OPS_01/02.
  • If a link-saving facility exists then save a link to the document as an ordinary user with no elevated privileges.
  • Change the authoritative release of PROC-OPS_01/02 to Rev D.
  • Perform the search again for PROC-OPS_01/02.
  • Open the previously saved link.

Pass Criteria

For an ordinary user, the document identity lookup must always resolve to the controlled document's authoritative release. In the first instance of the search the result must return exactly one record: Rev C. The document ID must also be discoverable without first navigating to a folder, library, workspace, project or other location associated with the document. That is, the user must not require any knowledge other than the document ID.

In the second instance it must return exactly one record: Rev D.

When the previously saved but now-stale link is opened by an ordinary user without elevated privileges for viewing historical releases it must either fail or resolve to Rev D.

The system must prevent an ordinary user from inadvertently using obsolete documents. Elevated privileges, or some alternative pathway, must be required to access Rev C.

The system must have a document-ID-equals search function that searches document IDs. One box, one ID, one result.

The document ID search must not rely on text content within the file, file location, filename, or attributes stored within the binary file.

The search must not rely on the user first knowing any details of the document metadata, interpreting the structure of the document ID, or knowing where to look.

Test 8. Release Succession Atomicity

Principle

At any point in time, including during a transitional state, the authoritative controlled document result must not expose two releases as authoritative. In this instance ‘atomicity’ relates to observable behavior, not necessarily database transaction atomicity. This test asks: Can the system transition between authoritative releases without exposing an invalid intermediate state?

Use Cases and Failure Scenarios

A new revision is being made authoritative. During the transition users must not encounter a state in which both the old and the new revisions are presented as current.

Action

  • Create a Rev E of PROC-OPS_01/02.
  • Link a dummy blank file to Rev E.
  • Change the state of Rev E to the authoritative state.

Pass Criteria

There must never be an intermediate state where both Rev D and Rev E are authoritative, for example due to scheduled or asynchronous indexing, crawling, harvesting or synchronization process. The transition must be instantaneous, for all intents and purposes.

Test 9. Historical Immutability

Principle

Once a release has been issued, its controlled content must be immutable in normal operations.

Use Cases and Failure Scenarios

A user is applying the authoritative release of a document and later discovers that the content has changed but the release number is still the same. This undermines confidence in the controlled document management system and may result in a negative quality-system audit finding.

Action

  • Attempt to edit a document linked to a release which is superseded.
  • Attempt to edit a document linked to a release which is the authoritative release.
  • Attempt to link an additional file to a release which is superseded.
  • Attempt to link an additional file to a release which is the authoritative release.
  • Attempt to remove a file previously linked to a release which is superseded.
  • Attempt to remove a file previously linked to a release which is the authoritative release.

Pass Criteria

It must not be possible to alter the files or alter the set of files comprising a release which is either superseded or is the current authoritative release.

Test 10. Audit/Investigation Historical Reconstruction

Principle

It must be possible to assess what the authoritative state of a document was at any point in the past for audit or investigation purposes.

Use Cases and Failure Scenarios

An auditor or investigator needs to establish exactly which revision was in force when an event occurred and must be able to reconstruct that state reliably.

Action

  • Using appropriate access privileges, examine the release history and associated files and confirm that for any point in time in the past it is possible to know and retrieve exactly the content which was authoritative for a given document at a given point in time.

Pass Criteria

For any point in time at which the document was under controlled management, a set of files can be extracted which represents the exact authoritative release at that time.

Scoring

Note that poor scoring is not necessarily an indication of a poor product. It is evidence of an architectural mismatch for the use case of controlled document management.

Summary of Fundamental Invariants

The tests above are different ways of testing a smaller set of fundamental invariants. These are the conditions that must remain true regardless of how the system is used through its supported interfaces.

Fundamental property What must remain true
Document identity Every controlled document has its own organization-defined identity, independent of any file. The identity is unique, can be used to find the document directly, and can represent existing identifiers exactly as they appear on the documents.
Document releases A document's releases are distinct from the files used to represent them. A release can have multiple associated files and can retain the organization's existing release or revision identifier.
Controlled state A release represents a fixed, controlled state. Once issued, its files and contents cannot be changed. There can be no more than one authoritative release of a document at any time.
Safe document retrieval Searching for a document by its ID always leads an ordinary user to the authoritative release. Obsolete releases must not be inadvertently presented or retrieved as the current document, including through stale links.
Controlled transitions When one release replaces another, the system must not expose an intermediate state in which both releases appear authoritative. The change must be effectively immediate from the user's perspective.
Historical reconstruction The system must preserve enough information to establish which release was authoritative at any point in the past and to retrieve the exact files that constituted that release.
Migration without degradation Existing document identities, release identifiers, historical states, and file associations can be brought into the system without changing their established meaning or forcing them through workflows that do not reflect their existing controlled state.

These invariants describe the underlying behavior expected of a controlled document management system. They are not individual features or implementation requirements. A product may implement them in different ways, but the resulting behavior must be reliably maintained across all supported ways of using the system.

Silkwood Software

October 2026