Computer Software Assurance: Perfect Solution or Confidence Trick?

Computer Software Assurance (CSA) is a development from the Food and Drug Administration (FDA) Case for Quality initiative that was purported by industry proponents to replace Computerized System Validation (CSV). The aim of both approaches is to demonstrate that a system or software is fit for its intended use. This article takes an analytical approach to the various CSA claims to determine if CSA used in regulated pharmaceutical laboratories can be the perfect solution or is a confidence trick. The approaches outlined in this article are intended for GMP laboratories in the pharmaceutical and biotechnology industries. 

Introduction

Good Manufacturing Practice (GMP) regulated laboratories in the pharmaceutical and biotechnology industries contain many analytical computerized systems. Computerized System Validation (CSV) has been a regulatory expectation or requirement since the mid-1980s. However, the original FDA GMP regulations (21 CFR 211) had only two references to computers in clauses 211.68(b) and 211.180(c).¹ To help industry interpret the regulations, FDA issued Compliance Policy Guides (CPG) 7132a.11 defining hardware as equipment and application software as records,² CPG 7132a.08 to identify persons on computer records³ and CPG 7132a.07 for input/output checks of programs.⁴   

Since then, regulations for computerized systems have evolved: Annex 11 (1992, 2011 and in revision),^5,6 21 CFR 11⁷ and WHO TRS 1019 Annex 3, Appendix 5 (2019).⁸ Several industry CSV publications from scientific societies such as Drug Information Association (DIA), Parenteral Drug Association (PDA) and GAMP Forum,^9–12 as well as guidance from regulatory authorities such as FDA¹³ and PIC/S^14,15 have been issued to help industry interpret the regulations. In addition, data integrity guidance from regulators^15–19 and industry^20–23 impacting computerized systems have been published over the last 10 years.

Although the pharmaceutical industry works to the same regulations there are wide differences in company interpretations. Often the interpretation for CSV can be inflexible, onerous, slow and a non-added value activity that generates Great Mountains of Paper, thus giving GMP a new meaning. Risk management should be applied, but the industry is very conservative and compliance-driven, resulting in CSV becoming associated with producing documentation for audits and inspections. This leads us to the FDA Case for Quality initiative.

FDA Case for Quality

FDA Center for Devices and Radiological Health (CDRH) started the Case for Quality program in 2011 to focus on ways to produce high-quality medical devices. During the program, CSV was identified as a bottleneck being burdensome and expensive to perform e.g.: 

• No consideration or acknowledgement given when the software is purchased from a reputed vendor, industry software leaders or software providers with quality certifications.
• Duplication of effort at the client site was common practice and a significant issue.
• CSV was seen as a major deterrent to pursuing automation. 
• Owing to the perceived regulatory burden, the focus of CSV was on gathering documented evidence for auditors and not on testing and assuring that the software works as intended.

In 2015, FDA set up a joint Agency and industry team to investigate and evaluate alternative approaches. The outcome was Computer Software Assurance (CSA). 

FDA CSA guidance: Waiting for Godot?

FDA promised a draft guidance for industry in 2018, but nothing appeared. This vacuum permitted some of the industry team members to promote CSA with the message CSV is dead and you must transition to CSA. This resulted in “regulation” by presentation and publication, as there was nothing to cross-reference the claims against. GAMP forum has included a CSA in Appendix S2 of the Good Practice Guide (GPG) Data Integrity by Design,²² a section on critical thinking in the GPG Enabling Innovation²⁴ and CSA testing approaches in GAMP 5 Second Edition¹² before publication of the draft guidance.

The title of this section is deliberate as the Samuel Becket play Waiting for Godot consists mainly of two people on stage, talking rubbish for two hours and Godot never turns up. Oh wait!  The draft FDA guidance was finally published in September 2022, entitled Computer Software Assurance for Production and Quality System Software.²⁵

This article will focus on CSA for regulated GMP laboratories in the pharmaceutical industry.  For a detailed and critical review of the draft CSA guidance for medical devices, please read Ozan Okutan’s excellent review, available on LinkedIn.²⁶

CSA: The promised land

Prior to publication of the draft FDA guidance, the main principles of CSA were touted as a paradigm shift over CSV including:

• Focus on intended use: Define the intended use of the software.
• Use critical thinking: Instead of writing voluminous test documentation use critical thinking to focus testing.
• Allow unscripted and ad-hoc testing: Test Scripts are too detailed with click level instructions, often run into “hundreds of pages”, cost a lot of time & effort – and result in test script deviations due to their prescriptive nature. The typical CSV initiative splits with 80% of deviations due to tester or test script errors, with 20% actually detecting and resolving functional errors. However, there did not appear to be any supporting evidence to support this claim.  Possible reasons could be testing custom rather than commercial software or implementation of a new application where the testers were not trained or familiar with its operation.
• Use trusted suppliers: No consideration/acknowledgement is given when the software is purchased from a reputed vendor, industry software leaders, or software providers with quality certifications. Duplication of effort at the client site was common practice and a significant issue.
• Evidence should add value: Most evidence consists of screenshots or paper printouts of the testing process. This slows the testing process.
• Use automation: To help software development and testing, use automated testing tools.
  
  

General Principles of Software Validation: Review baseline

Before we begin the review of the draft CSA guidance, we need to consider the FDA guidance General Principles of Software Validation (GPSV)¹³ issued by CDRH and Center for Biologics in 2002. This forms the baseline for our discussion about CSA and is at the center of the claim that CSV is over-burdensome. For the pharmaceutical industry, there is GAMP 5 first and second editions^11,12 along with the Good Practice Guides for Validation of Laboratory Computerised Systems second edition and Risk-based Testing of Systems second edition.^27,28 

The structure of GPSV is shown in Figure 1. In the table of contents, the title of section 2.3, printed in bold upper-case letters, THE LEAST BURDENSOME APPROACH. The section simply states that this should apply to all areas of medical device regulation. Put simply: do as much validation as you need to but no more, as reiterated in Section 4.8. Most companies have ignored this advice.

Figure 1: Structure of the FDA guidance General Principles of Software Validation.¹³ Credit: Bob McDowall.

Figure 1 shows sections 3, 4 and 5 of the GPSV in green, which are the main part of the guidance dealing with validation of software. A key point is in Section 4.8: Validation coverage should be based on the software's complexity and safety risk – not on firm size or resource constraints.¹³  

A summary of the key parts of these sections is listed below.

• Section 3 Context for Software Validation: Requirements and Specifications, Verification and Validation, Software Development as Part of System Design, Benefits of Software Validation and Design Review.
• Section 4 Principles of Software Validation: Requirements, Defect Prevention,  Software Life Cycle, Plans, Procedures, Software Validation After a Change, Validation Coverage, Independence of Review and Flexibility and Responsibility.
• Section 5 Activities and Tasks: Software Life Cycle Activities, Tasks supporting validation: Quality Planning, Requirements, Design, Construction or Coding, Testing by the Software Developer, User Site Testing, Maintenance and Software Changes.

You will notice that requirements features in all three of these sections, which begs the question of why defining intended use is claimed as a novel CSA task? Indeed, GPSV Section 5.2.2 states that without requirements you can’t validate software.¹³

The guidance is intended mainly for medical devices and software used in their production and the quality system. Therefore, some interpretation is required to apply the principles to laboratory systems as a supplier will be involved: the validation tasks will be split between the two. This is where Section 6 is useful.

Figure 1 shows Section 6 in red and focuses on validation of automatic process equipment and Quality Management System software. It also offers some good guidance for pharmaceutical industry CSV.  

Validating commercially available software

Section 6.1 provides an example of validation and upgrade of commercially available software:

• … who chooses not to use all the vendor-supplied capabilities of the software only needs to validate those functions that will be used and for which the device manufacturer is dependent upon the software results as part of production or the quality system¹³ 
• When software is upgraded or any changes are made to the software, the device manufacturer should consider how those changes may impact the “used portions” of the software and must reconfirm the validation of those portions of the software that are used¹³

Summarising: This is simple to understand. Take a risk-based approach to the automated operation and software complexity. Define the level of user dependency to the automated process and establish user requirements. Understand and document your requirements for the software functions used, decide on the controls required and test the configured application against them. When the application is upgraded, show that system continues to operate as before.

Specifying intended use requirements

With the focus on Off the Shelf software, Section 6.2 emphasizes the importance of a user specification for defining the intended use of an automated system or software. Reiterating, it is reinforced by Section 5.2.2 that states It is not possible to validate software without predetermined and documented software requirements.¹³ 

Additionally, preamble comment 136 to 21 CFR 820, the Quality System Regulation²⁹ showed that FDA incorporated feedback to change 21 CFR 820.70(i) so that software used in production and automated systems must be validated for its intended use. It’s in the regulation!

FDA made a further point in 2018 in the data integrity guidance question 4: if you validate the computer system but do not validate it for its intended use you cannot know if your workflow runs correctly.¹⁹

These existing references to software’s intended use beg the question of why the FDA-Industry task force did not read the QSR regulations and GPSV?

Leveraging supplier development and testing

Further validation advice about leveraging supplier material is provided in Section 6.3¹³:

• If the vendor can provide information about their system requirements, software requirements, validation process, and the results of their validation, the medical device manufacturer can use that information as a beginning point for their required validation documentation.

• The vendor’s life cycle documentation, such as testing protocols and results, source code, design specification, and requirements specification, can be useful in establishing that the software has been validated.

• The audit should demonstrate that the vendor’s procedures for and results of the verification and validation activities performed the OTS software are appropriate and sufficient for the safety and effectiveness requirements of the medical device to be produced using that software.¹³

This information completely refutes the CSA claim that vendor development and testing work is ignored and work is replicated. Leverage of supplier’s work has been in GPSV since 2002.

There is a caveat to add from the Medicines and Healthcare products Regulatory Agency (MHRA) in Section 6.19 of the GXP data integrity guidance:

The acceptance of vendor-supplied validation data in isolation of system configuration and user’s intended use is not acceptable.¹⁸

Therefore, you can leverage the supplier’s software development and testing into your validation effort. BUT as a supplier does not know how you will use the software, it is only part of the overall effort that is reinforced by the MHRA statement.¹⁸ The end user is responsible for the overall risk management and validation of any software used in a regulated environment. This is consistent with any GXP regulation in the pharmaceutical industry.

GPSV summary

Section 6 provides helpful advice about defining the intended use of the system, validating commercial software and leveraging the work of the supplier. As Section 6 is useful, it is a mystery why CSA emphasizes the importance of defining the intended use of software, especially when it is already present in the applicable Quality System Regulation and the GPSV?

In summary, GPSV contains the following advice for QMS and automation software¹³:

• You must document the risk analysis and intended use of the system: both software and analytical instrument.
• You only need to validate the portions of a commercial application you use.
• If upgraded, ensure that the portions of the application you use function as previously and remain in control.
• You can leverage the supplier’s development and testing as part of your validation.
  
  

Simple! But have you read and implemented this? Probably not!

Publication of the draft CSA Guidance 2022

Finally, in September 2022 Godot arrived! The draft CSA Guidance was published for public comment.²⁵ The structure of the CSA draft guidance is illustrated in Figure 2. Reading it one is struck by the large gaps between what the proponents of CSA said and what is in the draft guidance. 

Figure 2: Structure of the draft Computer Software Assurance Guide.²⁵ Credit: Bob McDowall.

CSA guidance only replaces GPSV Section 6

 Firstly, and most importantly, CSV is not dead! The CSA guidance will supplement the GPSV and replace 5 pages of Section 6.²⁵ Therefore, CSA is a subset of CSV and must not be used for validation of medical device software. CSA and CSV exist in parallel. Since they exist in parallel, the least you can expect is that they do not contradict each other. This is not the case.

The problem is compounded by the fact that there is very good advice contained in GPSV Section 6,¹³ discussed earlier, which might be lost if totally replaced by the final version of the CSA guidance.

What has not been stated by the FDA is how GPSV Section 6 would be replaced. It would be preferable to update and reissue the complete GPSV incorporating what remains of CSA into a single document. Alternatively, replace it with two guidance documents (Medical Device Software and Software for QMS and Production).²⁶ (O. Okutan personal communication).

CSV and CSA definitions: Software or system?


The definitions of software validation and computer software assurance are shown in Table 1.  The first problem is that both refer to software and not system. The definition of Computerized System includes the controlled function consisting of instruments controlled by the software and trained users as illustrated in Figure 1 of PIC/S PI-011,¹⁴ which appear to have been forgotten by the authors of the two FDA guidances. You can’t operate a computerized laboratory system without trained users with procedures and controlling an analytical instrument.

Table 1: FDA definitions of software validation and computer software assurance.

Comparing the two definitions:

• Both mention intended use
• Only CSA mentions it is a risk-based approach, although risk assessment is embedded throughout GPSV
• GPSV looks for consistency of operation
• GPSV requires objective evidence
• CSA only requires confidence (which is not defined in the guidance).

Whilst there are differences, the aims of the two definitions appear similar. However, the concern is that the confidence of CSA is a weakening of objective evidence required by CSV.  However, objective evidence does not explicitly or implicitly require mountains of paper and screenshots; for laboratory systems, it will include electronic records, including contextual metadata. GPSV Section 6.1 states that the evidence required is proportional to the risk posed by the software and process automated.¹³ This is reinforced by section 5.2.6: User site testing should follow a pre-defined written plan with a formal summary of testing and a record of formal acceptance. Documented evidence of all testing procedures, test input data, and test results should be retained.¹³

Table 2: Regulatory requirements for defining intended use of a computerized system 1978–2011.

Computer software assurance risk framework

The CSA risk framework consists of four stages in lines 137 to 602 and constitutes the bulk of the draft guidance,²⁵ the subsection numbering below reference those in the guidance:

A. Identifying the Intended Use: Why was defining the intended use even considered a novel approach for CSA? 

a. As shown in Table 2, defining the intended use of equipment and computerized systems has been in the pharmaceutical regulations since 1978 in 21 CFR 211.63,¹ since 2011 in clause 4.4 of EU Annex 11⁵ and in Section 6.2 of GPSV.¹³  

There is nothing novel as it is expected deliverable for validation. However, lines 182–183 and footnote 5 in the draft CSA guidance use the terms feature, function and operations but fail to explain exactly what these mean.²⁵ See chapter 4.3.1 in Okutan’s critique which attempts to explain these three terms.²⁶

b. However, both the GPSV and CSA guidance do not mention use of process knowledge, process improvement/redesign or a data integrity risk assessment required by MHRA.¹⁸ Process redesign will be discussed later.

c. There is mention of direct and support software. The former must be validated under 21 CFR 820.70(i)²⁹ but supporting software is lower risk and can use the CSA approach. Here is the contradiction to the QSR and GPSV which validates all software.^{13, 29}

d. FDA recommends that all decisions on the approach taken are documented.²⁵ Annex 11 clause 1 is phrased better: As part of a risk management system, decisions on the extent of validation and data integrity controls should be based on a justified and documented risk assessment of the computerised system.⁵

B. Determining the Risk-Based Approach: For lower-risk software FDA states that the risks are different from those associated with medical device software and the risk assessment approach in ISO 14971: 2019³⁰ is not appropriate.   

a. The draft CSA guidance suggests users should consider those factors that may impact or prevent the software from performing as intended, such as proper system configuration and management, security of the system, data storage, data transfer, or operation error.²⁵ ICH Q9(R1) on quality risk management offers alternative risk assessment methodologies³¹ to FMEA.

b. The guidance divides process risk into a binary high and not high with examples and then notes that process risk is on a spectrum (!?). The intended use requirements are an input to assess the risk.²⁷ This requires the URS  contains sufficient detail, as laboratory systems usually don’t need a Functional Specification but a Configuration Specification. However, risk management throughout the system life cycle has been a requirement in Annex 11 clause 1 since 2011.⁵

C. Determining the Appropriate Assurance Activities: The focus here is on testing the software and consists of:

a. Unscripted testing: Based on ISO 29119 for software testing³² there is ad-hoc, error-guessing and exploratory testing, which do not require large amounts of documentation. Note, that unscripted testing does not mean undocumented testing, a list of record requirements is listed in section D Establishing the Appropriate Record.²⁵ Exploratory testing will be discussed later. 
b. Scripted testing: An FDA term which is divided into robust scripted testing and limited scripted testing. Robust scripted testing includes evidence of repeatability, traceability to requirements, and auditability.²⁵ In a pharmaceutical context, this is the only option if a laboratory must comply with requirements traceability in Annex 11 clause 4.4⁵ as shown in Table 2.  However, there are ways to simplify test execution instructions and still comply with traceability and robust scripted testing as discussed later.   

D.  Establishing the Appropriate Record: in an indirect reference to the Least Burdensome Approach in the GPSV, CDS mentions Documentation of assurance activities need not include more evidence than necessary to show that the software feature, function, or operation performs as intended for the risk identified.²⁵  

a. There is a table comparing the different evidence required for each of the assurance activities as well as a worked example of exploratory testing. The problem with the latter is that a spreadsheet is tested which is not the best example to use in a GMP-regulated environment.²⁵ A SaaS application with quarterly upgrades would be a more interesting option to present (Eva Kelly, personal communication).

b. Automation may be used for CSA work. Annex 11 clause 4.7 has allowed the use of automated test tools since 2011: Automated testing tools … should have documented assessments for their adequacy.⁵ Note that an assessment is not validation.

c. The CSA guidance states: As a least burdensome method, FDA recommends the use of electronic records, such as system logs, audit trails, and other data generated by the software, as opposed to paper documentation and screenshots, in establishing the record associated with the assurance activities.²⁵

Again, this is déjà vu all over again as GAMP 5 first edition section 8.5.3 states: Systems may have audit trails that capture much of the information that is traditionally captured by screen prints. If such an audit trail exists, is secure, and is available for review by the SME, then capturing additional evidence may not be necessary.¹¹ Moreover, in GAMP 5 Second Edition section 25.5.3 it states In many cases, prolific screen shots do not add value and are unnecessary.¹² The audit trail still needs to be validated to demonstrate the function works.³³

CSA appendices

• Non-conformance management system
• Learning Management System (LMS)
• Business Intelligence Applications
  
  

The CSA approach for each one is presented and summarised in tables. However, the examples are very simplistic and do not appear related to the process automated by the software, making it difficult to demonstrate the intended use of the software. The attempt and explaining feature and function is confusing, especially as the terms are not defined.

What CSA principles are missing?

From the list of principles listed at the start of this article, there are two main items missing from the draft CSA guidance: critical thinking and use of trusted suppliers. We will discuss each and see where we can seek additional help to fill the void.

Critical thinking? Dead on arrival!

One of the promised components of CSA was critical thinking. This is defined by the GAMP Guide on Records and Data Integrity²⁰ as:

A systematic, rational and disciplined process of evaluating information from a variety of perspectives to yield a balanced and well-reasoned answer.

However, critical thinking is conspicuous by its absence in the CSA guidance. Nothing! Nada! Zip!! Perhaps the FDA lawyers were worried by how to legislate the topic? However, in the FDA GMP regulations for laboratory controls, critical thinking can be found under another name and hiding in plain sight since 1978. In 21 CFR 211.160(b) anything that is done in a laboratory must be scientifically sound:

Laboratory controls shall include the establishment of scientifically sound and appropriate specifications, standards, sampling plans, and test procedures ….¹

Scientific soundness is also found in EU GMP Part 2 or ICH Q7(R1) Good Manufacturing Practice for Active Pharmaceutical Ingredients clauses 8.34, 11.12 and 19.80.³⁴

Does scientific soundness equate to critical thinking? Yes. The former is based on scientific principles, formulation of hypotheses, unbiased and objective evaluation of evidence which requires accurate and reliable data to provide confidence in the outcome. Critical thinking by another name and an existing regulation for the laboratory.

Trusted suppliers are missing

There is no mention of trusted suppliers in the draft CSA guidance. However, as shown earlier, GPSV Section 6.3 is adequate for pointing the way forward. 

However, this article must discuss and refute that supplier certification is adequate for software validation:

• GPSV Section 6.3 discusses ways to leverage a supplier’s work but never mentions quality certification.¹³ The problem with ISO 9001 certification is that there is no explicit requirement for quality in the standard and suppliers may prioritize certification over delivery of a quality product or service. The assumption is that a documented process equates to quality, but this is a fallacy.
• Preamble comment 65 from the Federal Register in 1997 for 21 CFR 11 regarding validation in 21 CFR 11.10(a) is much more explicit:

o The agency disagrees with the comment’s claim that all commercial software has been validated.

o The agency believes that commercial availability is no guarantee that software has undergone ‘‘thorough validation’’ and is unaware of any regulatory entity that has jurisdiction over general purpose software producers.

o The agency notes that, in general, commercial software packages are accompanied not by statements of suitability or compliance with established standards, but rather by disclaimers as to their fitness for use. … ⁷

If in doubt of the last item: compare a supplier’s marketing literature with the End User Licence Agreement (EULA) and see for yourself the difference. It is preamble comment 65 that prevents FDA from incorporating any statements on using trusted suppliers in the CSA guidance.

However, in 2012 GAMP published the GPG on A Risk-Based Approach to Testing GXP Systems²⁸ and Section 2.5 discusses leveraging supplier involvement:

Regulated companies should seek to maximise supplier involvement throughout the system life cycle in order to leverage knowledge, experience and documentation, subject to satisfactory supplier assessment.²⁸

Available since 2012 but ignored by the FDA industry team. We will return to leveraging supplier software development and testing later in this article.

Additional comments on the CSA draft guidance

There are several issues that are missing or can be subject to a different interpretation in the draft CSA guidance. These can be used to implement a computerized system. These are:

• Defining the intended use of the software
• Leveraging supplier development and testing
• Use exploratory testing for prototyping
• Writing efficient test instructions
• No glossary
• Spreadsheet example – can FDA be serious?

Each topic will be discussed in this section.

Problems with only defining intended use

In both the GPSV and the CSA guidances, there is no mention of the business process improvement or redesign. In simple terms: there is no point wasting money on a computerized system to automate the status quo: where is the business benefit of a more efficient electronic process? ICH Q10 requires process understanding for continual improvement and knowledge management³⁵: this will not be possible if key data are on paper locked in an archive.

Item 3 of the proposed update of EU GMP Annex 11⁶ mentions updating controls to include digital transformation. Working electronically is the best option for laboratories as this will streamline and eliminate paper records and transcription error checking. Process mapping and redesign is an essential prerequisite prior to digital transformation. 

The lack of process understanding and how the application software automates the business process is shown in Example 1 for the Nonconformance Management System.²⁵ The electronic signature function is tested and the execution record is stated to be in the audit trail.  However, the electronic signature must also be on the signed record for compliance with 21 CFR 11.50(a).⁷ Furthermore, there is no detail about creating the record to sign nor of the review process by a second person. As a result, there is no or little consideration of a controlled or electronic business process that sits underneath the software to use e-signatures.

To obtain business benefit, map the current process with inputs and outputs, identify data created and transformed, document the different systems (including spreadsheets) and the transfer between them and listen to the analysts for improvement ideas. Then redesign the process to eliminate the bottlenecks and work electronically. Then, and only then, integrate the application into the process and refine the process using the configuration settings of the application.

The redesigned process maps provide a good basis for writing and updating user requirements and system configuration settings. They are also a good aid for determining and evaluating the impact of changes to software.

Once a URS is complete, it should be sent to selected suppliers for comment to determine if a product can meet each laboratory requirement. Key to this is to ask each supplier if each requirement can be classified to determine it can be met as follows:

• S = Standard functionality of the product
• Config = Configured to meet a requirement
• NA = Not available
• Custom: Requires customization

The supplier should respond formally and this become part of the agreement between the laboratory and the supplier. The statements from a supplier must be verified before purchase.  These areas are not discussed in the CSA guidance.

Leveraging supplier development and testing

As discussed earlier, GPSV Section 6.3 mentions the use of supplier work and material for reducing the overall validation effort for commercially available software.¹³ However, there is little information on how to take advantage of this, as one of the complaints against CSV is that validation replicated supplier testing. GAMP GPG on Testing goes further:

Testing is an area of the system life cycle in which there are potentially large efficiency gains to be made by the appropriate leveraging of supplier involvement; in particular through the:

• Avoidance of duplication in testing

• Leveraging supplier test activities and evidence to the maximum practical extent while still ensuring fitness for intended use.²⁸

To avoid testing duplication, review the supplier’s qualification protocols before execution. For a chromatography data system, if the Operational Qualification (OQ) tests peak integration and processing of standard chromatograms result in areas or peaks within the acceptance criteria – why would you want to test the same function yourself?

To leverage a supplier’s work into your validation effort requires an assessment of their software development and testing so that you can have confidence in both the supplier and the application. Section 3.5 of the GAMP GPG on Testing discusses Leveraging Supplier Testing.²⁸

A traditional way of qualifying a supplier is to send a questionnaire for a supplier to fill it in.  There could be follow-up questions to verify what the supplier has written. However, this approach is inadequate to leverage the supplier’s development and testing into your validation effort for GAMP software category 4 systems.

What is needed is an audit, either on-site or remote, of a supplier’s software development, testing and support processes. This can be achieved in a day plus the time to plan and write the report. The effort to do this will be repaid several times over by streamlining the User Acceptance Testing (UAT) or Performance Qualification (PQ) phase of the project. The audit will assess the items such as:

• The tools and methodology used to manage the process, e.g., Agile development, code management, automated testing, etc.
• How requirements or user stories are written and interpreted into more detail with the Definition of Done
• Coding by a developer with informal testing
• Coding standards with peer review of modules of code
• Formal module and integration testing
• Handling, classifying and resolving software errors
• Specification and testing of configurable elements of GAMP software category 4
• Criteria for release
• Support for the product and notification of critical software errors.

Assessment of the supplier QMS can be performed off-line, ideally before the software development audit, and some items (e.g., training records, document management) can be assessed briefly.

Figure 3. Division of software functions into GAMP software Categories 3 and 4 for a chromatography data system.³³ Credit: Bob McDowall.

A laboratory software application may be classified as Category 4 (configurable to match  the business process), but many functions within it are Category 3 (non-configurable but are parameterized). The simplest way to understand the difference is to consider two functions within a laboratory system:

• Configurable function: The application has the option to use electronic signatures for a performer and a reviewer of a test. This requires the configuration settings to be changed to enable the use of electronic signatures. As a result, the business process changes from a hybrid system to an electronic process eliminating paper.
• Parameterized function: Many instrument control functions are parameterized.  The flow rate of an HPLC pump can be set to between 0.1 and 5 mL/min depending on the method used. The business process – pumping mobile phase – is unchanged, only the volume pumped is modified or parameterized.

This is shown for a chromatography data system in Figure 3.³³  

• Here the overall application is Category 4.
• Instrument control, acquisition, processing functions and system audit trail are Category 3 functions shown in green. According to Table 12.1 of GAMP 5 second edition:
  Run-time parameters may be entered and stored, but the software cannot be configured to suit the business process.¹² These functions should be accepted and tested from an adequate supplier software audit, BUT they will be implicitly tested in the UAT. The testing will be based on the process(es) automated by the system and defined in the URS, but the testing focus should be against the configurable functions.
• Category 4 functions are shown in yellow and are system application policies, e-signatures and custom calculations etc. This is where the main UAT / PQ testing should be focused.
• Calculations must be verified or critically examined according to 21 CFR 211.68(b) and Chapter 6.16 respectively.  Standard calculations can be accepted as validated providing there is sufficient evidence of supplier verification, otherwise they must be verified in the user acceptance testing.

Therefore, in your URS identify all those requirements that are Category 3 functions. IF the supplier software development is adequate and acceptable, then you can accept all Category 3 requirements as tested. In the PQ, only test them implicitly in process flows that focus testing on the configurable Category 4 functions. Always remember: The acceptance of vendor-supplied validation data in isolation of system configuration and user's intended use is not acceptable.¹⁸ Moreover, this approach is applied only to laboratory Category 4 software. Be careful of suppliers that provide a language to configure when you are writing custom software (GAMP Category 5).

Consider exploratory testing as prototyping

The CSA guidance discusses exploratory testing as Experience-based testing in which the tester spontaneously designs and executes tests based on the tester’s existing relevant knowledge … ²⁵ This can be adapted and used to the advantage of a laboratory when evaluating what configuration settings and workflows to define before finalizing the User Requirements and Configuration Specifications, e.g., application settings, user roles with access privileges and electronic workflows. This is shown in Figure 4.

Figure 4: Using prototyping to refine user requirements and application settings. Credit: Bob McDowall.

The validation plan should state that prototyping will be a project task which can be performed either on an unqualified or qualified software instance. The inputs to the task are the draft URS and configuration specification. Configuration settings will be evaluated and then changed and reevaluated to find the optimum settings that generate business efficiencies, remove paper records, and ensure data integrity and regulatory compliance. I suggest that this is informally documented with notes taken of the options evaluated and the advantages and disadvantages of each. The end of the prototyping task will be the final configuration of the application that will be systematically evaluated in the user acceptance testing and used in operational use. Outputs from the prototyping are formal versions of the URS and configuration specifications.

It is important that trained users perform prototyping. Gathering feedback from the user population on the different configuration settings and workflows is critical. Learn and implement the changes for the next prototyping cycle. Listening to the users of the system is imperative otherwise the system can be rejected as users will find alternative ways of working.

Writing effective test instructions

To comply with EU GMP Annex 11 clause 4.4, shown in Table 2, for traceability of requirements throughout the life cycle, you must use robust scripted testing according to the draft CSA guidance.²⁵ Can we reduce the volume of testing documentation within the potential straitjacket of robust scripted testing?

Yes, by working smarter, not harder. Writing the test scripts for the majority of validation projects takes the most time. However, this can be reduced in several ways, especially if the laboratory does not have access to an automated test tool.

• Simplify test execution instructions for execution by trained users, not idiots or novices.  Compare the two sets of instructions; which is simpler to execute?
• Eliminate screenshots unless there is no way that the software can capture the activity in an electronic log or audit trail.
• Avoid printing paper wherever possible. If printing is required select a PDF option so that it can be searched easily and stored securely in a document management system.

Table 3: Writing test execution instructions for idiots and trained users

For the idiot test script, you can add the following non-added value activities:

• Take a screenshot for each test step
• Initial and date each line of the test script

For the trained user test script:

• The key check of streamlined test instructions is that they are reproducible: an independent tester can execute the instructions correctly as the writer of the script intended.
• There is no need for a screenshot as there is no record created
• The test step has no need for an expected result as it is simply an instruction
• The tester and reviewer should initial and date all completed test instructions once at the bottom of the page (e.g. the same as a laboratory notebook).

No glossary: Waiting for Godot – again!

There is no CSA glossary. Key terms are not defined: what is COTS software? The guidance says it means Commercial Off the Shelf software. But can a COTS application be configured to meet the business process? We don’t know. It would be much better to use GAMP software categories where there are clear definitions and discussions rather than a single abbreviation with no explanation. Without a glossary, the CSA terms feature, function and operations are presented but not defined.

FDA has history here, when the GPSV was issued in 2002 the Federal Register entry noted under Terminology:

Several comments noted inconsistencies in terminology …  At their next revision, we expect to update other software guidance documents and the FDA glossary with consistent definitions of validation, verification, and risk analysis.³⁶

A technical glossary is imperative when a new topic is introduced so industry can understand it in more detail when it is required. A major failing. Underneath in the same section is the following:

We have added a section that refers to the various types of qualification, but we have chosen not to adopt ‘‘qualification’’ terminology in explaining software validation requirements.³⁶

The differences between qualification and validation continue to be a problem today as regulators have chosen to separate the two, e.g., Annex 11 for computerized systems and Annex 15 for equipment.^5,37 Similarly, World Health Organisation (WHO) Technical Report Series (TRS) 1019 Annex 3 has two separate appendices.^8,38 In contrast, USP <1058> on Analytical Instrument Qualification (AIQ) takes an integrated approach with Group C systems.³⁹

A spreadsheet example

Is the FDA serious? As an auditor, spreadsheets are the gift that keeps on giving. Yet there is a spreadsheet used for monitoring CAPA in lines 529–582. I have published two articles available for why you should NOT use spreadsheets in a regulated laboratory environment for compliance and business efficiency reasons.^40,41

Quo Vadis CSA?

There is little in the draft CSA guidance that is novel or earth-shattering. Existing risk-based and least burdensome approaches that have been ignored cover the same topics as CSA. However,  people have not read, understood or implemented them.

What CSA has done is opened a Pandora’s box of hyperbole and misunderstanding, which initiated a debate about CSV, especially in ultra-conservative pharmaceutical companies and their Quality Assurance departments. This is coupled with ultra-conservative inspectors and auditors who require screenshots and unnecessary QA approvals. However, as there is nothing new, will CSA follow the dodo to extinction? 

If not, what the CSA final  guidance should include are:

• Process improvement/digitalization which leads to documentation of intended use of software/system
• Leveraging a supplier’s software development and testing compatible with meeting compliance requirements
• For systems under EU GMP Annex 11, test execution instructions can be simplified and streamlined whilst ensuring requirements traceability⁵
• Screen shots can be mostly eliminated using an adequate audit trail to self-document testing
• Many paper printouts can be eliminated if there are electronic records available to record testing
• Automation can be used for requirements management, traceability and automated testing. However, for standalone systems testing using two keyboards will be akin to watching a Rick Wakeman concert without the harmonies. 

What must be remembered is that CSA is a subset of CSV: it only replaces Section 6 of the GPSV.

Above all, flexibility is essential as one size CSV does NOT fit all. Fit the CSV approach to the system and the risk to patient, process and data – not the other way round.

All companies must learn from their CSV experience and not stand still: you have to adapt and update your approaches all the time. It’s the c in cGMP!

Is CSA a perfect solution or a confidence trick?

Let us return to the title of this article: is CSA a perfect solution or a confidence trick? 

Many of the claimed CSA approaches were already in place in regulations and guidances, but apparently, no one has bothered to read or implement them.

Let us take stock:

• Defining intended use of software: this has been a GMP regulatory requirement since 1978.¹ This is a requirement in Annex 11⁵ and the General Principles of Software Validation (2002). What is the fuss all about?
• Critical thinking: Dead on arrival and missing from the CSA guidance. However, scientifically sound has been a GMP regulatory requirement since 1978.¹
• Trusted suppliers: Missing but is discussed in Section 6.3 of the GPSV¹³ and preamble comment 65 of 21 CFR 11.⁷ However, leveraging the supplier’s work is discussed in GAMP 5 second edition.¹²
• Test documentation: Requirements traceability requires robust scripted testing to meet Annex 11 requirements in clause 4.4.⁵
• But, write test instructions for trained users not idiots to streamline the testing process and use electronic means where possible to record testing.
• Appropriate evidence: Don’t use screenshots for test evidence was in GAMP 5 first edition¹¹ and GAMP 5 second edition.¹² If test data are electronic they should remain electronic; if data are printed use PDF output so that it can be searched easily.
• Automation: Using automation to help software development and testing has been in  Annex 11 clause 4.7⁵ since 2011.
• Risk management: Although not in the list at the front of this article it is a basic requirement in clause 1 of Annex 11.⁵
  
  

Therefore, CSA can be considered a confidence trick. 

Acknowledgements

I would like to thank Ozan Okutan, Yves Samson, Mahboubeh Lotfinia and Eva Kelly for their constructive comments and detailed review during the preparation of this article.