The Museum of Analytical Antiquities
Discover how to keep your analytical systems current and compliant and why this is so important.
Complete the form below to unlock access to ALL audio articles.
Most labs have them. Working analytical systems purchased back in the mists of time but have obsolete operating systems and/or instrument applications that have not been updated since initial qualification and validation. Welcome to your Museum of Analytical Antiquities. We explore the reasons for this and suggest ways to keep your analytical systems current and compliant.
Introduction
In this article we discuss why it is important to keep current with software updates; it is complementary to last year’s article on how to keep your LIMS current.1 Here, the focus is on analytical systems consisting of an instrument controlled by an application installed on a network or workstation for acquiring, processing, reporting results and storing data. Typically (depending on intended use), these are classified as United States Pharmacopoeia (USP) <1058> Group C systems.2
We will look at the Museum of Analytical Antiquities from two perspectives. The first is from the business rationale and why software should be kept up to date and second from the regulatory expectations of keeping current and meeting regulatory requirements. These are your front-line analytical systems, and not just those retained to read legacy data – although, that is another discussion altogether.
Why do labs create a Museum?
In GMP-regulated laboratories, there is the requirement to validate software to demonstrate that it is fit for its intended use.3–7 This is perceived, incorrectly, as a major bottleneck. So once validated, the company mantra is DON’T TOUCH IT. The rationale is that revalidation is perceived as a major cost. This is not the case as we will demonstrate in this article.
This results in laboratories failing to upgrade their application software and operating systems as well as failing to fix known cybersecurity flaws which could be exploited if, for example, uncontrolled USB devices are used carelessly.8
Unlike a LIMS, which is a network solution, most computerized analytical systems are designed for standalone use and hence are not connected to the IT infrastructure. Partly this is due to IT not wanting to manage laboratory systems or laboratories not wanting to be controlled by IT. Ideally, these systems should be connected to the IT network at least for backup and recovery, time synchronization and user account management, etc.
However, the usual situation is that most sit in majestic isolation in your laboratory. This provides some degree of protection from external cybersecurity threats but introduces other competing risks, like individual data backup/restore procedures (especially if USB devices are used), as well as business continuity, disaster recovery and data governance.
Figure 1. Areas
for failing to update an analytical computerized system. Credit: R D
McDowall.
Instrument issues and obsolescence
Hardware issues are ways of providing candidates for the Museum as shown in Figure 1. If an analytical instrument is obsoleted by a supplier, the availability of replacement parts and consumables may prevent the use of the instrument. If instrument firmware is either not supported or updated, then software drivers may not work. In addition, if an old computer system fails, is the new replacement workstation able to run an outdated operating system or a new instrument not supported by the old application?
The focus of this article is on software updates and the consequences of not being current.
Software = change
Unlike an analytical instrument with moving parts software does not wear out. However, software is not static but changes over time for a variety of reasons until it reaches its end of life, as shown in Figure 1:
- Application enhancements: Adding new features or functions for useability and better performance driven mainly by market forces.
- Compliance enhancements: Incorporation of new or updated GXP regulatory compliance, data integrity or pharmacopoeial requirements.
- Error fixes: Fixing software errors in the application software through a specific hotfix or minor release and keeping compatibility with technology.
- Security fixes: Remediating cybersecurity vulnerabilities of an operating system, software component or application.
- Outdated application or software: Obsolescence or discontinuation of the application software with no supplier support. Obsolescence or discontinuation of an operating system or middleware components with no supplier support or a database has reached the maximum size the older versions can support. Instrument firmware can also be a reason for obsolescence, as failure to update firmware can prevent interfacing to the application software. Software drivers for the instrument may not be compatible with a new application.
When faced with one or more of these situations, what should a laboratory do? Please note, that the ostrich option is not one of the choices here.
Update of software
As noted above updates of the software can occur for one of the reasons given in the section above. Let us take them in turn:
- Application enhancements: Depending on the business process being automated by the system as well as the system’s intended use, some of these features may or may not be of interest in a release. It is important to read the release notes to see what is new in each version. If an upgrade has features that a laboratory wants to use, often it is not installed as the validation is perceived as too difficult or the perception that the whole system must be revalidated. This will be discussed later in this article.
- Compliance enhancements: For example, when a Pharmacopeia mandatory general chapter changes, the requirements will be incorporated in the latest version of the software. Therefore, the new version is required to comply with the Pharmacopeial procedure. If you don’t upgrade, how will you comply? A spreadsheet, like the ostrich option, is not an effective solution.
- Error fixes: If a laboratory is impacted by a software error, it is advisable to install the hotfix to resolve the problem. There may also be application updates to keep compatible with changes in other system components.
- Security fixes: Vulnerabilities in operating systems, databases and middleware are identified and patches are released to prevent exploitation by malware. The advice is to keep current to prevent loss of data.
- Outdated application of software: Application support isn’t just a tech support phone call to help solve a problem or resolution only on a known bug-only support. It may also include the lack of a piece of paper that says the vendor supports your system and the inability of a vendor to provide hardware qualification on a regular basis if the firmware/software set is not compatible with their qualification tool. This can be especially risky after instrument Preventative Maintenance (PM) is performed.
An additional factor not listed above is the contractual agreement with an application supplier.3,9 The quality of the software support you get, from training to helping you answer questions during a regulatory inspection, can depend on your risk management process, the supplier knowledge, their QMS and the infrastructure of the company.
Here, a laboratory pays for ongoing software support and as part of the agreement the application must be kept on current or the previous major release, otherwise support lapses or the company must upgrade to the latest version before support is provided.
Obsolescence of system components
All components, including the instrument and the computerized system, can be impacted by this. The instrument supplier may decide that the instrument and or the application has reached the end of life and is declared obsolete. Ideally, the supplier will manage this process well ahead of time through an EGS (End of Guaranteed Support) framework.
There may be limited software support offered by a supplier, where remediation of known bugs is the only option (for up to two years) to give users the opportunity to move to a new system. In some instances, the supplier may be able to provide extended professional services such as computerized system validation (CSV) to speed up the implementation of updates or upgrades.
More important is that Windows 10 operating system becomes obsolete in October 2025, after which there will not be any updates unless you pay for them. This option will last only for three years.
What are you going to do? Again, the ostrich option is not very helpful. The scale of recent global cybersecurity attacks has highlighted that some of the most at-risk organizations are those managed by governments – where Windows XP or other obsolete operating system software may still be in use!
Regulatory expectations and keeping current
The formal title of FDA manufacturing regulations is Current Good Manufacturing Practice for Finished Pharmaceutical Products or cGMP. The focus here is on the word current. There is an explanation of the term in preamble comment 17 in the Federal Register of September 197810 where the GMPs were published, but a better explanation is found on the FDA’s website11:
The CGMP requirements were established to be flexible in order to allow each manufacturer to decide individually how to best implement the necessary controls by using scientifically sound design, … testing procedures. The flexibility in these regulations allows companies to use modern technologies and innovative approaches to achieve higher quality through continual improvement.
Accordingly, the "C" in CGMP stands for "current," requiring companies to use technologies and systems that are up-to-date in order to comply with the regulations. Systems and equipment that may have been "top-of-the-line" to prevent contamination, mix-ups, and errors 10 or 20 years ago may be less than adequate by today's standards.
Note, the last sentence: obsolete systems may be less than adequate by today’s standards.
Those working in Europe who do not make products for the US market should stop laughing now as an equivalent regulation is found in the European Directive 2001/83/EC Article 23 §112:
… the authorisation holder must, in respect of the methods of manufacture and control provided for in Article 8(3)(d) and (h), take account of scientific and technical progress and introduce any changes that may be required to enable the medicinal product to be manufactured and checked by means of generally accepted scientific methods.
So, both FDA regulations and the EU directive state that companies must keep current, and this includes computerized systems. Therefore, how will laboratories handle questions from an inspector that their systems are not current or up to date? Two possibilities are that the company policy of not upgrading the system or a supplier’s support is inadequate. Here we can look at periodic reviews of computerized systems.
Role of periodic reviews
To ensure that computerized systems remain in a validated state, EU GMP Annex 11 clause 11 requires periodic evaluations:
Computerised systems should be periodically evaluated to confirm that they remain in a valid state and are compliant with GMP. Such evaluations should include, where appropriate, the current range of functionality, deviation records, incidents, problems, upgrade history, performance, reliability, security and validation status reports.3
Whilst not explicitly stated, obsolescence of software components is a part of a periodic review as a system not maintained by software updates may have security, backup and recovery flaws and data may be vulnerable to loss from malware.
In the unlikely event that there have been several software changes during the period covered by the review, there should be an assessment of the cumulative impact of the changes on the system.
However, PIC/S PI-041 in section 9.3.5 as part of a periodic review states that data integrity limitations of software should be addressed in a timely manner.13 Given the extremely close alignment of PIC/S and EU GMP, this may be an addition to the draft Annex 11 update although it is not mentioned explicitly in the 2022 proposal.14
Equally so, as hardware ages, the bathtub curve shows that the failure rate increases and the system and the data it holds becomes more vulnerable.
Increasingly, instrument and application suppliers can provide some level of instrument monitoring services, so that instruments that break down more are prioritized for replacement. Following Sod’s Law, any such failure will always come at the worst possible time. Therefore, one role of a periodic review should highlight to management the risks posed by obsolete systems and simplified through a monitoring service. We will explore the regulatory risks in the next section.
An inspector calls 1
Table 1: FDA 483 observations and warning letter citations for analytical systems failing to be current.
| Regulatory Citation | Company and Date |
| 1. No audit trail functionality in the software or failure to turn audit trail on |
|
| 2. Ability to change date and time |
|
| 3. Unrestricted access to data/overwriting of data |
|
| 4. Data acquired by staff no longer working for the lab |
|
Top of the Museum non-compliances list is either the failure of the system to have an audit trail or failure to turn it on. Keeping current is now critical. ALL GMP laboratory systems MUST have an audit trail. The proposed update to EU GMP Annex 11 clause 9 has the following:
18. An audit trail functionality which automatically logs all manual interactions on GMP critical systems, where users, data or settings can be manually changed, should be regarded as mandatory; not just ‘considered based on a risk assessment’. Controlling processes or capturing holding or transferring electronic data in such systems without audit trail functionality is not acceptable; any grace period within this area has long expired.14
An automatic audit trail is mandatory in systems handling GMP data and any grace period has long expired. This is why regulators are citing laboratories for systems without adequate audit trail functionality. Enhancement of Annex 11 clause 9 for audit trail is contained in 7 of 33 proposed updates of the regulation.14
Moreover, this is reinforced by PIC/S data integrity guide PI-041 in Section 9.6.1, which states:
Companies should endeavour to purchase and upgrade older systems to implement software that includes electronic audit trail functionality.13
Furthermore, failing to have an adequate audit trail in an application is a failure to keep current with FDA and EU regulations,11,12 as discussed earlier in this article. Analysis by the authors of 104 483 and warning letter citations for infrared spectrometers showed that 40% of the citations could have been prevented by an adequate evaluation of the compliance features of the application software.26 Caveat emptor!
Please note that recording audit entries manually in a log book or controlled record is unacceptable. Therefore, systems without an audit trail must be upgraded or replaced urgently. The draft of the proposed Annex 11 update is scheduled by the end of 2024 for industry comment and the final version is scheduled for September 2026. The clock is ticking. Act now.
Role of the supplier
A key support role for any supplier of analytical systems is to provide accurate and reliable release notes for each version or hotfix that they issue, e.g., new features and bug fixes. The aim is to inform users of changes and their impact on the system: e.g., is a change localized or does it impact the entire system?
This information permits any laboratory whether to upgrade or not and also the level of revalidation required from nothing to a full system revalidation. The process owner is accountable for the decision and extent of validation in conjunction with the validation team and QA. Typically, for most laboratory computerized systems, these updates are not frequent unless there is a critical software bug that requires urgent resolution.
This is unlike a SaaS LIMS where software is upgraded every 3–4 months whether you like it or not as this is in your supplier contract.1
Application upgrades
It may be obvious, but it should be stated, that when a supplier upgrades their software, they do not rewrite the complete application code. Therefore, the bulk of the software does not change and you can use this to your advantage when considering revalidation. Updates are typically incremental.
Special notice should be taken of the application version numbering, e.g., if version 3.2 is operational and version 3.2.1 is issued, this is a minor release containing mainly error fixes. Thus, the last validation protocol may be almost reusable, in whole or in part, with minor revisions. In contrast, a release to version 4.0 is a major update.
Many analytical systems acquire discrete data files that are processed by the software with user oversight, e.g., chromatography and spectroscopy. The initial validation project acquires one or more sets of these data files which can form the baseline for comparison when the application is updated. After the release notes have been read by the validation team and any of the new features incorporated into the specifications, the new features can be tested.
In addition, the original validation data files can be rerun or reprocessed with the updated application: are the results identical or equivalent? If yes, the application engine still works with consistent intended performance according to 21 CFR 11.10(a).5 Testing can be performed in-house, by the supplier or by a third party but with QA oversight.
Application replacement
When a new application replaces an existing one from the same supplier, the key question is can data acquired on the current system be migrated to the new one and equivalent results be obtained? Note the use of the word equivalent. Providing data can be migrated and reprocessed and the same decision reached?
For example, CDS data were migrated from one supplier’s application to another supplier’s application. When reprocessed, the peak areas were different but when a standard curve was plotted and concentrations in samples calculated the results from both systems were within pre-defined acceptance limits.27
Although there are data standards within analytical chemistry, e.g., J-CAMP, AnIML, AIA etc., full and consistent implementation between suppliers has not been achieved yet. This may limit the successful migration of data from one supplier’s system to another.
Alternatively, if the data cannot be migrated, then you may have a new candidate for your Museum of Analytical Antiquities. The electronic data must be available throughout the record retention period.
Operating system obsolescence
Operating system (OS) obsolescence is a more critical issue, where using a component of the operating system, such as the .net framework, can be vulnerable and cause problems. Microsoft has a dedicated web page with a list of supported and unsupported versions of .net (https://dotnet.microsoft.com/en-us/learn/dotnet/what-is-dotnet-framework ).
This is especially so if your laboratory system is connected to the company network, as many IT departments require all systems, including the operating system, to be current, including security patches, etc. There is also the regulatory requirement for change control, such as Annex 11 clause 10.3
When an OS is declared obsolete there is a period of notice from the OS supplier. For example, Microsoft declared in June 2021 that the end of support for Windows 10 would be October 2025 unless a company was in the Long-Term Servicing Channel (LTSC) when paid support would be available for a further 3 years. Interestingly, notice of obsolescence was given before the official launch of Windows 11 in October 2021. The overlap of Windows 10 and 11 provides suppliers with time to determine if they will port their application to the new OS or not.
Does your old app run on Windows 11?
The long notice period for Windows 10 obsolescence is an opportunity to inform customers that an application will not be ported and the instrument will become obsolete. But in its place, we have this amazing new software and instrument that will run on the new OS available for a very reasonable price. If this is the case, the laboratory has the choice of using the system with decreasing support or taking the option of looking at alternative systems and suppliers.
Another important consideration is you might have to upgrade the computer to run a new application software. In some cultures, there can be a mindset that the computer is part of the instrument and should not be upgraded!
A selection question at purchase must be: Can we continue and keep integrity to read our GxP records and meta data if the application is obsolete? The key point is that many laboratory systems have proprietary records formats and so you need the application to read them for the record retention period. Will a supplier be able to do this for 5, 10, 20 or 30 years, depending on the record retention period?
If a laboratory selects to use an obsolete system, they will be running on an OS no longer supported and will have known cybersecurity vulnerabilities. These will be standalone systems as IT will not accept a system that does not meet their current standards. When operating in standalone mode, data input is manual with additional checks and will be a hybrid system with two incompatible media to synchronize.
Backup and recovery will be performed by the laboratory. In practice, this means that it will be poorly done when analysts have spare time and will result in a high risk of losing data on an aging system. A word of advice: DON’T.
The best advice is to upgrade to the new operating system, if technically possible. However, there are problems as we shall see now.
An inspector calls 2
Even when a laboratory upgrades an operating system there could be problems as found during regulatory inspections. The first 483 citation deals with an upgrade of the Windows server operating system. Lupin had an FDA inspection in March 2023. In the resulting form 483, Observation 5 noted a change control was initiated in January 2023 for the update of a system from Windows server 2008 to server 2016.28 This is a change that was initiated just before the FDA inspection.
The problem was the end of extended support for Windows server 2008 was January 2020. The replacement operating system Windows server 2016 was already past its end of life in January 2022 and is only on paid extended support until January 2027. The company classified the change as minor but did not perform an assessment of it.28 There is no mention of the application and data that were ported to the new operating system or even consideration if the system needs to be tested to see that it worked after the upgrade.
The second 483 citation in Observation 6 is similar, Olon an API manufacturer had an inspection in April – May 2023 with the following citation:
In 2014, you then initiated change request #CR/M/14/377 to upgrade existing Windows operating system software for the CDS and database but again failed to perform a requalification of all three components (HPLC, conductivity detector and the chromatography interface).29
The key learning point is that an inspector can go back 9 years to generate an observation.
Both citations show the same approach: update the operating system but without any documented work to show that the application and analytical instrument still work as intended. Figure 2 shows that the operating system is the foundation software layer upon which all other system components depend. When upgrading to a new operating system version, the process owner must ensure that the whole system works the same as before e.g.:
- New hardware (if used) and operating system are qualified.
- Does the database and application need to be reinstalled and existing data need to be ported (new hardware) or is the upgrade in situ on the same workstation?
- Instrument control and data acquisition especially when software drivers are updated.
- Data storage via the database.
- Existing data can be reprocessed with the same results.
This list assumes that the database and application versions are the same. The amount of revalidation and requalification depend on the extent of the changes and how well it was documented, but it is unlikely that a complete revalidation is required as will be discussed in the next section.
Figure 2: Analytical system showing the layers of software installed on the workstation. Credit: R D McDowall.
Revalidating the system
To avoid your prized analytical systems being candidates for your Museum of Analytical Antiquities it is best to keep the system updated. The most common question when upgrading is how much revalidation to perform? The questions to answer are very simple:
- How well does the laboratory understand the operation of the application?
- How well documented and how good was the validation by either the supplier or the in-house team?
- What are the changes to the application?
- What is the impact of each change on the functions of the system across URS and Configuration Specification (CS) and the data?
Often companies will validate all functions in an application. This is not necessary as the FDA Guidance on General Principles of Software Validation in section 6.1 states only validate those functions you use:
… who chooses not to use all the vendor-supplied capabilities of the software only needs to validate those functions that will be used and for which the device manufacturer is dependent upon the software results as part of production or the quality system.4
More importantly, the next section looks at revalidation after an upgrade, stating:
When software is upgraded or any changes are made to the software, the device manufacturer should consider how those changes may impact the “used portions” of the software and must reconfirm the validation of those portions of the software that are used.4
Therefore, the message is clear: only validate the functions used and confirm that they work when upgrading. To help with the latter, the initial validation would have acquired data sets that can be reprocessed to determine that the system works as expected and produces the same results. Revalidation is not difficult; all it requires is a clear analysis of what has changed and the impact of those changes. Here, accurate release notes from a supplier should help you.
A final piece of advice: back the whole system up before any actual upgrade work begins! You need to have at least two full and verified backup copies in case of a disaster when upgrading so you can roll back to the original system.
Summary
As noted in the summary of the LIMS article,1 it is much easier to have small incremental upgrades which are easier to handle, rather than major projects if left too long. The same applies to computerized analytical systems.
Many citations for older systems focus on their failure to have audit trail functionality. With the update of Annex 11, audit trail functionality will not be a regulatory expectation but mandatory. To avoid potential technical and regulatory issues, start upgrading the systems in your Museum of Analytical Antiquities NOW.
In October 2025 Windows 10 will be obsolete. Many laboratory systems run on this operating system – what will you and your supplier do? This appears as déjà vu all over again as there are still systems with Windows XP and Windows 7 that are currently entries in the Museum.
Will your Museum of Analytical Antiquities increase in size?
Acknowledgements
We would like to thank Mahboubeh Lotfinia, Mike Korbel, Simon Mitchell, Steffan Nielsen for their review and comments in preparation of this article.
1. McDowall RD. How to future-proof your LIMS: Handling software updates. Technology Networks. https://www.technologynetworks.com/tn/how-to-guides/how-to-future-proof-your-lims-handling-software-updates-370757. Published March 3, 2023. Accessed September 5, 2024.
2. United States Pharmacopeia. General Chapter, 〈1058〉 Analytical Instrument Qualification. USP-NF. Rockville, MD. 2024.
3. European Commission. EudraLex - Volume 4 Good Manufacturing Practice (GMP) Guidelines, Annex 11 Computerised Systems. European Commission: Brussels. 2011.
4. Food and Drug Administration. FDA Guidance for Industry General Principles of Software Validation. Rockville, MD. 2002.
5. 21 CFR Part 11; Electronic Records; Electronic Signatures Final Rule. Federal Register. 1997;62(54):13430–13466.
6. Good Automated Manufacturing Practice (GAMP) Guide 5, Second Edition. Tampa, FL: International Society for Pharmaceutical Engineering. 2022.
7. GAMP Good Practice Guide A Risk Based Approach to GXP Compliant Laboratory Computerised Systems, Second Edition. Tampa, FL: International Society for Pharmaceutical Engineering. 2012.
8. McDowall RD. Consigning SneakerNet to the graveyard of technology. Spectroscopy. 2021;36(4):14-17.
9. European Commission. EudraLex - Volume 4 Good Manufacturing Practice (GMP) Guidelines, Chapter 7 Outsourced Activities. European Commission: Brussels. 2013.
10. 21 CFR 211 - Current Good Manufacturing Practice for Finished Pharmaceuticals. Federal Register. 1978:45014–45089.
11. Facts about the Current Good Manufacturing Practices (CGMPs). FDA. https://www.fda.gov/drugs/pharmaceutical-quality-resources/facts-about-current-good-manufacturing-practices-cgmp. Published 2021. Accessed September 5, 2024.
12. European Union Directive 2001/83/EC on Medicinal Products for Human Use. European Commission: Brussels. 2001.
13. PIC/S PI-041 Good Practices for Data Management and Integrity in Regulated GMP / GDP Environments. Pharmaceutical Inspection Convention/Pharmaceutical Inspection Cooperation Scheme. 2021.
14. Concept Paper on the Revision of Annex 11 of the Guidelines on Good Manufacturing Practice for Medicinal Products – Computerised Systems. https://www.ema.europa.eu/en/documents/regulatory-procedural-guideline/concept-paper-revision-annex-11-guidelines-good-manufacturing-practice-medicinal-products_en.pdf. Published September 19, 2022. Accessed September 5, 2024.
