Focus areas

Previous Issues

Subscribe

Performance benchmarks
Dr. Fredric J. Cohen discusses procedure-related benchmarks to inform and streamline study design.

Managers typically use performance benchmarks retrospectively – following completion of projects – to indicate areas for process improvements and to motivate personnel to stay the course for long-term process-improvement initiatives. Empirical evidence suggests that these uses of performance benchmarks can be effective. There are also opportunities to use certain benchmarks prospectively to drive performance, for instance, by lowering costs. One such opportunity is the use of study-procedure benchmarks to motivate and inform the design of less complex and less costly clinical studies.

Examples of useful study-procedure benchmarks include:
• contracted procedure costs
• the frequency of procedure employment (e.g. number of times a procedure is performed in a study)
• the procedure execution complexity (i.e. time and effort requirements)

The first two metrics are routinely used today after study design to benchmark costs for budgeting and contracting purposes. Procedural complexity measures are not currently routinely used in the pharmaceutical industry but are broadly used by governments and insurers to assess health care delivery effectiveness and to guide physician payments.

Making practical prospective use of procedure-related benchmarks during study design requires that study designers be capable of easily relating the chosen benchmarks to key study concepts that determine basic design decisions. Consider study objectives. Ideally, designers will create a set of procedures that fulfills but does not exceed the data collection requirements implicit in study objectives. For this to happen, there must be a clear accounting for each procedure of its directing objective(s). This task can be straightforward for studies with one or two objectives, directly correspondent outcome measures (i.e. endpoints), and a modest number of procedures. However, for more complex studies, keeping track of these objective-procedure relationships as both are changed during study design is not so straightforward, particularly when there are multiple intervening endpoints.

Another factor facilitating the prospective use of procedure-related benchmarks during study design is the ability of study designers to see in real-time how changes in the design affect benchmarked study costs. This real-time feedback is needed if designers are to accept benchmarks and use them routinely early in the study-design process.

At least one commercially available extensible clinical protocol-authoring package (TrialSpace Designer XCP by Fast Track Systems) now includes the capabilities described above. Consider three hypothetical examples that illustrate how procedure-related benchmarks might be used during study design.

Example 1
A study designer generates an automated report detailing procedure costs grouped by study objectives. It shows that procedures performed to satisfy the proposed tertiary efficacy objective are twice as costly and nearly as complex as those proposed for the primary and secondary objectives, owing largely to the high cost of a quality-of-life (QOL) survey administered four times during the study. Real-time design scenario feedback indicates that decreasing the QOL survey administration to an acceptable frequency of twice would reduce the cost of the tertiary objective to below that of the secondary objective.

Example 2
A few investigators voice concerns during site feasibility about the complexity of the proposed study. An automated procedural complexity report suggests that the study is more complex than 90 percent of its peer group, lending objective credibility to these opinions. Real-time design scenario feedback indicates that reducing the amount of blood sampling for PK measurements would reduce the procedural complexity to the peer-group median.

Example 3
A protocol reviewer creates an automated benchmark procedure frequency report, which indicates that 80 percent of studies in an appropriately matched peer group contain at least one head MRI scan. Further, the median frequency in such studies is two MRI scans. However, head MRI has not been included in the study being reviewed. The reviewer raises the issue with the lead study designer who reconsiders the need to include a head MRI scan at the outset, potentially avoiding an amendment to add it after enrollment has begun.

These examples suggest methods to realize research efficiency gains by incorporating procedure cost, complexity and frequency benchmarks prospectively into study design decision-making. It remains to be seen how best to implement such benchmarking practices into the protocol design process. There are forces opposed to introducing what they view as a ‘corruptive influence’ of cost considerations into the clinical research design process. But these forces must be opposed if the industry is to achieve the clinical research efficiency and productivity improvements necessary to maintain the industry’s heritage of returning a high proportion of sales to R&D. A transition to a cost-disciplined approach to study design will be facilitated by the new technologies described here.

About Dr. Fredric Cohen
Dr. Fredric Cohen is the VP of Clinical Strategy for Fast Track Systems, Inc. He joined the company in April 2007. Cohen has been involved in pharmaceutical R&D, licensing and business strategy for 12 years and medical research for 20 years. He is particularly interested in strategies and tactics designed to improve industrial research productivity.