Medical Affairs Focus

Thoughts on Global Medical Affairs

Tag Archives: Metrics

Topic 43 – Use of Net Promoter Score Measures to Evaluate MSLs

A couple of my clients have discussed the use of the Net Promoter Score lately so I thought I would address it in my blog.

Quick background:

The concept of the Net Promoter Score was introduced in a Harvard Business Review article in 2003 by Fred Reichheld of Bain & Company. The net promoter score is measured by asking a single question: “How likely are you to recommend the company/product/service to a friend or colleague?” and is usually measured on a 0 to 10 basis. Scores of 9 and 10 are called Promoters, scores of 0 to 6 are Detractors and scores of 7 and 8 are called Passives. The Net Promoter Score is calculated by subtracting the percentage of customers that are Detractors from the percentage of customers that are Promoters.

People like the net promoter score because it is a simple measure of loyalty and when it is paired with an open ended question that asks why the particular score was given, it provides insight into what is important to the customer.

So, does the Net Promoter Score (NPS) provide value to MA? My research has not been able to find a single academic or metric-driven study on the use of NPS in MA specifically related to MSL activity. Common sense says this approach should be helpful but for now anyone using this approach is in experimental mode.

In MA the NPS question is often modified to be:

  • “How likely are you to recommend engaging with [COMPANY X] Medical Science Liaisons to your colleagues or peers?” OR
  • “How likely are you to recommend working with [MSL NAME] from [COMPANY X] to your colleagues?

PRO’s of Using NPS in MA:

  • Brief nature of survey makes it suitable for rapid deployment immediately following MSL interaction to avoid the “blending” affect that occurs when HCPs are asked about MSL performance on a standard survey often weeks after their last interaction
  • Relatively inexpensive to conduct compared to other market research
  • NPS can help gather insights into what an HCP value in an MSL interaction, if open ended questions are employed as well

CONs of Using NPS in MA:

  • Message vs Messenger: When an HCP recommends working or engaging with an MSL is that recommendation based on the quality of the content of the interaction or the interpersonal qualities of the MSL herself or himself?
  • Not comparative: NPS does not give insight into whether HCPs recommend your MSLs any differently than they recommend competitor MSLs. Perhaps HCPs in a particular therapeutic area simply recommend all MSLs the similarly regardless of company.
  • Not clearly actionable: If your NPS drops from one month to the next, what action should be taken? Some insight might be provided by the open ended questions but those responses are often only provided by the most dissatisfied

Given the inherent challenge, it is my opinion that the NPS is still a worthwhile measure, but it needs to be gathered as a part of a broader market research effort to give it the context that can help tease apart the reason for the scores.

The most effective NPS is gathered as soon as possible after the last interaction. In the case of MSLs, a system should be established to seek this guidance directly after a contact has been noted in the company’s contact management system. And, like all market research with HCPs, participation is highly impacted by compensation, so sufficient compensation must be offered to ensure enough participation to make the measure meaningful.

What is your experience with NPS? How do you frame the question? Share your experiences by clicking here.

Topic 26: Highlights of Day 3 of the Medical and Scientific Communications 2013 Forum

­

The final half day for the MSC meeting had an excellent and well received discussion about MSL metrics.  I also attended an interesting talk on coordinating global publications.  My take on it below

DISCLAIMER:  My focus is on interesting information, I did not try to take verbatim notes and will not try to assign comments to a particular speaker.  Any mistakes are mine alone.

In no particular order, some of things are found interesting included:

–  Metrics

  • Must vary as focus of the organizations varies
    • Plays back to earlier comments about staying focused if you want to see improvement over time
    • Mix of Activity, Value and Compliance measures are needed
    • IIS Metrics
      • Good to track in terms of just a sense of the program
      • NO goals should be put in place
      • Additional metrics discussed
        • Engagement with ACOs or Cancer Networks or other new healthcare entities
        • Congress attendance valued by quality metric of the outputs not the activity
        • Dollars saved by asking MA to assist CD, especially when CD has a bid from a vendor to provide the same service
        • Internal stakeholder feedback
          • Important – best to solicit directly via email/conversation
          • BE CAREFUL – some organizations do not allow feedback from Commercial

–  Reports/Dashboards/Scorecards

  • Need to develop different outputs for different stakeholders
  • What is needed internally at MA is not what you want to focus on with other internal stakeholders
  • Always pair summarized numbers with text explaining value and highlighting accomplishments/learnings

–  MA IT Systems

  • As mentioned earlier, a number of companies seem to be moving toward the Veeva Platform which is built on top of Salesforce.com CRM.
    • Veeva seems to be driven by the commercial side, but it has MA capabilities
    • Some people mentioned that you need to be careful about Veeva training and documentation as it often defaults to commercial language
    • Some organizations use it to track not only KOL interactions but also time spent in the office doing various projects
    • Focus on time tracking seemed to be for internal management not for external publication

–  Global Publication Coordination

  • Increasingly important to have global plan
  • Much easier to manage when Pub Team gets involved earlier in the development cycle
  • MA on Pub Team should work as a facilitator for Pub Team’s interaction with other functions given their central role
  • Press release management globally remains a major challenge
    • General consensus that prior to first presentation at a meeting, press release should be simply binary – met endpoints, did not meet
    • Too much data in press release can tick off journals threatening publication

–  Odds and Ends Ideas

  • Heard from a number of different companies that MSL groups focused on payer support in the US need to be between 6 and 10 in size to cover all the needs – not scientific but an interesting convergence
  • MSLs at clinical study initiation visits:
    • Good idea BUT has GCP implications
    • Need to hammer out how MSLs can fit within CD’s SOPs with ClinOps before engaging
    • May ultimately require update to CD’s SOPs

Obviously, this is merely a few key concepts from the day but it should give you some flavor of the type of discussions and the focus of the meeting.

If you were at the meeting and have other thoughts please leave them in the comments.

Topic10: MA Reporting Relationships in Small Companies

A reader wrote in to discuss a situation at a small pharma company.  This company was just starting up its MA group and it was not sure where the MA group should report.  The reader was well aware of the issues with having MA report to a commercial leader that we discussed in this post.

The challenge in this situation was that senior management consisted currently of just a CEO and a COO.  While the CEO is a scientist who handles their development efforts, the COO handles all other responsibilities, including   Commercial, Finance, Legal and HR.  The CEO would be the natural choice for managing MA, but given workloads it was not practically feasible.  Could the COO handle the job of managing MA even though they also have some commercial responsibilities without putting the company at risk?

This is a very common situation for young organizations.  There is often very little senior leadership to go around.  Yet, as we mentioned in this post, it is critical to start MA up early to support a product launch that may be over two years away.  In situations where it is not feasible for MA to report to someone purely on the science side and they need to report to someone with commercial responsibilities, my rule of thumb is that the role they report to should have substantial responsibilities outside of commercial and should be measured on results that go beyond just commercial measures.

In the case above, the COO has commercial responsibilities but they also have non-commercial responsibilities as well.  Having MA report to the COO should not be a problem, as long as the COO understands the role of a compliant MA organization and follows the general rule about MA interactions with commercial that were outlined here.  In the end, the Agency looks for actions not reporting relationships, but the reporting relationship do send a clear signal about the priorities of the MA organization.

What has been your experience about MA reporting relationships?  Please leave a comment below.  And if you are new to the blog please sign up for email updates by clicking on the link to the right.

Topic 6: Measuring the Value of Medical Affairs

Introduction

A reader requested a discussion of different approaches for measuring the value of medical affairs.  This is a critical question and one that I touched on briefly when I discussed strategic planning in MA as it relates to launch.  But it is equally important overall for MA.

Like any function in pharma, MA must be able to communicate its value.  Value can be a tricky concept, however, for a non-commercial function like MA.  Value is ultimately a company-specific question.  It stems from the company achieving its strategic goals, which is why defining goals is so important.  Having said this, there are some general categories of value measures that we can discuss as well as some best practices around measuring value.

Value measures in MA typically fall into two types:

  1. Objective measures
  2. Subjective measures

Objective Measures

In MA, objective measures tend to focus around activity-based measurements.  Since MA cannot be seen as promotional, the other available objective measures like prescriptions written, sales numbers, profitability, etc. are not relevant.

Activity-based measures have both strengths and weaknesses.  On the strength side activity based measures are:

  • Easy and cost efficient to gather
  • Easy to explain to non-MA colleagues
  • Non-controversial

Unfortunately, activity based measures also are:

  • Not outcomes focused
  • Not usually tied to the direct concerns of the organization
  • Potentially misleading

While it may be interesting to know that the MSL group conducted 150 meetings with KOLs in the last quarter, it tends to beg the question – So What?  Activity-based measures tend to focus on the tactics (e.g. reach out to KOLs) rather than the goals (e.g. ensure broad awareness of the latest disease state information).

Activity-based measurement is very appropriate for certain MA functions.  For example, on CME funding, which by definition must be a hands off process, the only important measure is activity-based: Did the organization fund the number of CME programs it had intended to fund?

Or, in the Medical Information function, the key measures relate to answering inquiries and responding.  Activity-based measures (e.g. total number of inquiries managed, average turn-around on inquiries, number of inquiries requiring a second contact) are very appropriate for this type of service.

Finally, activities with multi-year goals like post-approval clinical trials, use activity-based measures to ensure that the overall program is on track.  So activity-based measures like patient recruitment and data collection provides valuable insight.

Activity-based measures are a necessary for MA’s ability to track progress against its goals, but they not sufficient to account for all of MA’s goals.

Subjective Measures

Most MA organizations have goals that require a more subjective type of measure.  Some of these goals relate to the degree of awareness or understanding in the healthcare community.  Measuring understanding or awareness is not as simple as measuring the number of discussions about a certain topic.  Instead, to measure progress on these types of goals, MA needs to gather more subjective data.  Simply put – it needs to ask.

For goals related to subjective measures, the only practical way to measure success is through research.  Unless a company is very lucky and finds that one of its objectives just happens to align with a topic that is already the subject of someone else’s research, this almost always means conduct primary research.

Research-based measures have their own strengths and weaknesses.  The strength of research-based measures include:

  • Direct connection to goals related to healthcare community awareness and understanding
  • Insight into the knowledge and beliefs of the healthcare community
  • May provide insight into other, unmet needs

Research-based measures have some significant drawbacks, including:

  • Expensive to conduct, leading to limited number of data points
  • Subject to research bias
  • May be taxing to the community that the organization is trying to serve

For research-based measures to be understandable, they usually need to be measured against a baseline.  For example, the measure of awareness of new disease-state information in the healthcare community after the efforts of the MA organization is only relevant if the measure of awareness before the efforts were known.  This requires the development of a baseline which both adds to the cost and requires good up-front planning.

But by far the biggest challenge of research-based metrics is the lack of experience and budget to conduct such research.  Many MA organizations do not have experience conducting this type of “market research”.  Without such experience, it can be daunting to initiate the research and structure it in the best manner.  Additionally, many MA organizations do not have money in the budget to conduct this research in a high quality manner.  This is both a product of the lack of experience and the view that this type of research is outside of the mission of medical affairs.

Best Practices

Clearly most MA organizations can benefit from a set of mixed objective and subjective measures.  Developing such a set of measures starts analyzing the strategy and goals outlined in the MA strategic plan.   From there a set of potential measures can be defined and prioritized.  Overall, MA should have between 5 and 10 metrics.  Specific functions within MA may have from 3 to 5 additional detailed metrics.  Once these metrics are defined, the approach for gathering each metric can be identified.  The activity-based metrics are typically gathered from existing tracking systems while the research-based metrics require more active management.

Once the research-based metrics are defined, a draft of a research outline can be developed, specifying the key questions, the research targets, the number of responses expected and the number of research samples to be taken.  This outline can be used as the basis for developing a request for proposal from a research company.

One caution – many organizations are tempted to combine this research with research already being conducted by the commercial market research function.  This may be a problem if the marketing questions are, in themselves, seen as promotional.  Most of my MA clients have been unwilling to risk this crossing of the line but some organizations do follow this practice.  Please work with your compliance function to discuss this approach.  Many organizations are not even comfortable with using research companies that are primarily commercially oriented but, frankly, there is little reason to avoid these companies.

After the approach to both activity- and research-based metrics is in place the next step is to determine how the results will be communicated and to whom.  This should become an ongoing process so it should be automated and standardized as much as possible.

Closing Thoughts

Explaining the value of MA remains one of the primary responsibilities of MA management.  For MA to be perceived as valuable to the organization, its progress against its strategies and goals has to be proven.

What has your experience been with value and measures in MA?  Leave your comments below.

Click on the email link to the right to suggest a topic for future posts.