First: Before starting the Use Case Development Guidelines, please read and review the attached PowerPoint slides. These slides provide a high-level overview on:
- Introduction to the Use Case Guidelines
- Overview of a Successful Use Case
- High-level Content of the Use Case Guidelines
If a use case team has any questions or comments after reviewing the attached slides, please reach out to the CodeX Program Management team (firstname.lastname@example.org).
|Date of Last Edits||Slides|
Second: Review a summary of the Use Case Development Guidelines content, attached below:
Third: Complete the Use Case Development Guidelines, found here:
Use Case Development Guidelines Checklist
|Category||Instructions||Current Status||QM Member Team Feedback||Caroline's comments|
|Use Case Concept|
This information provides a link to information available to the public and use case members
Public page available here: https://confluence.hl7.org/display/COD/Quality+Measures+for+Cancer
Member-only UC web page available here: https://confluence.hl7.org/display/CPP/Quality+Measures+for+Cancer
Is there guidance somewhere else on what content should be public vs. member-only?
AD: I think we can simply state that any content a UC team is comfortable having available out in the public can be in Public. Anything that would be considered to have a level of restriction should be in private/member-only section.
This information provides an overview of why this use case is needed, including a description of the challenges with the current state.
Relevant articles supporting problem statement:
ASCO Comment (SJ): Specific to the 3rd bullet, I don't think the structured data challenge is entirely unique to oncology. I would lean towards removing it.
No additional resources to add from ASCO.
This information provides an overview of the use case proposed solution.
Note: Regarding our proposed solution, if the measure steward(s) we work with (i.e., ASCO, VA, ASTRO) gives us the authority to publish the measure publicly:
Scope of Work
The use case team should define, in further detail, the full scope of work for the use case. This should include the overarching use case scope, the associated phases of work (high-level), and how the scope intends to solve the problem identified in the earlier criterion.
ASCO Comment (SJ): Since we determined early that the pain intensity measure was not a good fit for the project, should we remove it from the Phase 0 Milestone? Of did you intentionally want to leave it in to illustrate why it wasn't a good fit?
AD (4/6): We intentionally left pain intensity in to highlight that this team did attempt to model it and determined it was not a good fit. Our report-out of this work is linked in the project plan, as well.
CP (6/1): I think this info is captured in the "Planned Phases" row?
When I think of scope, I also think about what is NOT included in the use case. Might be helpful to add that to the 2nd bullet?
AD: Great point. I'll add a sub-bullet requesting that UC teams document what is not included in the use case (or what the 2-3 main things are)
The use case team should provide a description (quantitative and/or qualitative) of how the proposed solution will improve the current situation; this could include better quality, less expensive/burdensome, more equitable care, research, surveillance.
The intent of the QM use case is to highlight new quality measures that could be developed and new insights that could be made IF mCODE and other FHIR IGs (QI Core) was implemented and leveraged by an ecosystem of health organizations v. trying to retroactively map FHIR to existing measures. This is the value of using the ASTRO/VA radiotherapy measures to be an exemplar showing what can be achieved, by leveraging mCODE and RT IG, to create and hopefully roll out new/innovative quality measures.
Note: in our current phase of work, the QM team is demonstrating that mCODE/FHIR can be used to author and execute oncology measures, however, it's not currently easier.
Note: profile-informed authoring may be a sub-component of our QM use case. In addition to the support/collaboration with the CQI WG collaboration.
For the one I bolded, I think you can do this at a high-level during planning, but probably won't have the nitty gritty details until you get in execution.
AD: agreed - I'll note to include "high-level" for planning and "detailed" for execution.
|Use Case Team|
The use case team should provide an overview of the types of planned meetings to ensure that the members are engaged and continuing to formulate or drive towards the use case plan.
Channels of work
ASCO Comment (SJ): Agree with Telligen best to keep technical meeting ad hoc for now.
Maybe share examples of types of meetings? Use case team, leadership, technical, etc.
AD: great point. I'll add RTTD and QM's current meeting structures, as examples. See Slack message from Karen.
The use case team should have a listing of organizations willing to lead and drive the use case scope of work (also known as use case “champions”)
ASCO Comment (SJ): In keeping with your current role themes, I would also add 'Clinical Informaticist' to Yvette's role.
Should there be a section about “project management leadership” or something like that? Thinking about who is going to manage all these meetings, prepare for them, do all the use case leadership roles, etc.
AD: I think PM leadership is a combination of overall UC Coordinator support + technical and terminology leadership roles. I'm open to considering a new role, though.
Member Representation & Necessary Stakeholders
The use case team should have a list of all members – both current members and aspirational members. This information will help CodeX leadership determine if:
|CP (6/1) - not all the cells are filled?|
The use case team should identify what asynchronous communication platform to use for communication across all member organizations. The decision should be reflected within this criterion and the asynchronous communication platform should be stood-up and active before a use case can move into Execution stage.
Presently, the Quality Measures team feels communication via email is sufficient for the success of the use case. Existing CodeX members did note that, because of availability, Teams would be the preferred use of another communication platform, if we decide to move beyond email.
The use case team should provide an estimate of resources and funds needed from the CodeX member-fee-based budget (e.g., funding from CodeX Membership fees paid to HL7) that are not supplied from other member sources (in-kind support, etc.). Past experience suggests that a minimum of 0.25- 0.5 FTE (or 10-20 hours per week, on average) for a UC Coordinator plus 0.1-0.2 FTE (or 4-8 hours per week, on average) for overarching PM support is needed to operate UCs successfully.
Resourcing that is currently supported by members:
Note from Telligen: if there were additional technical members – then there could be either concurrent or differing work – this may add time to Telligen's current involvement.
ASCO Comment (SJ): I edited the ASCO bullets to be in alignment with how the other orgs presented their time.
Just a FYI comment about the "request for resources/funds"- I think this is going to be hard. Especially thinking about future prospective pilots that require a lot of money. Maybe provide some guidance on how much other prospective pilots for use cases have cost? It's why I bolded it for execution.
AD: Great point. This section may want to just be a high-level assessment of the $/time/resources that are anticipated to be required for a pilot. Caroline Potteiger Do you have prospective pilot #s you'd be able/willing to share?
Member Provided Resources
The use case team should provide a list of potential members who could provide support through in-kind support or membership fees to bridge the gap between what is needed and what is currently available.
Note: Are there additional needs in terms of in-kind project support (e.g., providing clinical SME input on a QM terminology meeting) that we should consider and quantitatively estimate here (in hours per week/month)
Note: Are there additional needs in terms of paid support (e.g., request to use CodeX member funds to support QM comms, etc.) that we should consider and quantitatively estimate here (in FTE hours per week/month)
I think “Member Provided Resources” and “Resources Requested” could be combined in one. You could just list all the resources you need and then mark the ones that could be member provided
AD: Agreed – I'll consolidate into one criteria.
The use case team should identify consortia or other organizations outside of CodeX with which the use case may want to interface, coordinate or partner, or at least remain aware of. Could be competitive.
Note from Telligen: suggest reaching out to AMA for assessment of interest/participation (non-competitive).
ASCO Comment (SJ): Update on the 3/9 Presidential Budget in Brief — we had a call with Vinitha and she wasn't even aware of the blurb, but said it simply means CMS is trying to incorporate more oncology measures into more CMS programs (e.g., IQR, OQR, etc.) in response to the Cancer Moonshot.
|Use Case Plan|
The use case team will need to define short, agile phases of work that make up the larger use case scope. These phases should be between 6-9 months in length and should define (at a high level) the specific tasks that will be required of the team in order to achieve the phase. Additionally, success metrics and key performance indicators should be defined for each phase.
Phased Workflow for measure development:
Measure Authoring Process:
|CP (6/1) - add a column for stakeholders required and what's needed?|
HL7 FHIR Implementation Guides
The use case team should mention how (if at all) new and/or existing FHIR Implementation Guides might be used or updated within the use case scope.
Existing FHIR IGs the QM team is leveraging include:
Implementation in Health Information Systems
The use case team should identify existing health information technology systems – either implemented at health sites or external to stand-alone organizations – that might be interested in participating on the use case by implementing aspects of the FHIR standard in their system.
For the purposes of this use case scope – we are focusing efforts on the rectangular portion of the above image. As Telligen works through this technical environment, we’re looking for additional organizations to implement our team’s solution. Thus, we’re looking to Oncology Information Systems (Varian, Elekta, RaySearch, iKnowMed, Flatiron), Health System EMRs (Epic, Allscripts, McKesson, Cerner, etc.), Payers (Cigna, UnitedHealthcare), and Associations/Accreditation Programs (ASTRO, USON, ASCO, etc.) to serve as endpoints for this implementation (Phase 2/3). The QM team is working through an outreach plan to share/demonstrate our completed phase 0 of work with these organizations to understand their level of interest and feasibility to implement.
Telligen: recommends staying with proposed AWS environment. Future-forward phases may want to implement/coordinate with OMOP.
Piloting in Real World Settings
The use case team should describe a plan for how the proposed use case solution could be tested in a real world setting through a pilot project to demonstrate its effectiveness and identify any real-world challenges.
For a pilot to be successful in a real-world setting, we need to engage with organizations that are willing to be DEQM endpoints (see below diagram). This would allow new organizations to participate in the AWS environment.
Description: Use Case Evaluation Environment is hosted on Amazon Web Services (AWS) cloud-based platform and I is available to authenticated users. It serves as the source for measure data (FHIR bundles) to be stored, loaded and archived and A FHIR Server FHIR server with mCODE capability and CQF Ruler which offers plugins that provide an implementation of FHIR's Clinical Reasoning Module, serve as a knowledge artifact repository, and a cds-hooks compatible clinical decision support service. FHIR data including measure reports and supporting data are transformed to an analytic database that serves as the source for an AWS Quicksight based dashboard for reviewing performance.
Process Flow (see below diagram):
The use case team should (1) identify any risks or barriers that could impede success of the use case and (2) state, if possible, proposed mitigations to address these risks to ensure success of the use case.
Quality Measures risks include:
Note from Telligen: to truly implement use case in the wild, we will need end point implementation. Future phases really need to focus on relying on engaging from organizations and their commitment to serve as endpoints
Note from ASCO: CMS' main interest is in anticipated MVP measures
Probably helpful to list risks by phase of work. Risks during the proof of concept can be completely different to risks during the prospective pilot
AD: Great point! I'll update the instructions to request that UC teams list risks by phase
Adoption and Scaling
The use case team should identify how the proposed use case solution can be adopted or scaled in a sustainable manner.
To ensure the use case can be adopted and scaled in a sustainable manner, the QM team identified three key components:
|ASCO Comment (SJ): |
Do we want to include MADiE (the future replacement for the MAT/BONNIE) here and a future endpoint for implementation would be to include the mCODE profiles in it?
I think this is important, but hard to do in detail at the beginning of the use case. We had manyyyy conversations about this for Trial Matching and went back and forth based on what we learned. Like who is going to host things? How much will it cost for people to implement in the long-term (not just pilots)?
AD: IMO - the point of this criteria is to just get the conversations started. It's OK if the details aren't sorted out but this allows the team to start throwing together thoughts/ideas.
Alignment and Decision Making
Decisions within the Use Case
Use case Leadership team (paying, sponsored, and government members) should define a standard approach for making decisions.
Decisions for the Next Stage
This criterion documents the process to request transition, and the subsequent approval, from one stage to another.
Hi CodeX Program Management Team,
The Quality Measures for Cancer use case team is requesting that we move into the Planning Stage of CodeX. We are introducing the idea/opportunity of moving directly to the Execution stage, given our current state of the use case, if possible.
The Quality Measures Leadership Team
***EVERYTHING BELOW THIS LINE IS DRAFT MATERIAL***
During Discovery. Before approval by the Steering Committee to move to Planning.
Quality Measures Member Comments
Status: Complete; In-progress; Not started
|Web||Create public and Member-only UC Web areas on Confluence. PM/Coordinator will help set these up using standard templates. Add content as noted below.|
Confluence pages for the Quality Measures for Cancer Use Case have been setup - including standard templates describing the overall measures and elements.
Action: CodeX QM team to review home page content and update, as needed
Telligen: We discussed some updates to diagrams in the terminology call. Once the full team is able to review and we have consensus, we do not have additional updates based on currently defined scope. As we proceed into phase 0.5 and phase 1, etc. we should be able to update a bit more on target measures, if there are specific issues/problems we can demonstrate solving in those measures, etc.
ASCO - No edits
On public page, provide a nearly complete description. References to studies regarding the quantitative and/or qualitative extent of the Problem would be helpful.
|Solution / workflow||On public page, provide, preferably, a single, proposed solution and workflow diagram (PM/Coordinator can help with graphics) around which Discovery UC Members (and those committed to join) are aligned. Should be a novel solution, but doable.|
Solution – Complete
Workflow – in-progress
|Telligen - See note above re: discussion of diagram on terminology call. As we get further along into technical solutions, we can provide additional high-level technical architecture diagrams.|
|Potential impact||On public page, provide an estimate of quantitative and/or qualitative improvements that might be realized if the solution proves successful. Categories of improvements could include better quality, less expensive/burdensome, more equitable care, research, surveillance.||Complete|
|Scope (in addition to above and below)|
Concept and solution are consistent with the vision, principles of CodeX:
Discovery stage → scope in terms of high-level objectives for the broader project work
Planning stage → dedicate more time to defining tasks/goals/owners of tasks/resourcing for Phase 0
While we're in Phase 0 - follow similar approach to defining tasks/goals/owners of tasks/resourcing for Phase 0.5
QM use case focus is oncology
Meetings with individual, potential Members is the best way to discuss specific interests.
Schedule, as early as possible, weekly or bi-weekly public calls to build broader interest and gather ideas that help to form the Concept, Team and Plan. PM/Coordinator will help target participants and schedule.
Schedule CodeX Leadership (Paying, Gov't and Sponsored Members or those who commit) calls to consider all input and make decisions.
See meeting and messaging recommendations - Communication Rhythm
ASCO - I think given our teams current bandwidth, the schedule of calls now is working well for us.
CentanniPark - current schedule of calls is working well. Great teamwork and valuable discussion on the calls.
|Member leadership||At least 1 "Member Champion", who commits to lead concept and planning alignment, engagement of stakeholders, etc. Add names to public CodeX Confluence page.||ASCO - Stephanie Jones (would be helpful if we could come up with some talking points )|
|Member representatives of necessary stakeholders and actors||Additional CodeX Members at any level or those who commit, such that there is at least 1 organization to fill each stakeholder/actor role in the UC.||ASCO - Still working to get Integra Connect contacts.|
|Resources requested from the CodeX Member-fee budget (in addition to the in-kind resources to be provided from UC participants)|
Estimate of full-time equivalent (FTE) resources (and skills) being requested to be funded from the CodeX Member-fee-based budget (e.g., funding from CodeX Membership fees paid to HL7). These are resources that are not likely to be provided through in-kind support from UC participants.
Note: Experience suggests that a minimum of 0.25- 0.5 FTE for a UC Coordinator plus 0.1-0.2 FTE for overarching PM support (engagement, governance, communications, education) is important to moving fast and in an organized fashion. If additional expertise is required for terminology, FHIR, architecture, software development, pilot planning/execution, etc., another 0.5 - 2.0 FTE could be needed.
ASCO - Adding up the time now spent by Stephanie, Caitlin and Yvette we'd estimate we are at 0.25 FTE.
Telligen- Agree with general estimates though this will need to be evaluated through phases on continuous cycle. Would also add that specialized collaboration and input from additional HL7 committees (CQI) and potentially stewards (AMA) would be idea. The commitment for CQI might evolve to a member(s) participating in CQI committee meetings on regular basis.
|Sufficient Member-fee and/or grant funding and/or Member in-kind resources (to meet the request in the previous row)|
List of specific, potential UC participants who are not yet Members or not committed to join.
Note: For resources that are needed, but not provided as in-kind support, assume that $150K or more of funding could be need through new paying Membership, grant funding and/or through freeing up of resources from existing and/or ending UCs.
|Note re: Health Systems: While health systems are most welcome to join CodeX at any level, we are not requiring health systems to formally join CodeX in order to participate in implementations, pilots and other non-decision-making, but important work.|
|Outside initiatives||Consortia or other organizations outside of CodeX with which the UC may want to interface, coordinate or partner, or at least remain aware of. Could be competitive.|
Add additional thoughts
ASCO - In 3/9 Presidential Budget in Brief: https://www.hhs.gov/sites/default/files/fy-2024-budget-in-brief.pdf. Page 75 mentions what appears to be a new 'proposal' to Expand Cancer Care Quality Measurement, but no specifics are given. I've reached out to Vinitha and Dr. Kline to try to get more intel.
Telligen: Suggest reaching out to AMA for assessment of interest/participation. (non-competitive)
|Agile, short Phases and success metrics for each Phase||On the public page, list draft, high-level time-frames for Planning work and initial Phases during Execution.|
|HL7 FHIR IGs|
Roughly, how existing FHIR IG might be used or updated.
Initial thoughts on the need to develop new IGs as stand-alone artifacts and/or supplementary to existing specific IGs, working with existing HL7 WGs and/or outside organizations, complex terminology work, balloting in HL7, etc.
QMIG (Quality Measures implementation Guide)
mCODE STU2 (3?)
DEQM (Data Exchange Quality Measures)
FHIR 4.0.1 and forward.
|Implementation in health information systems|
Existing IT systems that might be candidates to be updated to support existing/new FHIR IGs and functionality during pilots.
New systems that might need to be developed.
|Telligen: Recommend staying with proposed AWS environment Future Forward phases may want to implement/coordinate with OMOP.|
CentanniPark: Ideally could target systems implementing mCode to provide data inputs for the QMs that can be housed downstream (AWS etc)
|Piloting in real-world settings||Initial thoughts on the location and nature of pilots, preferable sharing data across real-world settings.||Telligen: Need to be exploring willing participants for DEQM endpoints.|
CentanniPark: Ideally could target systems implementing mCode to provide data inputs for the QMs that can be housed downstream (AWS etc)
|Risks||The most important, potential challenges to success. For example, divergent views in the community, competing efforts, insufficient resources, technical or clinical barriers, legal work, etc.||Telligen: Most significant risk is allocated resources having competing priorities and participation of endpoint organizations. Lack of regulatory participation?|
|Adoption and scaling||Assuming successful pilots, the 3-5 most important challenges to be addressed to gain adoption and scale for impact.||Telligen: Most significant risk is allocated resources having competing priorities and participation of endpoint organizations. Lack of regulatory participation?|
|Aligning and Decision-Making|
|Meetings within the UC||Experience has shown that a combination of some public and separate Member-only meetings is a good way to build broad interest, receive broad input, gain engagement and commitment, and still leverage Member meetings to make decisions.|
|Decisions within the UC||Decision makers (approval of all of the above) must be CodeX paying, Gov't and/or Sponsored Members or must submit written commitment to join at one of these levels if/when the UC transitions to Executing.|
|Decision regarding move to the next Stage|| |