Skip to main content

The Acquisition that Helped the Department of Veterans Affairs Modernize Its Claims Appeals System

By Clair Koroma and Brent Maravilla


Back to Case Studies


Summary

This case study provides the following information:

Introduction

The VA competitively awarded a task order for digital services off of the agency’s Indefinite Delivery Indefinite Quantity (IDIQ) contract vehicle called Transformation Twenty-One Total Technology Next Generation (T4NG). A coding exercise was leveraged as part of the evaluation.

Background

Every day, Veterans apply for health benefits from the Department of Veterans Affairs (VA). Those whose claims are denied can file an appeal to the Court of Veteran Appeals by using the VA’s Enterprise Appeals Process. However, customers using the system encountered several issues that prevented them from completing the appeals process online. These issues, coupled with other systemic inefficiencies, meant that hundreds of thousands of Veterans endured waiting for long periods of time for a claim to be resolved.

To help resolve the technological inefficiencies in the claims process, the U.S. Digital Service embedded a team at the VA. Their work focused on modernizing the VA’s Enterprise Appeals Process to enable efficient adjudication of appeals. On arrival, the team began to quickly develop remedies to fix the system. In order to continue the pace of work, they realized they needed more developer support. They decided to use a vendor to provide the additional developer support services to execute the project on the scale needed.

To ensure they got the right team to assist on this project, they evaluated the offerors with a coding exercise. This case study tells the story.

Lessons Learned

  1. In a coding exercise, ensure you have the technical talent on the evaluation team that knows how to evaluate quality of code submission.
  2. Technical team should have ability to understand and replicate an ideal submission.
  3. Determine if it is necessary to have a written submission when doing a exercise. The written portion can present an additional financial burden to the company that may not be necessary if evaluating the code submission will provide sufficient insight into technical ability of company.
  4. Consider use of a 2-step (down-select) process in the solicitation to decrease vendors’ bid and proposal costs
  5. If requiring a written portion, consider when is the best time to require it. In the case of this acquisition, vendor feedback indicated that upfront submission of the written portion was preferred since the code submission was more expensive.
  6. Within the evaluation, the Government should be cautious about imposing go/no go wording. For example, stating within Code Quality a requirement that “there are no flagrant misspellings or typos” can become very binary, and lead to unwanted deficiencies.
  7. Consider whether your evaluation criteria allows for sufficient strengths and weaknesses.
  8. Prior to the solicitation, ensure that your team clearly understands potential deficiencies. For example, must the vendor complete all user stories or are there specific stories that are key? During the evaluation phase, the team realized that theoretically a vendor could have made an excellent submission even if they had not completed all user stories.”
  9. Much of the coding evaluation made for a very objective analysis (e.g. Specific security vulnerability identified), which is helpful to both parties for debriefing purposes.

The Problem

The Digital Service team at the VA had two developers on arrival. Those two developers began work on modernizing the VA’s Enterprise Appeals Claims system. On delivery, their work would help alleviate the backlog in claims appeals that prevented veterans from getting benefits. As work progressed, it became clear that in order to continue the rapid pace of development, additional developer support was required. Using a vendor to maintain the dev pace was the fastest way to get high-quality support on the project. The vendor that could best meet the needs of the product would have developers with expertise in agile development, user-centered design, User Experience(UX) research, modern technology stacks, and DevOps. The question was how would they find this vendor? More importantly, how could they verify developer acumen in the aforementioned areas before contract award? Simple, they needed an innovative evaluation approach.

Finding A Solution

A hybrid VA-Digital Service team, including Contracting Officer Mark Junda, Contract Specialist Brandon Caltabilota (both graduates of USDS’ Digital IT Acquisitions Professional(DITAP) Contracting Officer training), Digital Service Expert (design), Kavi Harshawat, and Digital Service Expert (engineering), Shane Russell worked together to devise an innovative evaluation for submissions and unique acquisition strategy that were uncommon in their approach. Leveraging the use an IDIQ vehicle for the VA, the team hypothesized that they could use a code exercise to reach the right vendors. Although unconventional, a code Wil offered the best opportunity for a vendor to demonstrate how they would build a modern digital tool. It also presented the VA team with substantive code upon which the vendor could be evaluated. In theory this would help the VA to successfully identify a vendor who could help continue the pace and scale of the modernization work on the appeals system.

A Unique Contracting Approach

Separate contractual requirements from functional requirements

The VA anticipated that functional requirements will change, which is normal, and therefore separated contractual requirements from functional requirements. Since their contractual requirements indicate a repeatable process that results in working software code, this allows functional requirements to change as user needs change. The functional requirements can change without a modification because the changes to the features are within the general scope of the contract. For example, instead of adding contractual requirements that dictated the specifics of a user story like “the screen shall be blue” the requirement was “backlog of user stories shall be completed by the vendor.” This enabled the VA to reference the user stories as a requirement and allow for changes to the backlog as needed without contract modification.

Market intelligence

VA heavily leverages its T4NG IDIQ for software development services. Next the team had to develop a solicitation for the support services that evaluated vendors based on a coding exercise. Why a coding exercise? This is a common method used in commercial companies to help them selecting the right talent and vendors. It is also a common practice used by established software firms during their hiring processes. Therefore, it made sense to use this method to select the right vendor to provide digital services to the government.

Show Don’t Tell

Imagine it’s time for show and tell. Two kids show toys that day. The first, showed the class how to build a toy while continuously asking classmates what they wanted the toy to do. She modified it based on their feedback. She even had different classmates try the toy as she was building it and made changes for better functionality. The second, built an amazing toy step by step while the class watched. He never gathered feedback from the class during the build. After several kids tried each toy, most preferred the custom toy. Conversely, the toy built without any mods or changes based on class feedback, was seldom used because many of the kids found that it was difficult to maneuver use its features.

Sadly, the latter experience in this story is very similar to what normally happens when the Government buys IT. A company gives a shiny description of how they will deliver a digital service or tool, but absent a demo of their skillset to deliver it, or incorporation of user feedback as they work on the end product, we often end up with things that do not work for users.

In order to avoid that experience in their procurement, the VA team worked with the Technology Acquisition Center (TAC) to design a RFQ that included a coding exercise. While bidders had to submit a traditional written response that included management approach, technical approach, etc., they also had to demonstrate how they would get the work done.

The following is how the acquisition was executed:

  1. The VA Digital Service team designed the coding exercise parameters they wanted tested and then validated that the exercise could actually be run in the environment and timeframe they wanted to include in the solicitation. They used digital service team members from other teams as well as ran the tests themselves to validate.
  2. The Request for Task Execution Plan (RTEP) detailing the full requirement was released as a Service-Disabled Veteran-Owned Small Business set aside on the T4NG. It included the coding exercise submission instructions and evaluation approach. At this point the vendors could begin working on the response to the price and the written technical portion. The RTEP also provided guidance on the coding submission and detailed how the submission would be evaluated. Download RTEP PDF
  3. After release of the RTEP, VA provided notice that the coding submission information would be posted at a certain date/time.
  4. At the appointed time, nine user stories (See Appendix 1) were released on the T4NG repository where RTEPs are posted and proposals are submitted. Upon release, vendors had 4 hours to submit questions and the VA had 2 hours to respond to all questions.
  5. Vendors had 72 hours, from time of release, to provide a code submission.
  6. A detailed evaluation criteria (See Appendix 2) was developed to guide offerors on the requirements for successful completion of the exercise
  7. The exercise required the offerors to build an online payment system (venmo clone). The clone required extensive security considerations, so there were many opportunities for excellence OR mistakes!
  8. The source code for the coding submission and all relevant design assets and documentation were submitted via github repository with a clearly viewable commit history of the entire development process.
  9. Offerors formally certified that “the team who developed and designed the coding submission are proposed as key personnel to perform work under the resulting task order barring any unforeseen circumstances”

Evaluate the Right Things The typical evaluation criteria wasn’t going to work. The team consulted with GSA’s 18F team about their experiences running a coding exercise to select vendors for their agile Blanket Purchase Agreement (BPA). Some key suggestions from 18F included using automated testing to quickly show weaknesses or strengths in a vendors’ code base AND specifying the coding language for the submissions. The team then developed criteria that properly evaluated the quality of the submitted code and design.

The team used a combination of:

  1. Evaluation criteria in the RFQ that was written to allow for innovative approaches to the exercise, and
  2. Specific validation that enabled the team to evaluate submissions uniformly against the criteria.   An example of how this approach was used to evaluate submissions is in the following:

Example 1

Example 2

This combined evaluation approach was only possible because of the visibility that a coding exercise provides. Ultimately, that approach proved to be effective in identifying the best vendor. Submissions were evaluated against seven development categories and four design categories, each with subcomponents (See Appendix 2: Evaluation Criteria).

Results

The team received six submissions that each took 4-6 hours to evaluate. The rubric and evaluation criteria allowed the technical team to evaluate and select the vendor offering the best value and expertise for the project requirements. This submission style also gave way to well-informed debriefs that likely contributed to zero protest actions on the award. Ability to review code prior to contract award allowed:

  1. Test of the submitted code for errors that would be fatal to project success
  2. Determination of best code based on hard evidence of work completed in direct response to the RFQ, not just a narrative of how the vendor would accomplish requested tasks.
  3. Ability to identify the vendor with the developers that met the VA needs

Task Order Award Details

The period of performance is for a base plus 2 option years for a total awarded amount of $13,946,873.

Milestones in Modernizing the Appeals System Since Contract Award

  1. Product, Caseflow Dispatch moved from idea (before requirements gathering) to implementation in 5 months. This product ensures that all decisions made at the Board are uploaded to the proper VA systems, and that the claims are properly adjusted. This will close a gap through which thousands of appeals could get lost. The product also standardizes and correctly routes decisions of over 20 categories to over 70 VA offices around the country that use differing mechanisms for managing their work.
  2. Prototyped a document reader that can greatly speed attorney workflows at the board, which would have a direct effect on the speed at which the backlog appeals are processed.
  3. Automating Form 8, which is a part of the process for transferring appeals from regional offices to the Board of Veterans’ Appeals (BVA). This streamlines the process and speed up the rate at which appeals can be transferred to the BVA.

Conclusion

Incorporating a coding exercise into a digital services acquisitions better positions a team to select the right vendor to meet their needs. It allows teams the opportunity to work with a potential vendor in real time to learn about their capacity to deliver the quality of digital services needed. Additionally, it allows vendors an opportunity to learn the maximum amount of details to best respond for an RFQ.


Appendix 1. User Stories

Cashflow is a desktop web application where people can send money to each other, along with a message. It allows for users that use different currencies.

Login

Failed Login

Account Page - Not Logged In

Account Page - Show basic Information As a user, When I’m logged in and visit the account page, I should see

Bank Transactions

Account Page - Create Transaction

Account Page - Show Transactions

Account Page - Validate transaction

Account Page - Create Transaction Requiring Exchange


Appendix 2. Coding Exercise Evaluation Criteria

Traditional Requirements

A detailed description and diagram of the Offeror’s approach to architecting and implementing a secure, scalable, automated, fault-tolerant system that supports the ongoing integration of developed appeals-centric features and products in accordance with PWS paragraph 5.2.1.

A description of the Offeror’s approach to applying the design and development principles that will be used for the coding submission across the life of the task order to meet the continuous deployment requirements defined in PWS paragraph 5.2.2.2 for the products defined in PWS paragraph 5.2.1.

The Offeror’s detailed solution of the planned management methodology for executing the effort, including a proposed staffing approach, key personnel retention, security clearance considerations, and details of how the proposed effort will be assigned within the Offeror’s corporate entity and among proposed subcontractors.

Innovative Requirements

Coding Submission: The Offeror shall develop an application adhering to the requirements defined in Section B(3)(a)(3) Coding Submission and, in addition, shall be evaluated by the Government in seven categories as follows:

Security

Testing

Database/Data Modeling

DevOps

Code Quality

Application Quality

Coding Submission: The Offeror shall design an application adhering to the requirements defined in Section B(3)(a)(3) Coding Submission and, in addition, shall be evaluated by the Government in four categories as follows:

Visual Design

User Research

Interaction Design

Thoroughness


Download this Case Study

Download this case study


Back to Case Studies