Skip to main content

Veterans Benefits Administration (VBA) Salesforce Center of Excellence, User-Centered Design Challenge

By VA’s Technology Acquisition Center (TAC)


Back to Case Studies


Background

The Department of Veterans Affairs (VA) typically uses traditional approaches such as written technical evaluations to select and procure services from vendors. Evaluations of this type tend to consume significant time and effort both by the VA evaluation teams as well as the vendor teams developing the proposals. Additionally, written approach evaluations rely solely on the writing abilities of the vendors without allowing evaluation teams to get a feel for the culture and creativity of the team that will be performing the work. The current VA landscape consists of rapidly changing and evolving requirements/backlogs which require truly agile implementations and execution by vendors supporting VA. Traditional evaluation methods can lead to vendors with superior writing skills winning contracts over vendors that consist of creative, collaborative, and dynamic technical teams.

Jointly, VA’s Technology Acquisition Center (TAC), US Digital Service (USDS), VA’s Office of Information & Technology (OIT), and the Veterans Benefits Administration (VBA) developed an innovative technical evaluation approach to ensure VA and its customers awarded to the highest qualified vendor for a recent Salesforce Development effort which was looking to use User Centered Design (UCD), a methodology very new to VA. The details of how this came to be and our successes and challenges, follow.

The Problem

Typical technical evaluations focus on the end result of the Offeror’s technical approach, generally via a written technical proposal. However, as this project was to follow an agile methodology in accordance with VA’s Digital Transformation Center (DTC) policies, the exact requirements could not be determined up front. We needed to evaluate how a vendor would put the needs of the user first, and not an end state even we didn’t yet understand. After discussion between the procurement team at VA’s TAC and VA’s DTC, User-Centered Design was determined to be the answer.

The team, after further discussions with representatives from other Contracting Teams at the Department of Homeland Security’s (DHS) Procurement Innovation Lab (PIL) and DHS Digital Service, recognized the importance of UCD as a process that could meet our needs and which could act as the differentiator for the award. UCD is a commonly used term to describe the process of focusing on user’s needs throughout a development cycle including: defining a problem from the user’s perspective, designing a solution that prioritizes the user’s needs, testing the solution with real end-users throughout the development process, and defining success in terms of outcomes for users. The team contacted the USDS for technical advice to evaluate vendor UCD capability for the award.

The Solution

What worried the team was that it’s easy to sound like you understand UCD in a written proposal without actually knowing how, or being capable of, actually doing it. Thus we needed to come up with an evaluation strategy that allowed us to see the vendors actually perform UCD. From this need our UCD-Design Off was born. By utilizing a demonstration in lieu of a full written technical approach, we could make the Offerors show us they could do the work versus just telling us, increasing our confidence in their ability to successfully perform. The next challenge was determining what we actually needed to evaluate (and how).

Evaluate the Right Things 

After several discussions around the PWS requirements and the constraints of the in-person presentation format, the team came up with the following approach to evaluate UCD. This approach adapted the evaluation approaches by both the VA Appeals team and the DHS FLASH team to focus entirely on UCD capability and meet the limitations of an in-person demonstration. Focusing on UCD was also facilitated by the use of the GSA’s Salesforce BPA which provided a pre-vetted set of highly skilled firms specializing in Salesforce configuration and support. This removed the need to evaluate each Offeror with such scrutiny.

Design Challenge

Vendors were given a simple scenario or “use case” that was meant to test their innovative capabilities to respond to real situations faced at VA. They were given access to two VA employees (one playing the role of product owner and the other playing the role of end-user) to use at their own discretion throughout the scenario. Vendors were asked to create a prototype and provide any documentation created during the design process. 

Use Case

“VBA needs a way to manage system design documents. This process involves drafting documentation, collection and consolidation of feedback, and an approval chain.  Help VBA determine how to implement this process in Salesforce.”

Format

The in-person demonstrations had a few key characteristics that helped to set the boundaries on the UCD demonstration.

Evaluation Criteria

The following criteria were used to evaluate vendor performance during the in-person demonstrations as well as the resulting prototype. Vendors were provided with only the high-level bullets (like Visual Design) prior to the competition. The bullets below each factor were meant as guidelines for the evaluators to help round-out definitions so factors could be evaluated consistently, however, bullets were not meant to be evaluated individually and could cross from one area to the next.

Visual Design

User Research

Design Process

Content

Interaction Design

Conformance with DTC Standards

Bottom Line

The goal of the in-person UCD challenge was to determine if the vendors could, on short notice, assemble a team of competent designers, developers, and product owners which could demonstrate an agile, user-centered design approach to a simple problem.  

Evaluating this capability in-person avoided previous concerns expressed by Contracting Officers and Program Managers about vendors providing sound written proposals but lacking real depth in UCD methods and best practices. By reviewing a live design process, the evaluation team was able to quickly recognize how designers were integrated into the development process and whether vendors were following design and user research best practices. This approach also showcased the vendor’s ability to communicate, collaborate, and adapt their UCD process to fit the sample task on the fly. Review of the prototypes allowed the team to assess the vendor’s competence in producing prototypes from user research and further development iterations.

Lessons Learned

Task Order Details

Title: VBA Salesforce Center of Excellence

Total Task Order Value (Firm Fixed Price): $62,161,458.83

Period of Performance: 12 month Base plus four 12 month Option Periods and four Optional Tasks

Scope: The Contractor shall provide user centered design, Salesforce configuration and integration support for VBA including project management, requirements elaboration, design, configuration, content creation, testing support, and training support, for the configuration and enhancement of VBA Salesforce applications. The Contractor shall provide support throughout the lifecycle including compliance testing, deployment, operations and maintenance.

Milestones

Solicitation Issued: August 30, 2017

Proposals Received: September 8, 2017

Evaluations Complete: September 22, 2017

Award: September 26, 2017