Do you run crowdsourcing projects yourself or do you collaborate with partners?
We collaborate with partners for pretty much everything we do. Even for our internal platform, there’s a vendor who owns the site and helps us update the platform and figure out problems. We collaborate with platforms like InnoCentive, Topcoder, Kaggle, Luminary Labs, HeroX. Right now, we have ten vendors; they all have different skills, and often self-select depending on what they know best. Once a vendor is selected, they do most of the work and this is a really good selling-point for the problem owners. The vendors help frame the problem, develop the contest and help execute it.
Who selects and evaluates the ideas and solutions? Do you delegate this to the vendors as well?
The vendors help select ideas by evaluating them against the defined requirements. Therefore, you have to make sure that your requirements are clear enough to pick different solutions. For an algorithm challenge, it’s mostly easy. You’re almost always just picking the highest score. But for ideation or creative-type challenges, it’s more difficult. Our contracts specify that vendors filter the ideas. We’ve had challenges with hundreds of responses from the general public, with each response up to 30 pages talking about something in space, and many nowhere near what we want. Even in a curated crowd, many responses are not good. The vendors are reading all the papers and identify which meet the requirements. Then they deliver those to our NASA teams and problem owners only have to read through those to pick a winner.
How do you motivate contributors to participate in your challenges?
For a lot of the challenges, that’s really up to the vendors. They understand their crowd and it’s their job to maintain and curate that crowd and keep people interested. That’s part of what we pay them to do. They’ll know how much money it takes to get the right kind of answers. If you offer too much money, the crowd might think the problem is too hard and their solution won´t be good enough. If you offer too little, then people might think it’s not worth their time.
What are typical problems that can arise during a project and how can they be solved?
Probably the biggest problems emerge when the challenge-owning team is not ready to take the solution. For instance, if the solution is a software application and they haven’t, ahead of time, coordinated the implementation with their IT. If people haven’t done all of the leg-work, you get a problem. The people in-house need to be able to integrate the new ideas into their platform and their architecture. People often are afraid of the idea of outsourcing work, but really there’s still lots of work to do even if you’re handed a solution.
Are there any problems with intellectual property issues? Don´t you fear giving away secrets that people would use for their own purpose?
Together with the vendors we define what IP (intellectual property) we want to give the solvers as part of the challenge. Usually, the more IP you’re willing to give, the better the solutions will be, especially for hard problems. If respondents are pretty much inventing something new, they might back out part way through and say,” I don’t want to win this prize. I’m going to go start my own company.” So you have to be careful on how much IP you’re willing to give. Generally, because we’re the government, we give a government use license where they’ll continue to give us, in perpetuity usually, the ability to use whatever that thing is, but we still allow them to go start a business on the side, if they want. For private industry it will probably be a little more complicated.
Data might be too sensitive to be shared. How do you handle this challenge?
For a data science type problem, you can change the labelling of some data or share only part of your dataset, to make it unclear. For instance, if we’re doing a challenge on astronaut health, we can’t share health data. So we make sure that we have just columns of numbers or we scale data differently. There are lots of things you can do to prevent people from interpreting exactly what it was originally.
Do you always reveal that it´s NASA that seeks a solution or does it sometimes make sense not to disclose who the sponsor is?
Obfuscating a problem can be very good in some cases. If we reframe the problem to disguise who we are or what the problem is, it gets hard for folks to know what’s really going on. A few years back, the CIA actually ran a challenge without listing themselves as the CIA. The challenge was to utilize only social media posts to track particular bison in Yellowstone National Park. But in fact they wanted to use their algorithm to help track Russian actors in Crimea, and were finally able to do so. They didn’t post it as that to avoid attracting bad actors submitting bad responses that might change the outcome. Getting people help you find bison in Yellowstone National Park is the same concept and really harmless. The vendors are very good at helping folks like me reframe and restructure those problems so that we need not worry about leaking too much intellectual property or sensitive data.