As a nebulously-defined “Hackaday advisory judge”, I want to try out advising folks on their Hackaday Prize entries. On our podcast, I offered to take a look at a project and score it as I would if I were doing the contest judging. William offered up his last-year’s USB Tester as an example project.

Note: last year around this time, I wrote about how to win the Hackaday Prize on element14. That may be a better introduction to all this as I’m going to leap to details now.

If I had judged his project last year (I don’t think I did), William would have received comments along these lines:

Good job making a useful tool! I recommend using a single github repo for everything to make your project simpler to follow, and don’t put other versions in the repo. That may be useful for your business but not for easy reproduction. It looks like you’ve got a model enclosure: that’s great, I hope you finish and document that in your entry. Add step-by-step build instructions for both HW and SW. A BOM would be nice in github. Please make a case for how USB Tester is relevant to the contest theme. Finally, thank you for making it open source though license files are much easier to see than notes in readmes.

My goal with those comments was to give William some actionable things that would improve his scores. I wanted to give him relatively easy things to do, on the order of hours of work, not weeks. The hard part is that I didn’t score the USB Tester all that highly. My job as judge is to compare the project to the rules, not to decide some nebulous goodness. As a person, I want those comments to encourage William. He did this as a volunteer, I want the Hackaday Prize to be a good and useful experience for him.

But what about his score? How did I get to those comments? Since this is my first open-judging, I’ll try to lay everything open. I hope it comes out all right.

There are two things to know before I look more at the project. First, I was not the only judge. While my scores were reasonably in line with others, they didn’t match exactly; don’t expect my scores to match what you get from this year’s Hackaday Prize judging panel. Second, I’m a mean, mean grader. While the range is 0-10, I like my scores to be low so standouts have lots of room to stand out. Here are my average scores, judging all of the 2014 semifinalists and my third of the the 2015 semifinalists.

NOTE: No one’s average score was higher than a nine. Not even the ones I really, really liked. I am a very mean grader. Keep that in mind.

Here are the average scores over each of the judging criteria:

The 2014 and 2015 Hackaday Prizes had different criteria so not everything has two bars. The 2016 Prize also has slightly different criteria. For one thing, to get to the final round, you have to be in the top 20 of the preliminary challenges so play early and often. The Rules page has the details but the judging criteria seem a little wonky today so I’m going to use the 2015 criteria for William. They are:

• How “Open” is the design? (Open)
• “Wow” factor: is the entry innovative, is the build impressive? (Wow factor)
• Does the entry address a wide-ranging problem? (Address problems)
• Is the project reproducible and could the work be extended for other uses? (Reproducible / extensible)
• Is the entry innovative? (Innovative)
• Is the entry usable in the real world? (Real world usable)
• Will others want to continue perfecting on this well-documented idea? (Well documented)

There is a lot of overlap between these categories: reproducible, open, and well-documented all required similar artifacts from the entries. Sadly, this is one area that people didn’t necessarily understand, especially last year.  While they are separate things, the question to ask yourself when you enter your project: can a motivated high school student rebuild this?

If you want to see crazy-good scores on these categories, look at Raman Pi and SatNOGs where the answer to that was “yes!” Those are well-documented, leading you through the process of reproducing the build. And everything is open so you can get through it. This type of technical writing is very difficult to do well. On the other hand, these are the simplest set of points to acquire.

The innovative and wow factor criteria have a lot of overlap as well; though it is possible to have a great, innovative idea that doesn’t make me excited. For me, SatNOGs was more innovative in their approach but Raman Pi was more wow-y, partially because I love spectrometry, partially because the creator was so obviously enthused. (I should stop picking on those two.) Overall, though, innovative and wow factor tracked each other well.

Ok! So! Criteria set, let’s see how the USB Tester would do. First, look at the relevant Hackday.io page. Before judging, the Hackaday folks laboriously went through each entry to make sure they conformed to the rules: verified that entry had correct number of project logs, found the video link, etc. I’m not going to worry about those, they are auto-elimination things. I’m going to stick with the 0-10 scorables. (The Hackaday folks did the same scoring to narrow the entries from nearly all entries to the semifinalists.)

Now, before we go on, let’s thank William for being brave. My role as judge is to be critical. I’m looking for flaws and trying to weigh those flaws into scores. As long as I’m consistent in my criteria, I hope that I’m being fair though I often feel that I’m being mean. That is why no one before has gotten to see the inner workings… they may be too critical and not constructive enough. But I showed a draft of this to William, he thinks it may help others, so he agreed that I can share it here.

Let’s start with the terrible news. Innovative and wow-factor: these both look like they are going to be somewhat low scores. There is a v1 in the github directory, and this project is a v2. This weakens the newness score, since the project isn’t quite new. I have a Dave Jones uCurrent on my desk and two DMMs. But, most importantly, I have an Adafruit USB power monitor on my desk. (And yes, my desk is an absolute mess right now.) With all of this, what is innovative about this idea, design, and build? I’m going to need some help figuring this out… hopefully the entry will have it eventually.

Well documented: in the first paragraph of the details section I got thoroughly lost trying to figure out what this entry is about:

The USB Tester consists of two parts. The USB Tester Base which has the USB connectors and test points for use with your own DMM. Then the backpack which has the OLED display, microcontroller and INA219B for sensing voltage/current. The backpack can be connected to either the USB Tester or VA Tester. Then the Bluetooth Backpack can be connected to either base as well. One note is when the backpack is used with the VA Tester it will need its own power since there is no onboard regulator for the 5V supply.

There are two parts… the USB Tester Base and the backpack. Ok. But what is the VA tester? And does the Bluetooth Backpack make for a third or fourth part? And the power supply… is a fifth? Now, I care a lot about writing. I try to be more appreciative of good writing than critical of not-so-good writing. But I have a degree in engineering, I am a native English speaker, and I have read that paragraph about six times now; I have no idea what it means. Maybe there is a tester base (USB or voltage/current) and a tester interface (display or BLE)? This doesn’t mean well-documented gets a low-score necessarily, but this was a demerit. Some advice here: read your writing aloud. I think William might have figured out for himself that this paragraph got off-topic if he’d heard it.

There were three repos, all with similar names. The hardware one didn’t have any code, only schematics and gerbers (Yay! Gerbers! Points for open-ness!). The OLED backpack had code.  Its readme is quite confusing about how to do the build. Steps would be nice. It starts out listing things, different libraries, different versions. I suspect some of these are for different versions of the entry. Finally I get to instructions, “Download ZIP in each version folder for a working set with libraries and configuration of hardware SPI and display size”. Now, I actually know what this means because I’ve used Arduino IDEs and derivatives a lot. But these are not sufficient instructions for a high school student, a software engineer, or a hardware engineer. Only someone with a peculiar set of skills will recognize what you are saying. (The Java App repo is a little better on this though it is still notes for the developer, not instructions for a newbie.)

So does this really matter? Yes. Very much, because well-documented and reproducible are both criteria. A step-by-step guide to building would give points in both categories.

I’m moving on to whether it is real world usable since that’s mostly a yes. I have used similar things and found them useful, and this looks like a reasonable approach.  An enclosure would help it survive the rigors of either my desk or the field. That looks like it is in the plan (model) but not in the project as written. Also, no-burden low-current monitoring would be even better since I do work in micro- and nano-amps.

The Hackaday Prize has shifted from build something connected to build something that matters. The goal is a humanitarian project. Power is a huge deal. I get that. But this is a tool, a relatively niche tool. So the USB Tester isn’t saving the world; it isn’t even making a huge difference to a few people. The entry will need a whole lot of spin to make it seem more relevant to the contest. I saw such spin in other entries. It was … ok. Better to have the spin than not.

The open-ness of the project was good. There wasn’t anything that I couldn’t find. On the other hand, I had to read the readme or the Hackaday entry to get the license file. If one of the other judges missed that, they’d have to deduct automatically. A license file would be optimal.

I went through more of the project, watched the videos, looked at the linked sites (being sold, no added or subtracted points for that, just interesting; also, his Fried Circuits had more and better user docs), flipped through comments and project logs. There was a little too much telling me about the future plan and current issues but that’s really common. The goal is more to give current state and make it sound awesome.

Well, I bet you want to see the scores I’m giving for USB Tester. Did I mention that I’m a very harsh scorer?

How “Open” is the design?  8

“Wow” factor: is the entry innovative, is the build impressive?  1

Does the entry address a wide-ranging problem?  2

Is the project reproducible and could the work be extended for other uses?   2

Is the entry innovative?  1

Is the entry usable in the real world?  4

Will others want to continue perfecting on this well-documented idea?   2

Total Score     2.9

So, yeah. I feel sorta bad putting that up there. I want to make sure everyone understands that I think this is a good project, it is useful, it is interesting, and I bet William learned things. That’s all great stuff. I wish the best for William in selling his USB Tester. But I don’t think it is a contender given the goals and rules of Hackaday Prize 2015.