Product Update - December 2017
Welcome to our blog! In the upcoming weeks you’ll find some great content made by our team. Take a peek around our website, and let us know what you think!
Jacob Flood, December 15, 2017
Production is coming along well – we are assembling our engineering prototypes in the next couple weeks. We’ll send out pictures when we have some.
The update below is a series of stories, case studies, and examples of the complexities we’ve been dealing with over the past few months. Hopefully you get a laugh out of it all, and learn a little about the hardware design process!
Production is coming along well!
November has been a crazy month – we’ve been working night and day to get our engineering prototype ready in time for CES. We’ve made a lot of good progress and we’re still on schedule.
We will be assembling our engineering prototypes this week and next (and likely, continuing to work on them through the holidays). As a result, the timing for this update doesn’t align well to show you the really awesome pictures of the assembled units. We’ll post another “in-between” update with those pictures in a few weeks once the pictures are ready.
For this update instead, we’re going to focus on some of the challenges we’ve been facing in the past few months of moving from prototype to mass manufacturing. Hardware manufacturing is a ubiquitous process - every product you’ve ever used, at some point went through the difficulties of moving from prototype to production. In this update, we’ll shed some light on the process: the constraints we’ve discovered, how they affect the product design, and how we overcame them.
In our July update, we walked through the 6-step design process: proof of concept, engineering prototype, EVT, DVT, PVT, and mass production. Once we got our manufacturer onboard and working in September, we spent the next three months moving from proof of concept to engineering prototype.
This is by far the most difficult part of manufacturing: turning a rough works-like and a beautiful looks-like into a single, manufacturable product. This involves setting the product requirements, designing the mechanical structure, finalizing the electronics designs, figuring out how each piece will be made (a process called design for manufacturing), and how it will all be assembled together (called design for assembly). Depending on the skill of the team, the complexity of the design, and a million other unpredictable factors, this usually takes anywhere from 1 month to 12 months.
The units we’ll be demonstrating at CES are our engineering prototypes. These are on-spec, fully manufacturable units, identical in every way to the ones that you’ll be receiving in April (yay!!). The next step requires going from a single, handcrafted unit, to an assembly line developing thousands.
The challenges you read about below are what we’ve been dealing with over the past 3 months of moving from proof of concept to engineering prototype.
1 – SPEED, COST, QUALITY: PICK TWO
One of the defining elements of mass-manufactured hardware development is that it rarely takes place in your own country. For us, moving to China was a no-brainer – the quality, speed, and cost of headphones manufacturer in the tier-1 factories in Shenzhen are orders of magnitude higher than any other country.
This isn’t without it’s complexities however. It has been a challenge to manufacture and source very high-quality components as we are not as familiar with local suppliers when compared to North American suppliers.
In addition, while the production of electronics is quite cheap over here, the specifications need to be that much more strict to ensure the proper functionality. First iteration circuit board almost always have unforeseen issues, resolved by making the requirement more stringent.
Failure to provide proper requirement results in wasted time debugging, trying to figure out why the behavior of the electronics don’t match expectations. Sourcing mission-critical components need to be done carefully, to ensure that you’re getting high quality components rather than copies. Testing components also poses a challenge, as we need to design circuitry to test the components themselves – an interesting catch-22, if you can’t trust the quality of your test-boards.
All of these issues can be resolved by paying more for components, which is a dangerous cycle to get into.
Our solution has been to separate sensitive components from generic ones, and moving the sensitive development back to Canada. This way we can leverage the quality and consistency of chip development in America, at the same time as the speed, cost, and know-how that our manufacturer is known for. This best-of-both-worlds scenario is more difficult to coordinate, but results in a faster iteration cycle and less mistakes – at the end of the day, that’s what matters most.
2 – SCHEDULING WORMHOLES
One of the biggest advantages that startups have over large companies is speed – a small team can make decisions, test ideas, and iterate quicker than any big company. That speed comes at the cost of systematic, process-driven development. Unfortunately, we learned very quickly that this tradeoff doesn’t play nice in the game of mass manufacturing.
Over such a large project, having too loose of a schedule leads to less work being done in the beginning, and most of the work being packed tightly right before a big milestone. Mistakes inevitably happen, this leads to delays.
What we learned to do is to establish more numerous, smaller milestones. This let’s us put a small, frequent amount of pressure to achieve each milestone, as opposed to a large amount of pressure right before a big milestone. This alleviates stress on us, on our manufacturers, and ultimately better sanity for everyone.
This transition from startup speed to manufacturing speed has also made planning significantly more important. While we have been lucky to find great manufacturers, who are very responsive and pro-active, there have been times when events beyond anyone’s control have interfered with our timeline. In manufacturing world, processes take a fixed amount of time – ensuring that quoting, approval and supplier lead-times all happen in parallel with multiple suppliers, planning is required. Setting much more rigorous, day-by-day schedules on each parallel operation before starting a milestone has saved us from multi-week delays several times in the past month!
3- RESOURCE COMPETITION
While we talk often about the benefits of working with a large, experienced manufacturer, the reality is that there are a lot of tradeoffs compared to smaller partners. The main one, we’ve noticed, is competition for resources.
Compared to some of the bigger headphone brands, Mindset is still over 100x smaller. This means that, when push comes to shove on the factory’s end, we’re often the ones taking the delays. This happened several times, before we realized our main advantage: speed, and presence.
Large companies primarily communicate through email with their factories. As a result, they’re slowed down by the 12-hour time difference from America to China – they only get one opportunity per day to send or receive messages from their suppliers. Since our team is in Shenzhen, we can talk to our factory all day, every day.
We’ve been leveraging that benefit as much as possible, often spending several entire days at the factory each week. As the saying goes, out of sight, out of mind – our presence at the factory puts us first on the communication list when any problems come up. As such, we’ve been able to maintain our fast-paced development, despite our small size!
4 – MULTIDISCIPLINARY DEVELOPMENT
While we were prototyping, most development was discretizable – David worked on hardware, Chris on electronics, Warren on software, etc. This way, we could move fast, with minimal wait time – each person had their work, and we did so efficiently.
We were able to work like this by following an “interface” model of development: define the interface between two people’s work, and then let them handle the other side separately. Chris could work on the data stack with the understanding that he needed to feed 5 channels of digital, 3-byte EEG data through the Bluetooth module, and Warren could work on the software with the understanding that he’d be receiving exactly that same data into the computer. The interface between the two, if it was well-defined, didn’t need to be addressed by either party for now.
Moving from prototyping, we realized very quickly that this model no longer fits – the entire design becomes one big interface. The electronics need to fit inside the mechanical frame. The software API needs to read from the Bluetooth chip correctly. The power supply needs to be isolated from the EEG signal both spatially and electrically. All of these interfaces became snag points that could potentially cause delays.
Moreover, certain components were multidisciplinary by nature. The best example was the EEG signal stack, which presented a unique problem:
To develop electronics, you need the final electrode. To develop the final electrode, you need the final electronics.
This chicken and egg problem made it impossible to assign final specs to either component. Many times, we would have finished one part of the stack, and then realize that it was incompatible with the other half. The iteration process to converge towards our final design was unpredictable, and what we’re now calling startup speed – our ability to iterate very quickly, because we’re a small team – was the only thing that saved the day.
5 – FIGHTING FOR SIGNAL QUALITY
Following on the theme of competing interests, achieving a high signal quality required countless tradeoffs, each more difficult than the last.
The first step to a quality signal is a quality contact between the electrode and your head. From a signal perspective, the best thing to do would be to nail a spike through your scalp – I think we can all agree that this isn’t ideal.
On the flipside, a lighter, more comfortable contact directly implies worse signal quality. Many conductive rubber electrodes we tried faced this problem – while they were soft, they didn’t collect anywhere near the quality EEG we needed. Somewhere between a soft, useless electrode, and a rigid, high quality electrode, there exists a balance that we were aiming to hit.
The main areas for improvement are the interface between electrode material and the skin, and the method to penetrate through the hair and touch the skin. For the type of electrodes being developed, the closer we are to the source of the signal (i.e. the brain), the better – consistently contacting the skin, without movement or undue pressure, is critical.
The solution, once again, was quick iteration - we went through dozens of shapes, sizes, materials, stiffnesses, and even colors, before arriving at the final versions. The picture below shows two of our production electrodes – removable, retractable, comfortable, and very high quality. We’re really happy with how they turned out.
In order to not “waste” the signal quality from the electrodes, we’ve also spent a lot of time optimizing the downstream electronics – amplifiers, filters, shielded wires, ground plan isolation, and a million other features to make sure that the signal remains intact all the way to the computer.
Our headphones have many more wires than other regular headphones. You should see the face of our factory development team when we explained how many cables need to pass – it’s priceless.
In addition to the number of cables, the kind of cables that need to be used is important. Due to the nature of the signal being recorded, we need cables with very specific electrical properties – unfortunately, in this case larger wires are better. As a result, it becomes a challenge to fit many more cables, as large as possible, without interfering with the extension of the arms, the cup adjustments, and the mechanism that adapts to different head curvatures. It’s all been a huge headache.
The solution ended up being multifaceted - we worked with a sourcing partner to find some cables that fit our specification with a smaller-than-typical size, and we moved some components around (in particular: the I/O ports at the bottom of the headphones) to minimize the number of wires travelling through the headband. As of now, everything fits – we’ll have some cool pictures of it all in the coming weeks as we fully assemble the engineering prototypes.
6 – THE ULTIMATE BATTLE: HARDWARE VS SOFTWARE
Many people warned us against starting a hardware company. Now we’ve realized why.
An interesting challenge has been coordinating between software and product development cycles. The software team needs to be constantly collecting new data with the latest prototype, in order to develop the machine learning algorithms. The problem is that making prototypes isn’t very easy.
We end up spending considerable amount of time working on prototypes that are replaced, or that simply fall apart due to their prototype-nature. The product team is therefore constantly navigating between:
1) Working with manufacturers to design the headphone parts of the product
2) Finishing development of the EEG parts of the product internally
3) Develop temporary prototypes for software team to acquire data with
4) Maintaining, debugging, and improving the prototypes as they inevitably break
This tradeoff became more and more apparent as we made iterations of the electrodes and electronics – every time we get a new set, the software team wanted to integrate them into the data acquisition prototypes. By extension however, that would mean spending half a week assembling a test prototype that adds no value towards final production, and scrapping all of the data collected through the old prototype (since the signal changes dramatically)
Setting inter-department milestones became the best way to navigate this struggle: every few weeks when a milestone was achieved in the EEG stack, a new prototype would be made. The software team would then have the challenge of collecting new data, analyzing it, improving the architecture from the results, and prepping for the new prototype before the next milestone was hit. This constant chase between hardware and software kept everyone looking ahead, and made navigating the interface between teams much smoother.
7 – SIZE VS SPEED
Our manufacturer is one of the best headphone producers in the world. Part of the reason they’re so good, is that they’re very well optimized for scale.
When it comes to prototyping, however, this plays against us – large size necessarily implies slower speed. When we would produce only a handful of units of a given prototype, the lead times on components would often be exorbitantly high – on the order of 1-2 weeks. This makes quick iteration nigh on impossible.
For PCB production in particular, this wouldn’t do. We ended up working with a smaller manufacturer to produce small-batch orders of components when necessary: design them with the big player, and produce them with the smaller one. The end result is quicker, while maintaining the same design quality and expertise that we get from the bigger manufacturer. It took us several months to come to these conclusions, but everything has sped up dramatically since!
8 – WORKING IN THE WILD
In the wise words of Steve Blank, no plan survives first contact with customers. We’ve been testing our prototypes extensively to avoid that fate.
One way to approach the user experience testing is to test the system in decreasingly controlled environments — start with extremely controlled (controlled task, environment, and instructions), then move to somewhat controlled (controlled instructions, uncontrolled task and environment), to in the wild (completely uncontrolled – hand them the prototype, and let the magic happen). We’ve officially moved to somewhat controlled.
Experientially, this means people are now using the prototypes as-intended, completely independently. This transition was a HUGE hurdle – we would often run experiments, and halt them midway, realizing that a bug, flaw, or unforeseen factor had ruined the data. The number of testers that have run our experiment just to realize that the app had crashed midway is beyond comprehension.
For iteration purposes, this was fine - as we move closer to production, this simply won’t do. Rather than reacting to bugs as they come up, we’ve moved to a more rigorous framework, spending time developing the architecture that underlies the app.
Building on the development framework that companies like Slack and Spotify use, while adding-in the heavy-weight machine learning number crunching scripts to perform our AI in real-time — without bulking up the app too much – has definitely been a learning process. Moving forward however, this proactive development approach is much more sustainable, and will significantly improve our ability to build on top of existing features moving forward.
We’re on schedule to get the fully packaged, user-ready app by CES!
9 – PROTOTYPES MISSING FEATURES
In order to avoid delays, we often substitute features in our prototypes that are incomplete, but “good enough”.
The primary target for this was the electrodes. As the product team put together iterations of the mechanical shape of the electrodes (optimizing between comfort and signal quality), we would sometimes substitute medical-grade dry electrodes into our prototype. That way, we could collect data and iterate on the software without worrying about the non-final electrodes being the limiting factor in the stack.
Unfortunately, that backfired for us. We used Silver-chloride electrodes (which acquire great signal, but corrode quickly) beyond their expected lifespan, without realizing that they were no longer good. As a result, we wasted an entire data acquisition session with a dozen people – we had to scrap the data, and restart completely. The frustration is real.
Similarly, the medical electrodes don’t play very nice – since signal quality is their only concern, comfort takes a back seat. The small, rigid legs of the electrodes on your scalp, without any of the mechanism we designed to distribute and alleviate the pressure, is uncomfortable to say the least. Shout out to the community at HAX that has helped us test the prototypes – we owe you one!
Having transitioned to our production-ready electrodes, this problem is behind us. We are now collecting data at maximum capacity, testing the prototype on 5 to 10 people every day. All is well that ends well!
10 – ALL THE LOVELY HOLIDAY FACES
Unfortunately, overcoming one challenge only leads to the next. With a better prototype and more data, means more work sifting through the data.
To be able to give our classifier good training data we decided to start filming the people we collect data on. We film their faces and laptop so we can begin labelling focused and distracted periods. The unfortunate implication is we have to go manually go through hours of data to label it appropriately. Believe me – this is not the most exciting TV show.
Given that we will be heading home for the holidays in a week, we're taking advantage of our time in Shenzhen to collect as much data as possible. That means - you guessed it - we're going to spend our holidays staring at videos of people working. Best. Holiday. Ever.
That’s all folks!
Hopefully this update gave some insight into the good, the bad, and the ugly of hardware manufacturing. Contrary to popular belief, building an incredible product is not all sunshine and rainbows. Despite these issues, we’re still on track to produce our engineering prototype for CES, move to tooling in the new year, and ship our units in April.
With a new year around the corner, we’re more excited than ever – each step we take towards production builds our confidence that Mindset will be, hands down, the best neurotech device every brought to market. Speaking to people all around the world about the vision of what Mindset will turn to, and hearing your comments about how you intend to integrate Mindset into your work makes all of the sacrifices worth it.
From all of us in the Mindset team, we want to wish everyone a happy holiday season – here’s to an amazing 2018 filled with fun, happiness, and incredible focus!
- The Mindset Team