Fail Of The Week: OpenMV Kickstarter Project Hits Manufacturing Snag

Making stuff is hard, especially when you are making lots of stuff. The OpenMV Cam project knows this, because it has hit a problem while putting together their cheap machine vision module. The problem is with the BGA solder balls that connect the image sensor to the main board.

openmv-thumbWe’ve covered this intriguing project before: the aim is to build a small, cheap module that can run image processing algorithms to easily give robots sight. The sensor is a Ball Grid Array (BGA) package, which means there are a grid of small solder balls on the back that form the electrical connections. It seems that some of these solder balls are oxidized, preventing them from melting and fusing properly with the board. This is called a head-in-pillow defect, because the ball behaves like your head when you lie down in bed. Your head squishes the pillow, but doesn’t merge into it. There are 38 balls on the OV26040 image sensor and even a single bad link means a failure.

The makers of the project have tried a number of solutions, but it seems that they may have to remake the ball links on the back of each sensor. That’s an expensive process: they say it will cost $7 for each, more than the actual sensor cost initially.

A few people have been posting suggestions in the comments for the project, including using solvents and changing the way the sensors are processed before mounting. We’d like to see them overcome this hurdle. Anybody have any suggestions to quickly and cost effectively move the manufacturing process forward?

Thanks to [eDgE] for the tip.

99 thoughts on “Fail Of The Week: OpenMV Kickstarter Project Hits Manufacturing Snag

        1. whats its frequency tho – it might be better than h264 – or be more versatile than your average piece of crap webcam. h264, given vga resolution should give you 600hz or something? and webcams are usually inconconveniently clamped at 60hz. (imagine doing object tracking with 600hz advantage, even tho its only low resolution.)

          1. GM8135S can capture 1080P30, but it can only compress at 720P30. It is 600Mhz ARM9 with dedicated h.264 compression hardware. Chip also has OTG, 100Mb Ethernet, PWM, i2c, i2s, spi, uart, etc.

            0.7W for 720p at 30fps

      1. That is a different camera using the HI3518. It is about $10 wholesale. It is free ship – $10 + $5 for shipping = $15. HI3518 is similar to GM8135S, but it costs about $6 compared to $3.5.

        You can buy the other one at the $6.40 advertised and then when you get to checkout taobao will add international shipping. International shipping will vary between $5-30 depending on what you choose. Note that at the $30/3day level you can probably ship 10 boards for that $30.

    1. Yeah but no-one’s going to fund your kickstarter to develop an API (or just write some English documentation) for something that already exists. More’s the pity, really, as there’s so many cases like this where the thing to do the job already exists, but discovering it and deciphering how to use it is “too hard”.

      1. Why would you not fund a Kickstarter for software? AFAIK there is no face recognition or robotics software available for these cheap security camera boards. I think it would fine to say – we are developing software for these boards, the perk you get will be a board with the software pre-loaded. Just don’t hide the fact that the boards already exist.

      2. I think you underestimate the convenience of having everything you need put in a box and mailed to you including working software and instructions on how to use it. You might be surprised at how well a cheap security camera repurposed for robotic use might sell. The HI3518E SDK was linked at the bottom of that nemon.org post. You can find the Grain SDK on baidu.com. Both of these SDKs are documented in English. Once you get a little further along you can get a direct developer at Hisilicon or Grain Media. That will cost you about $1,000 but you get factory support that speaks English.

      3. Having someone pretested and getting the right version of PCB is worth at least a 20% or so premium tagged onto the retail price. Quite often you’ll find OSS pointing to 6 month or more out of date hardware as no one else other than the original author bother to do the hard work again.

        1. Big difference, large companies have significant control over their Chinese supply chain. Small entities will be taken advantage of. I’ve heard this from numerous people that you really have to get to the right people there or they will find a way to screw you (old parts, faulty components, poor shipping standards, etc etc).

    1. So what? I have a camcorder from the 80’s with that still works with a through hole image sensor.

      Just because something works doesn’t mean something else can’t work better.

    2. Sorry, but dumb analogy, yes something that was manufactured and passed QC then was sold retail is going to work better than something that is in the prototyping phase and just found a hurdle in the manufacturing process. BGA has nothing to do with it, can just as easily happen to things like QFN and SOIC, the pins can be oxidized and not accept solder, bad parts are bad parts period.

          1. Interesting thought, a vacuum oven would be a really hard problem. First I imagine it would heat really unevenly with dark color areas getting hot while reflecive or light surfaces staying cool. Second without circulating air moving around to have an equalizing effect you would see parts get really out of control hot. Third it would be tough to implement a control loop, in a vacuum temperature would be harder to measure – maybe by thermal pasting a thermister directly to the surface of the board.

    1. There are inert atmosphere reflow chambers that purge the air the fill it with argon, a bit expensive, but works good with sensible chips, exposed dies and what note, and there is also vapour reflow that use something north of boiling fluorinert.

        1. I’ve done a batch of flex pcbs with small pitch DFNs in nitrogen, made a 7% increase in the yield. If you count to a couple of thousand 7% is a lot. And thankfully the contractor did not charge extra for this process. I thought that the next step in increasing both yield and reliability would be to add some underfill to a package that usually don’t get any underfill (usually only BGAs get them for both oxidation prevention and mechanical stresses), but it turns out that going the COB route seems smarter. I can get a wafer of 50k+ ICs for what I would get 2-3 reels of 3k.
          Anyways, my point is- nitrogen helps, use it!

  1. If the parts have recently been ordered they should probably contact the supplier.
    Also making sure to store the parts in a dry place seems like a good idea to prevent more problems.
    For now they could apply flux to the BGA area, that should remove the oxidation.

  2. When we had BGAs with head on pillow defects, we were able to just reflow the balls by putting the assembly under an BGA rework machine. The machine has a shroud that fits over the BGA, then heats the part through the manufacturer’s reflow curve via hot air. Sounds like that didn’t always work for them.

    I would be nervous taking the BGA off the board, reballing it, then reflowing it back on. I’ve seen plenty of projects here that do just that and everything is peachy, but we always use a new part because it wouldn’t meet our workmanship standards otherwise (then again, we build aircraft avionics, not consumer electronics). It might be totally fine, but I don’t know how scalable that is.

    If reballing is going to cost “far more” than the original device, why dont they just scrap the bad parts, go after the distributor for crappy handling process, and buy known good parts? That’s assuming that the $7 is close to double the cost of a new part.

    1. Yeah my first thought was pretty much the same. If it is going to cost more to repair the defective parts than it would to replace them. Why not just replace them with good parts and save yourself the headache of repairing them.

  3. 1) Don’t try to fix them. Scrap them out, and chalk this up as an expensive learning experience. Kickstarter backers deserve a (reasonably) reliable product, not one that’s been through extra thermal cycles and is likely to fail prematurely.
    2) As others have said, escalate to the manufacturer/distributor to find out whether the degree of oxidation on these balls is normal or excessive.
    3) Flux early, flux often. For normal SMT parts, the solder paste applied to the PCB provides adequate flux. In this case, they could go a step further and work with their assembler to add a flux dip for the image sensor immediately prior to placement on the board. I’ve seen that method used for package-on-package assemblies, but it would might also be applicable here.

    1. 1. Agreed, these chips were left out in the atmosphere too long and have oxidized
      2. “Normal” is none, given most competent suppliers will use a desiccant pouch in a vacuum sealed bag
      3. Although a thin flux will allow some units to reflow, they will get lower design yield rates from residual crud glued under the chip between pads

      Note: the BGA process usually needs x-ray inspection equipment. ;-)

    2. Replying to this strip of comments because it’s one of the few that actually shows knowledge of the process variables. Head in Pillow defects are common enough that industry experts have excellent processes in determining root causes, proper rework steps, etc.
      There are a few clues in the write up on the kickstarter page that make me believe they are on the right track for the root cause. Older parts, improperly stored, etc. The manufacturer should have already known that you can’t bake parts to remove oxidation.
      Some comments:
      1) Reflow with an aggressive, no-clean liquid flux on a good IR or hot air rework machine with an elevated profile. In some scenarios, if the alloy in the BGA balls that are oxidized become active enough once they become liquidous, they can coalesce with the solderpaste (or in this case, the solder from the solderpaste).
      2) I would request a full Root Cause Analysis by the manufacturer. Do they have proof of the reflow profile validation? Did they run a board through with a profiler?
      3) There are three metals involved in the reflow process, have those been reviewed for optimized processing? (the three metals are ENIG for the PCB plating, the solderpaste (SAC305?), and the balls on the BGA (SAC305?).
      4) The fact that the parts were stored improperly and yet still were used means the manufacturer has some gaps in their handling processes. Defects caused by ball oxidation can become costly (as seen here) so proper handling of MSL devices is critical.
      5) Any reputable manufacturer should know how to handle the issues of this product, so if the one that built these can’t…… well, there ya go……

  4. Maybe inject a very highly activated flux below the device and then try to reflow again? It may be able to eat away at the oxides.

    Make sure to contact the supplier of the device, it was probably mishandled somewhere!

  5. The flux is failing to break the surface tension of the heated oxidisation layer. Far more thermal energy is needed to break the surface tension originally than is required for the process to complete.

    A simple solution would be to very quickly go over the oxidised BGA with a soft backed emery paper. The slightest ‘scratch’ on a BGA ball will be a place where the surface of the oxidisation has been broken making it a hundred times easier for the flux to complete it’s process. You need soft backed backed emery paper so that it conforms to the differing ball heights and scratches every ball. This can be done very quickly as you only need only one scratch per ball, but every ball must be scratched.

    Doing this along with selecting a solder paste that exactly suits the thermal profile will optimise the result.

    There was a special material that would be perfect for this but I don’t know if it still made. I will try and find it in google.

    1. I had a similar thought, though maybe more like a really fine grit sandpaper, laid across a surface plate to help ensure parallelism across the peaks of the ball array. Anything that could lightly “rough up” the surface of each ball should go a long way to fix the pillowing issue. And as many others have mentioned already – FLUX THE HELL OUT OF IT!

      Of course that would be AFTER they take the component supplier/mfr to task over the matter. I don’t know how the parts were packaged, but I would think it would be hard to argue that the parts weren’t mishandled by the supplier or manufacturer if they’re coming straight off the reel with that oxidation present.

    2. The Micro Mesh is very similar. The next closest thing I could find was 3M Abrasive tapes – more like a stiff cloth than a tape.

      I was looking for a material that we used to clean the contacts of relays back in the days when telephone exchanges where electro-mechanical ie all relays and cross switches. It was rather unique and expensive.

      It consisted of a firm and stiff backing that held a mass of straight(ish) strands of plastic and on the end of each strand was a very small industrial diamond. The sticks were about 6mm wide, 450mm long and 4mm thick. If you pressed it between your fingers it would compress very little. There were thousands of these strands per square inch.

      Anyway It would be perfect because the strands were very stiff lengthways but could easily move from side to side so when it was passed over a raised contour it would tend to conform to the contour shape and clean the whole area instead of just the tip as you may expect.

    1. Hmmm. That suggests that oxidation of the solder balls in storage can’t easily cause this problem, and that a more common cause is the PCB or BGA chip warping during reflow so some solder balls aren’t touching and oxidise badly before they comes back into contact with the solder paste. Interesting. (Other contributing factors that they could reproduce were bad reflow profiles and poor-quality or badly stored solder paste.)

      1. Well, it can definitely help cause it, although it’s usually in conjunction with something else.

        My instinct would be low-Tg PCBs, since they’re *extremely* common from many of the cheap board houses now. Going to PCB Shopper and picking out the lowest prices will almost always get you Tg-140 FR4, and if you’ve got a high layer count, that really can warp the board during reflow.

  6. Vacuum ovens are actually pretty common in fields that need them (I used one as a physics undergrad) but they are not really designed for fast temperature profiles. On the other hand, inert gas or even mildly reducing gas would likely work fine, and controlled-atmosphere ovens are probably even more common than vacuum. But all of that is pointless because that’s exactly the kind of rework that costs money. Heck, even testing the things to verify that they’re fixed might cost more than it’s worth.

  7. There are thousands of instances where companies has thrown out 100’s of reels because the bag was opened and the one year had expired.

    It is nice to have the components on hand at your workbench, loading dock area, warehouse but rarely do the doors sit closed. If the place is too dry you have to worry about ESD… too humid… oxidation.

    The really high end shops use Argon displacement for the component storage… Yet I am puzzled as to why not use Nitrogen. Or even drop a bulk set of desiccant packets when you open the package. *shrug* Whatever.

      1. Hah. *tears of woe* A company my Dad worked for (procurement and inventory) had excess inventory of $15 million of those reels. The kicker is if you have a company even if you don’t have a product those components have book value via taxes on taxes.

        The biggest issue is that companies were sending out reels of components that were already opened and missing 1 meter or 2. That means the OEM/ChipFab company received a return and to ensure quick turn around skipped QA/conditioning and re-reeling.

        You get a nice certificate saying it was disposed of properly. That is stamped with a PRC seal. Insert smiley face.

        However they go to a even shadier shop.

        http://www.google.com/url?sa=i&source=imgres&cd=&ved=0CAYQjBwwAGoVChMIrYbxyor7yAIVBiQmCh227Qo-&url=http%3A%2F%2Fi2.cdn.turner.com%2Fcnnnext%2Fdam%2Fassets%2F130530163826-pkg-aoc-watson-china-e-waste-00001606-horizontal-large-gallery.jpg&psig=AFQjCNHpIDWlGQ_ttwKJTalPQh7-UOV_sg&ust=1446874744946256

        To keep the books balanced most companies have “junk floats”, where dead tech is stored in containers. These containers never pass customs as they are ALWAYS “In Transit” to one port to another, thus the property is not in their hands (off the books) but the hands of the transport carrier and insurance provides…ergo zero taxes to the next so called “MFG site” that exists but somehow closes before the container docks to port.

        Finally the chips land in the hands of either Yak, Triad (PRC Mil backed) or Tong hands. Where the company does a pawn shop deal to pretend those reels are still good.

        P.S. You would think a company like Marvell Electronics would at LEAST keep a data sheet or schematic for 3-4 years. Nope, They wanted $100k contract to reverse engineer their own chips as it was 2 revisions old.

        TL;DR – No those reels are dead and gone. If you have someone from a marketplace trying to push them? Walk away. Fast as hell.

        1. Marvell is some of the lowest quality chips and engineering on the planet. Honestly everyone is better off when you avoid Marvell at all costs no matter how cheap of an offer they present.

      1. CORRECT GOOD MAN!

        Most parts also have datasheets regarding MSL and “shelf life”… But sometimes through-wire and QFN techs can get away with it and don’t need to worry.

        BGA on a laminate foundation component? Worry! Worry like made. Xilinx clearly states bga to laminate bga (bcbsga) can only survive 2 reflows max.

  8. It would be great to hear other people’s experiences about dealing with China. Just buying hobby stuff, I have seen some shonky stuff. I have some Arduino – Mini-Pros that I suspect have an ATmega168 in them even though the chips are marked 328.

    I have received ICs that have had the parts number ground off and the parts numbers of higher speed parts put on in their place.

    Some parts have just been crap. Break the first time or don’t work as expected.

    I am not complaining, I take my chances and most of the time I come out well ahead. The shonky part is more often the exception.

    It would be nice to know some more ‘clues’ about parts from China. I now know to look at manufacturer dates and if there is one date that is ‘over used’ then it’s a fake. Also the Yellow easy to read part numbers on chips is *not* original.

    And these cameras …

    New Old Stock! with free oxidisation.

    I agree – sounds like some shonky seconds entered the process. They probably perfect except form the oxidisation. In any case it’s the manufacturers issue.

    1. Well, There is 2 cents worth of rote fact I can share.

      1.) Broker Houses – Just like stock for the stock market that handles stock trades there is a EPIC business of “who got what…when.where.how.” Fully Legit and Legal with no oversight. Basically, shorting and inflating components is what they do.

      2.) Samples –
      a.) Before 2008 – You could literally cold call a company say “start-up” innovative tech with 20 staff and you’d get 2.5x your staff in requested samples
      b.) Companies with market cap of over $5 mil had ZERO overhead in shipping/parts and datasheets (on the off chance you’d use their gear). In fact products at >$2500 were able to build 4-9 prototypes with samples alone.
      c.) The Kickstarter/Indiegogo/AVR-uino Das Blinken Lights crowd killed free samples (unless you are LLC, dot.com with 7 letters or less, crazy site page, and prove a hard bank certified balance sheet and 2 years of taxes for employees (granted if you have connections or contacts you might have some luck)).
      d.) Get in good or Work AT the local recycling center your boss might be cool if he takes a cut from your webstore of surplus. Granted it HAS to be a passive part.
      e.) Shark smile – If it’s a business meeting to talk about a item anyone who is smiling in the first 5 seconds to 5 minutes. Without even talking business… BEWARE.
      f.) Finally, Remember always. Western culture and public educations has taught us to assume nodding head Up and Down and saying “yes” means “I Understand”. The fuq-all-truth is nodding head up and down and saying “yes” means “I heard you.” It does not mean good hard workers willing to work or help with us.

      1. Any reputable company will not buy from brokers.

        A company I worked for did that ONCE, and ended up with an overpriced counterfeit. The broker wouldn’t take them back, either.
        China is the Wild West. Buy from legitimate distributors of take your chances.

    2. Orders from Aliexpress have automatic escrow on them. When you pay on Aliexpress, Alibaba holds the money and does not give it to the vendors. Only after you click “accept” on the web form does the vendor get paid. If you reject you can get a refund and at most you have to send the part back. If you say they are counterfeits I believe you need to mail them back to Alibaba. So you have a no risk window to verify the authenticity of the parts.

      Another hint – use some common sense. Do you really think $20 chips selling for $3 are the real thing? Do you think a $100 Gucci bag is a real $3,000 one?

      1. +1 gazillion
        My main reason for disliking people who complain that their latest $5 buy from a Chinese eBayer isn’t what they expected. “You get what you pay for” is not just a saying, it’s a fact.

  9. Without knowing the full story about supply chain, storage conditions, time between delivery and manufacture of the parts and all the process details it’s almost impossible to tell who is to blame here. There are some good ideas for “workarounds” to get the oxidized bad balls to maybe solder, but in the end, you have to find who/where screwed up in the supply and manufacture chain to remove that bad link. If you don’t do this investigation properly you will always run in behind and try to fix stuff that should never have happened in the first place.
    Reballing can be expensive, but i guess the question is, if you reorder new parts, do you get a guarantee that they are good this time? And how long does it take for the new parts to arrive (stocked parts, or manufactured to order)?

  10. I don’t have a solution, but I do happen to work with xray PCB inspection machines that are designed to catch these defects. I would be happy to run some PCBs through our machine if it would help…

  11. Hey folks, if this has not been resolved please reply to this post. Contact me at DfR Solutions and I’ll tell you some easy things to fix it. I do this type of consulting all the time. And being a crowdsource I don’t mind lending a hand. Hit the contact us form with the head on pillow defect and the sales guys will send it over to me. I’ll need a bit more info such as datasheet, board layer and Cu content info, solder ball and paste type, and reflow profile. You could probably put a rework process together for a hundred bucks or so.

  12. Total hacker / novice suggestion:

    1) Find a tooltip to thread the hole.

    2) Tap said hole with tooltip.

    3) Find very small screws to finish up the job.

    Really, I am a completely hobbyist and novice to this complex problem. This is just my mechanical mind looking for the path of least resistance. I could be WAY off.

  13. A guy from Denmark produced the v3 boards 7-8 months ago, as I recall it he did not have issues with the BGA soldering. I bought a few boards of ebay, and they still works fine.

  14. These guys are my bible:
    http://www.circuitinsight.com/programs/50687.html
    http://www.circuitinsight.com/programs/53362.html
    http://www.circuitinsight.com/programs/50798.html
    http://www.circuitinsight.com/programs/50733.html

    Check out all of those white-papers.
    Head and pillow defects can be caused by a number of things, improper flux, bad solder paste, bad profile, warping parts, and more.

    Oxidized parts is a small component of a much larger issue. Even if the package has been opened for months the parts can still be usable with the right process. There is specialized equipment that can be used to remove oxidization from parts and boards using chemicals or plasma. Mechanical methods, like an ESD safe brush can be used to remove much of the oxidization. Of course, you should follow standards, like baking the parts prior to rework to remove any trapped moisture.

    Failed boards can be recovered most of the time. A good process by use of a rework machine will allow you to reflow the package and achieve acceptable results. You can flood the BGA with a good flux, then reflow on a higher temperature profile. This is assuming that the part is not warped too much (a part once cooled after warping, will never regain its original shape).
    After the boards are reworked, I would recommend a good underfill to give mechanical reliability.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.