I have a pretty large GitHub repository, with all of my boards open-sourced there. Now, I’m finally facing the major problem it has – it can be uncomfortable for others to work with. I don’t store Gerber files in the repository because that will interfere with how Git functions – you’re supposed to only have source files in the repo. Yet, when someone needs Gerbers for my PCB, or a schematic PDF, or just to see how the board looks before they clone the entire repository, I often don’t have a good option for them.
In my experience as a hacker, being able to find others’ PCBs on GitHub is simply wonderful, but a PCB repository without a README feels barren, and a PCB README without pictures makes me sad. On the other hand, not having these files autogenerate is uncomfortable – updating a picture every time is a major drawback in particular.
Let’s take a look at some KiCad Git integrations, and see what they have to offer.
kicad_cli
We’ve mentioned kicad_cli back when KiCad 7 got released, and in the recently released KiCad 8, it’s only become more powerful. Before, it could do gerbers and schematic PDFs, but now, it can even do DRC checks – which is ideal if you want to add a hook for any pull requests you might encounter.
The benefit of kicad_cli is that it’s extremely simple to add – all you need to do is to install KiCad from a PPA in your workflow, then run kicad_cli on your files. How quick? Well, here’s a KiCad makefile that lets you run make pcb in your commandline – you don’t even have to open KiCad anymore if all you want is Gerbers! It’s also reasonably quick.
However, there are limitations. For one, it cannot generate images – in KiCAD v8, it has gained the ability to export 3D models, but those have neither copper nor silkscreen. (Update: kicad_cli now has image generation support, read more at the end of the article!) Plus, kicad_cli is quite limited in which options it supports – if you want to tweak 3D model export settings, you might just find that the necessary checkbox isn’t exported as commandline option.
So, for my own purposes, pure kicad_cli won’t do – I want people to have board images, preferably, the same way they’d get one if they were to open a 3D viewer. Also, I want to be able to customize the output. Thankfully, there’s a KiCad integration juggernaut we can use, that you might even have already seen.
KiKit
You might have seen KiKit be used for panelization as PCB editor plugin with a nice toolbar button, but did you know it can also generate manufacturer-tailored files for you? Or that you can run it from commandline? Or that it can even help you generate board images, either SVG, or exported 3D viewer screenshots?
I first realized that KiKit could be used for integration when I stumbled upon a tweet showing how the epdiy project makes its KiCad outputs accessible on an autogenerated webpage – generating gerber files, schematic PDF, and even exporting the STEP model. This looked ideal, and I started looking into what’s up. I’ve found great things, and horrible things, let’s start with the great ones.
This is how simple it is to generate things automatically for your repository – it’s a ready-to-go GitHub workflow. You can likely get the relevant workflow files from this repo into your repo right now and have nice exports automatically generated with every commit within an hour’s time. Gerbers are wonderful, so are PDFs and STEPs, but in particular, again, I’m interested in images – those must be present, and they’re the most pain to regenerate.
Pcbdraw
There are two ways to generate images that everyone’s been using until now, both of them are through a Pcbdraw tool written by KiKit’s creator. One of them is more diagram-ish but requires some effort on your part, and another way creates a perfect 3D viewer render .jpeg but requires a blood sacrifice. Let’s start with the first one, the pcbdraw plot method.
This rendering method creates an SVG file for your board, complete with silkscreen, traces and pads – way nicer than digitizing a 3D model from kicad_cli output. However, any components you might expect, will be missing – you need to have SVGs for your components too, they won’t just be present. There are SVG libraries, but in general, expect yourself to have to work in Inkscape if you want the diagram to look anything like the most bare board you get from the factory.
However, I have to say that you must not sway from the path of light and you really should consider investing effort into pcbdraw plot generation. For one, it’s very nice – SVGs are low-footprint, they’re easy to embed, render, modify and convert. The pcbdraw plot even has a command to generate an assembly manual for you, how cool is that? And surely, you could also add pinout diagram nice arrows to it with a bit of SVG modification?
The most important thing, however, is that your other option is pcbdraw render, and it’s not quite easy. It’s the one that creates 3D viewer 1:1 images, and given that kicad_cli has no such export option, you might guess that this involves black magic – your main hint being that the images take anywhere from 30 to 150 seconds to generate.
There’s no good way to say this – pcbdraw render opens the KiCad PCB editor in a virtual display window, analyzes open window headers to figure out whether the PCB Editor has loaded the PCB yet, blindly sends keyboard/mouse movements with xdotool, then exports the picture of the 3D viewer window, cropped used Pillow magic.
If you encounter a problem, I hope you are of strong enough spirit to debug it. For me, this is the only viable way, because my repository has about 200 boards and counting, and sitting in front of Inkscape is time I don’t have.
Me and my friend, who is a Docker witch, are currently hacking at creating a monorepo-viable integration for generating PCB output files, and we hope to release it soon – whenever we do, you too will be able to push all your boards into a single repository and have pretty pictures magically appear in your READMEs.
Update: this article was written ten days ago. Just a couple days ago, a kicad_cli pcb render command has been merged into KiCad 8, which actually exports 3D viewer images out of kicad_cli in the same way that pcbdraw render does, except it’s blazing fast! [arturo182] has made a “rotating PCB” animation script with it, open-source, too. So, the pcbdraw render method above is likely not viable anymore, unless you’ve got a very specific usecase in mind – pcbdraw plot, however, still has a whole bunch of advantages.
In the meantime, I hope that this summary and the integration and script links can help you in your own board publishing. You now know what it takes to auto-generate a board image or a Gerber .zip each time you make a commit into your repository, and now is possibly the best time to take a look at making your PCB repository more approachable. Got any favourite KiCad integrations? Tell us about them in the comments!
Nice. Arya: Thank you for bringing CI/CD to electronic enginering.
KiCanvas (https://kicanvas.org/) is another great tool for sharing KiCad project. It’s a web based, interactive schematic and board viewer.
oh oh yes, can’t believe I forgot to mention it! I did cover it a while back, and, it’s seriously an underappreciated tool still, we ought to revisit it!
“I don’t store Gerber files in the repository because that will interfere with how Git functions – you’re supposed to only have source files in the repo. ”
It is a horrible idea! Always save production files near project files after pcb factory validation!
1. Each factory can make some changes according their local capabilities, better to save final result before starting production for resolving potential issues after facing pcb manufacturing defects.
2. ecad software a not ideal as programming language standards and compilators: there are no guarantee that different versions of ecad produce identical gerber files (I faced with these issues in orcad), so when you update your ecad, output files may be different!
4. some settings that can impact on production files stored in global eCAD settings file, not in the project file!
how hardware professionals(not hobbyst) work with git you can find in these book:
https://content.allspice.io/en-us/git-for-hardware-guide
I can kinda see #1 though would probably argue that it’s better to back propagate these changes to the sources so that you don’t have to make them again on further revs.
#2 and #4, however are not unique to hardware. Different toolchain and workstation setups can absolutely have an effect of the machine code generated from source code. This is why hermetic builds, storing build artifacts from CI, and releasing artifacts from CI instead of someone’s workstation are good practices. ECAD tools definitely lag in the ability to implement these practices but, as this article points out, it’s nice to see KiCad making some strides here.
Agreed on all of your points! If something is difficult to manufacture or has to be tweaked for a fab, it gets into the source file and is committed into the repo accordingly. And, also 100% agreed on the #2 and #4 being not unique to hardware! Problem is, not only storing gerber files in the repo is a bad idea as far as git is concerned, it’s also that storing output files in git means they have to be constantly maintained alongside the PCB files you’re actually changing, creating a huge pain for making commits, which inherently a manual process. Plus, it’s a failure point if the in-tree files are ever actually relied upon.
A very true and fair point. With kici’s example projects, I build all Gerber’s etc (e.g. generated artifacts) on each pipeline run. Pipeline artifacts are stored a week by default. The ‘latest pipeline’ results arwbstored until the next run, which then replaces the duration for that. This is gitlabs default.
Anything built on master, is served as a ‘project snapshot’ and published to gitlabs pages, with all artifacts stored in the ‘webroot’. Since you dontnhave multiple webroots, gotta pick something.
Now all of these are pretty volatile if you will. The ‘rule’ is that once you’re ready, e.g. ready to send it to a board house, you tag your the release branch, with its first release candidate git tag. Anything tagged gets its artifacts now pushed to the gitlabs package repository and is kept forever. E.g. youbsend off your first release candidate to the house. This then comes back and either requires some fixing, and gets a second release candidate. Rinse, repeat. Once ‘happy’ with the protons, you tag your proper release, get your artifacts as before for mass production and gitlav creates an actual release based off this tag, keeping all the artifacts.
But as you said, board houses tweak stuff. Ideally those tweaks come back into your repo. But how to store these changes, which are done on the generated d output? Ideally fix them back into your layout, but if different board houses require different tweaks … I suppose the only way is to add/upload them to the generic package repo and add them to the release … But painful for sure!
The random black bars can be detected and stripped using ImageMagick or even Gimp.
I’ve written KiCi (KiCad/KiBot CI) template for gitlab. The pipeline component https://gitlab.com/ci-includes/kici allows one to just have a basic (minimal?) gitlab-ci.yaml template to generate all possible outputs. From the PDF’s, diffs on Merge requests (albeit in PDF form), to blender 3D renders and all common manufacturing outputs.
The pipeline template isn’t that special (thought I’m proud :p), it leverages for the most part the power of KiBot (https://github.com/INTI-CMNB/KiBot) and supplying a few (opinionated) job tasks for it.
It’s not been released for KiCad 8.0 yet (there’s a test branch somewhere) but for all KiCad 7 files it works a charm. I bet once I start pushing KiCad 8 projects to it though, things will complain left and right, but I’m dependent on KiBot so there’s that. I have installed KiCad 8 on my arch machines, so I’ll have to make the move sooner or later :)
Hope this helps (the wiki has some links to some of my projects where I use it, though the repo itself is also an example how to use it).
After reading the article’s end more closely, I shuddered a little.
Arya, your google too is weak ;) was their not a(n) hack(a day|ernews) article about this not too long ago about https://sschueller.github.io/posts/ci-cd-with-kicad-and-gitlab/?
Also you and your witch friend are reinventing kibot which also offers docker containers already!!
I strongly urge you to check kick and kibot out first. And while kibot even offers a github action thing, I obviously think my gitlab action (due to gitlabs features) is far superior and maybe can see you migrate to the light aside :) or have a mirror … gotta start somewhere.
Btw, why a monorepo? How are you tagging and releasing boards? Adding a tag tonthebrepo to indicate vX.Y would mean all boards are tagged?
I get the desire to share some things, but for 200+ boards having a shared submodule, subtree, remote SQL library might be a better choice (which can be versioned in itself …
Info get the convenience factor of a single repo, but 200j+ projects sounds like a bit much :)
> but for 200+ boards having a shared submodule, subtree, remote SQL library might be a better choice (which can be versioned in itself …
See, that sounds like effort =D For me, adding new PCBs, or someone cloning my repo, everything is super super simple.
> Adding a tag tonthebrepo to indicate vX.Y would mean all boards are tagged?
No, because I do tags like board_codename_v1 to show which commit specifically relates to which PCB, and, of course, I do “one PCB affected per commit”.
That’s a way around it I guess. But ‘super super simple’ vs organized tend to be opposites :p
Anyway, do checkout kici and kibot!
yessss we shall, thank you ^~^
> I have a pretty large GitHub repository, with all of my boards open-sourced there
No, don’t give us the link, that would be too easy!
Fixed, and here you go!
Check out https://github.com/opulo-inc/lumenpnp/blob/main/.github/workflows/export-ecad.yaml. We’ve been doing CI based ecad exports for years, though it could likely improve more. We use https://kibot.readthedocs.io/en/master/usage_with_ci_cd.html for our work.
One of the additions that I’m most proud of is the replacement of a `<>` tag on the silkscreen with the commit hash so that you could trace the PCB back to the exact commit from which it was generated.
(both of your messages got into approval queue because of first time posting and two links, I’ve left this one up!)
oooh that is nice! ^~^ and I should do the same for the commit, though, wonder how well it would work in my monorepo, maybe I would want to use something like `git blame` for the last commit that affected the related PCB file specifically!
Kick allows you to set a variable to decide for which board(s) to run the pipeline for ;) so should work great, albeit a little weird on a monster repo :p
Kibot has a preflight to do that for you. Check my repo for an example ;)
> I don’t store Gerber files in the repository because that will interfere with how Git functions – you’re supposed to only have source files in the repo.
Couldn’t the Gerbers be made available as releases on Github? If a PCB changes, create a new release with the new Gerbers. Goes with the philosophy that the repo contains sources only but builds are available as releases.
yes, but someone’s gotta generate those, and manual generation is such a chore!
some time ago i’ve started project in a similar spirit:
https://github.com/el-bart/kicad_sdk
idea is to have a reproducible build system (docker) that’d provide:
* STL of a PCB (reference model for designing 3D-printed enclosure)
* gerber files (PCB manufacturing)
* PCB graphics (docs)
* BOM (shopping list ;))