When you’re working with a development team, especially in a supporting capacity, you can often find yourself having to invent tools and support systems that are fairly involved, but don’t add to the system’s functionality. Still, without them, it’d be a dead duck. [Aidan Chandra] was clearly in a similar situation, working with a bunch of postgrads at Stanford, on an exoskeleton project, and needed an accurate data plotter to watch measurements in real-time.
This particular problem has been solved many times over, but [Aidan] laments that many solutions available seem to be too complex, hard to extend, or just have broken dependencies. This happens a lot, and it simply leads to yet another project to get going, before you can do the real work it supports. Based on Python and PyQT5, serial-plotter is a new beginning, with an emphasis on correct data acquisition and real-time data visualization with a little processing thrown in. Think, acquire data, show the raw values as well as the mean value, and RMS noise all on the same windows side-by-side, all of which is easily tweakable with a bit of programming using Numpy and Matplotlib.
One particularly important point to highlight is that of the handling of time-stamping. [Aidan] needed to ensure samples were logged together with a local MCU timestamp so that when displayed and possibly later post-processed, it was possible to accurately determine when a particular value or event occurred. With the amount of buffering, data loss and multiple-thread shenanigans, it is easy to forget that the data might get to the application in a non-deterministic way, and just relying on local CPU time is not so useful.
If you need to visualize data transported over the serial port, we have seen many projects to help. Like the highly configurable Serial Studio, for one. If your needs are a bit more complex, especially with multiple data transport methods, then a Supercon 2022 talk by [Alex Whittemore] might be a jolly good place to start.
RRDtool was probably not appropriate in this situation, since it’s not realtime, but it’s worth looking into if you do need some data collection and graphing that is not too hard to set up. It’s very old school and doesn’t have a lot of dependencies, I used to use it to keep network bandwidth graphs on my servers.
What’s wrong with a terminal program logging the data to a file and running gnuplot from chron?
Seriously, this is not even a shell script level problem.
People are reinventing the wheel with pathologically more complex “solutions” to the point that you can no longer find a proper solution in the haystack of idiocy.
Old man wanders off shaking his head and very glad he’ll die before the calamity of this silliness collapses civilzation back to 800 AD.
Plus one to this post
Gnuplot saved my hash in grad school. Too easy to plot data in real time from a model.
To be fair, I have been known to reinvent the wheel, enjoy the process and occasionally get paid to do it.
My primary tools throughout my career as a research scientist/programmer working for large independents to super majors in the oil industry were awk & gnuplot. The Marquardt-Levenberg routine in gnuplot is the best L2 solver I’ve ever used.
Awk would easily reformat the weird formats the data came in and gnuplot did the rest and made publication quality figures.
I did curve fits of exponentials with 3rd and 4th order polynomial exponents on a regular basis. It’s a bit tricky to do, you have to get close with a constant exponent and add the higher order terms very carefully. But I got the hang of it 30 years ago. I WOULD NEVER consider using any other L2 solver. I wrote an M-L solver in grad school, but it was a toy in comparison.
I remember about 5 years ago, I was working with another site in our company, they had a setup where it was datalogging from machines and I thought it was pretty spiffy, so I asked where they got it from. Turns out some intern wrote it (in python) a few years prior to that. I brought it back with me and ended up in dependency hell.
Now we are tracking down libraries of a specific version, which didnt play nice with something else and wont work with a 64 bit OS blah blah blah, screw this stole the two or 3 idea’s I really liked and wrote my own in .Net
I have learned throughout the years to not rely on “those kind of development platforms” where its community driven and in a constant state of flux and basically recreating a dev environment just to run an application … not just python but Java was pretty bad about it as well.
Point of all this is I find it amusing that they were complaining about broken dependencies, while writing yet another e-waste program which will be borked as soon as joe blow the library developer decides to change something in the new version.
Everything involving Python has always felt to me like building a Jenga tower. As long as you don’t touch it, it’s fine, but as soon as you try to change anything it starts falling over in every direction.
Every “simple” thing, such as setting up some sort of data logging system using “numpy and mathplotlib” always takes two weeks to a month to figure out why it doesn’t work, or what it even means with the obscure syntax and sparse documentation.
Point in case. It’s possible and common to create figures “implicitly” with mathplotlib, which means you’re relying on some assumptions and defaults which are not declared, which makes following the examples and tutorials an exercise in futility because they don’t explain what’s really happening.
It also creates scripts which are fragile, because the implicit assumptions may not hold true, and you’re dealing with hidden variables that can be different from what you assume, so debugging such code is a huge pain.
Working with it is like alchemy where the wizards are intentionally obscuring information from their students lest anyone steal their secrets.
I use to do this kinda thing in Qt. I recently switched to ImGui since I have much higher speed data now… ImGui is soooooo much better in my opinion.
PlotJuggler?
You are talking about outdated dependencies and then start your project with PyQt5 over the more up-to-date PyQt6 or Python for Qt?
Or just pay for https://www.megunolink.com/
How about that one: https://hackaday.io/project/5334-serialplot-realtime-plotting-software ?
I’ve used it for a while, and I think it’s great!