For someone programming in a high-level language like Python, or even for people who interact primarily with their operating system and the software running on it, it can seem like the computer hardware is largely divorced from the work. Yes, the computer has to be physically present to do something like write a Hackaday article, but most of us will not understand the Assembly language, machine code, or transistor layout well enough to build up to what makes a browser run. [Francis Stokes] is a different breed, though, continually probing these mysterious low-level regions of our computerized world where he was recently able to send an Ethernet packet from scratch.
[Francis] is using an STM32F401 development board for his networking experiments, but even with this powerful microcontroller, Ethernet is much more resource-hungry than we might imagine given its ubiquity in the computing world. Most will turn to a dedicated hardware ASIC to get the Ethernet signals out on the wires rather than bit-banging the protocol, so [Francis] armed himself with a W5100 chip to handle this complex task. Since the W5100 was on a board meant for an Arduino, there were a few kinks to work out, including soldering some wires to the chip, and then there were a few more issues with the signaling, including a bug in the code, which was writing too many times to the same memory, causing the received packet to be enormous while also completely full of garbage.
In the end, [Francis] was able to remove all of the bugs from his code, reliably send an Ethernet packet from his development board, and decode it on a computer. This is an excellent deep dive into the world of signalling and networking from the bottom up. He’s done plenty of these types of investigations before as well, including developing his own AES cryptography from scratch.
We’ve looked deeply into Ethernet, too. You can even make it work on an FPGA.
What about bit banging Ethernet in 2014, on an attiny85
https://hackaday.com/2014/08/29/bit-banging-ethernet-on-an-attiny85/
And plenty of more recent examples, Raspi Pico is popular apparently.
https://html.duckduckgo.com/html?q=ethernet+bitbang
Don’t forget Ethernet on ESP8266
https://hackaday.com/2016/04/01/ethernet-controller-discovered-in-the-esp8266/
Something I’ve never understood about Ethernet is given it’s usually a single wire and thus there’s no control lines how does the system arbitrate whose turn it us to talk. In particular can’t two devices just step on each other when trying to talk at the same time? If there’s a master polling known devices in the cable for their turn how does a new device join and register itself without accidentally talking out of turn ( since if won’t be polled yet)
How is this collision avoidance managed ?
Google CSMA/CD – it’s the clever way Ethernet handles contention over a shared channel
It’s not one big wire. An Ethernet cable has at least 4 twisted differential pairs. The insulation on the outside wraps them in a neat package.
These days, yes. But the first iterations of Ethernet were over coaxial cable, in which a network was literally one piece of coax (terminated with resistors at each end) that ran from computer to computer; each computer had a tap in the coax. To transmit, a station had to listen for a certain amount of time for silence on the wire; it then started transmitting, and if it detected a collision it would back off for a random period, then listen and try again.
Thicknet was inconvenient. The cable was expensive, it has a relatively huge minimum bend radius, and it’s stiff. And vampire taps were a pain to install. But it was a technology for moving 10Mbit of data long distances between any two computers. The other options in the day were awkward, slow serial cables that physically connected two (and only two!) devices, and could only reach about 75 feet before needing a technical boost like a pair of short haul modems.
Look at 10BASE5 (thicknet) and later 10BASE2 (thinnet) for what your elders had to work with back in the day.
In a nutshell CSMA/CD (carrier sense multple access w/ collision detection) works like this. Look for voltage on the wire (carrier sense). If it is quiet, go ahead and send. If the voltage on the wire exceeds what you are sending during transmission (two signals will randomly add or cancel each other) there are two people transmitting at once (collision detection). If there is a collision, wait for a random time (within a range given) and try it all over again (if we both waited a preset identical time we would guarantee another collision) If the next transmission collides again, lengthen the random waiting time window ( easing network contention ). All listeners on the wire detect a collided packet and will discard it before passing it up the protocol stack.
Also, there is always at least one pair of wires in use never a single wire because there is no guaranteed ground reference on twisted pair ethernet systems.
Original coax ethernet can be thought of as a single pair (shield and center conduction). Half duplex only. CSMA/CD is always used.
Original 10mbps twisted pair ethernet uses wires 1-2 and 3-6 as pairs, transmit pair and receive pair, half or full duplex. Minimum of four wires in a cable. You could be on a dumb hub (which can have collisions) or a switch (which does not have collisions) so you need CSMA/CD.
Fast ethernet (100mbps) and gig (1000 mbps) and up ethernet uses more pairs for faster transmission. The standards evolved on how many and which pairs were used (now a gig cable uses all four pairs). In fast and gig ethernet you can get rid of the CSMA/CD (altough it is still in there as a backward compatibility option) because you are on a switched network that cannot have collisions because it is only the switch and one device talking to each other with a dedicated pair in each direction. If two packets are destined to you at the same time, the switch will buffer (store them) and put them in line for transmission (called queueing). If too many fill the buffer, they overflow and get discarded. At that point the higher level protocol ( like TCP) would be responsible for resending unacknowledged packets.
Lol, now that’s impressive!
Do IT students still have to memorize the CSMA/CD flowchart these days? I know that OSI layer memorization has gone by the wayside…
I have made same rough calculations and RP2040 overclocked to 250 MHz should be able to comfortably transmit 100Base-TX with PIO and 3 DMA channels, performing all the 4b5b and MLT-3 without CPU involvement. I currently do not have any use for it, but if somebody else was bored, it might be a fun project.
See https://github.com/rscott2049/pico-rmii-ethernet_nce for one example. Though it does use LWIP…
I meant without RMII, using only GPIO plus some passives. Obviously plus resistors and ideally a pulse transformer.
You can reliably bit-bash 100Mb/s Ethernet wires directly in software, while doing other processing with guaranteed hard real-time performance. No FPGAs nor ethernet chipsets in sight :)
The XMOS xCORE chipset, software, toolchain and libraries show how to do it. Buy them at DigiKey.
It is easy, provided you have an intelligent software+hardware ecosystem, with an multi-core MCU and an RTOS implemented in hardware :)
Ben Eater had a series of videos where he basically reinvents the internet. Is amazing
A lot of us old-school old farts, some of whom programmed major applications in pure assembly language for close to 25 years long before Windows was even a gleam in IBM’s or Bill Gates’ eyes, struggle not to sneer at “you youngsters” for not knowing how computers actually work, under the good — let alone all the way down to the “bare metal” of assembly language. Fortunately, I myself have managed to moderate that impulse somewhat, by realizing that I myself likewise do not know quite enough about transistors and PCB layout (I might have gotten there, but I switched majors from EE to CS just when it might have gotten interesting) and could not build a CPU from scratch without considerably more skull sweat than I’m willing — or maybe even able! — to exert at age 60+. So, “everything is relative.”
Still, I am tempted to require, as the price of admission to my inner circle of respect, a few small challenges which I myself performed in the course of my daily grind, in the 1990s:
1) Create a point-and-click image viewer from nearly bare hardware, with minimal built in software support: Start with a video card that comes with just enough software support to let to you initialize the card and match it to your monitor; draw a line; draw text; design-and-position an overlay sprite; and send a horizontal line of pixel data to the screen — and a mouse that plugs into what would ordinarily be a terminal port. Write, in assembly, the software to initialize the card (which requires knowing the specifics of NTSC video waveform features and their timing, and your monitor’s requirements in those terms); read the terminal port and interpret the incoming characters as mouse motions and buttonpresses; draw rectangles on the screen and combine them into windows and buttons that respond appropriately when clicked; and read, scale appropriately to any requested dimensions,, and display, 24-bit RGB images having thousands of pixels height and width, all in a reasonable time interval to strike the user as “interactive.” For full credit, impress bosses and customers enough to enable the sale of a significant number of additional units of the company’s product, now that you can actually see the images you’re working with, after years without this ability. For bonus points, implement a Mandelbrot Set fractal image generator on this same hardware, just for fun.
2) Discover and fix a small error in the support software for the abovementioned video card: Confirm the error, in the C source code supplied by the vendor — but, not having a C compiler to rebuild the source, and not wanting to have to patch every single .exe built with the buggy library, fix the problem “in situ” by patching the object module in the vendor-supplied library — i.e. in a file format never intended to be patched, using tools never intended for that job. For full credit, have this fully fix the problem. For bonus points, propagate the fix back to the vendor and continue without a C compiler for several more years yet.
3) Decompile executable code back to Fortran source code, by hand: Reconstruct a piece of accidentally-deleted Fortran source code by first, finding the relevant executable machine code in the release .exe file; then, disassembling it into machine / assembly (not quite the same thing!) instructions; then, recognizing subsets of the instruction sequence as the typical output of the Fortran compiler when given specific Fortran statements; and finally, typing those statements into a new Fortran source file, one at a time as you identify and accumulate them. For full credit, compiling the new Fortran source shall produce the exact same sequence of machine instructions as was found in the original .exe file. For bonus points, do all of this between 8 PM and midnight on a work night, after having yourself been the jackass who accidentally deleted the only copy of the original source code in the first place. For double bonus points, never tell anybody else, even your boss.
For double bonus points, complete all of the above challenges using a text-only, monochrome, character-cell, serial terminal: no color, no graphics, no fonts (so no GUIs, no Windows, only one activity at a time), and no Internet. All documentation is either in books on a shelf in another room, on scraps of paper supplied by vendors, begged from strangers on Usenet via dialup, with up to three weeks’ turnaround time — or entirely non-existent. There is no World Wide Web, certainly no Google. If you had money you might be able to spend 50 bucks of your own money to have the user group for your OS ship you a tape full of interesting stuff, but you don’t have the 50 bucks — and besides, your bosses don’t trust free software.
3) Use C libraries and APIs from assembly: Years later, when your own OS vendor begins supplying and supporting graphics capabilities via an API designed-and-intended to be used from C, but you still don’t have a C compiler, reverse-engineer how to write assembly code that “looks like C” to the rest of your toolchain, and use the vendor API from assembly, to write numerous useful tools, including a complete image-display app. Because your screen can only display 256 colors, implement Floyd-Steinberg dithering on a “6×6×6 color cube” palette, to trade off spatial resolution for color resolution and “fake it.” For full credit, your resulting display should be adjudged superior to competitors’ real, physical, 24-bit (“millions of colors”), displays in side-by-side comparison at a trade show. For bonus points, while waiting for the 256-color display to show up, experiment with displaying images on everything from a 3-level-grayscale graphics terminal, to a black-and-white (no grayscale) graphics workstation. Produce useful tools for viewing and inspecting images, on all these platforms, all while working entirely in assembly, of course.
Bonus challenge: One night, just for fun, put the black-and-white workstation into hexadecimal keypad mode, and compose an entire email to your work buddy by typing the hex codes for each character. 48 45 4C 4C 4F 20 57 41 4C 54 45 52….
If these are not enough, I’m sure I can think of some other challenges that have slipped my mind just now. There’s no substitute for understanding how things actually work.
You sound extremely disagreeable and I suspect you yourself are taking up valuable lawn space. Your manifesto is not doing what you think it is.
There is value in understanding the layers of abstraction below the happy home of your development environment. Often everyone assumes that you already know it or don’t need to know it.
Johnny BigBalz couldn’t get a C compiler FOR YEARS ladies and gennulmen when Lame-o Hobo, this guy, could get one for free every day of the week?
Listening to this guy is a subscription because he’s got issues. Thinks he’s protecting the lawn.
The game Pong wasnt even written in assembly, 100% circuits
You only need to understand the Von Neumann architecture of a CPU to get the insight to the hardware enabling technology for low level software.
I am certain you can program color into a monochrome display with your greatness, sir
Wow! Thankfully I don’t like to spend time in small, cramped tents! While I am no youngster and I do in fact recall the good old days of assembler, I for one believe that anyone agreeable is welcome in my life, be it a youngster who does not know how the bare metal hardware works, or an old fogie who knows the hardware like the back of his hand… My tent is large and everyone who is willing to give people a chance and a hand are welcome. Good day to you, sir.
And here is something that comes between bitbanging and using a Wiznet chip. http://www.tuxgraphics.org/electronics/200606/article06061.shtml
It’s a bit ironic that he used a chip implementing a whole TCP/IP stack to do raw Ethernet. But I get it, it’s much easier (and cheaper) to get W5100 than a proper MAC+PHY.