Jake Robert Read

Visting Shenzhen

Tue, 20 Jan 2026 00:00:00 -0500

“Everyone has already heard about the young person who comes to Shenzhen to make something and is astounded by the city, this is an old story - let’s talk about something else”

An MIT Alum said this to me in our cab ride back from our visit to a market leading Robotics firm after about a week of my own time as a young person having my mind expanded in Shenzhen (thanks to MIT’s INM for bringing me along on that daytrip as a guest). The funny thing is that I didn’t feel as though I had heard that story too many times, or I had but had kind of saved it to chew on for later. I have been nervous to see in the flesh what I have long felt is true - that while I have been whittling away in the basement of the CBA (I am so close to graduating) trying to make myself excellent in the fields of manufacturing and robotics, this whole country on the other side of the globe has been doing the same - while my country(s) have been abandoning it. That cab ride was about a week into the trip. At the moment (one more week later) I’m sat on Cathay Pacific flight CX812 from Hong Kong direct to Boston, it’s 11:47pm according to my body clock but 10:47am in my new local time and so I am going to sit down and chew on it; if you’re reading this you’re along for that ride.

In this note: some vibes based economic analysis from the morning of my arrival, some notes on my glancing blow with the markets and manufacturing ecosystem in Shenzhen, and some photos and thoughts about cities and scale.

Special thanks for this experience goes to Cedric Honnet who (at the behest of nobody, apparently) goes out of his way every year to organize in cowboy-style the Research at Scale Residency, including the Scalable HCI conference. This was all generously enabled by Seeed - some more on them later. Thanks also to the CBA for funding my own travel so that I could thread the needle between a Shenzhen visit and also saving enough time to get back to Boston to finish writing my dissertation.

From 2026 01 04

This chunk is first impressions upon arrival, with added notes and photos post-hoc.

After arriving late last night to Shenzhen and having a kind of strange jet-lagged sleep, I got up this morning and went for a wander - and to find a coffee. Now I’m here trying to articulate this sensation: I have travelled pretty well (grew up in Canadian Suburbs, studied for eight months in Halifax and four months in Rome (thinking specifically about cities), interned in San Fransisco and at Princeton, lived briefly in Toronto during the Spaghetti Years post-undergrad, have been to Mexico, Switzerland, Vancouver, New York, and taken work trips to Kerala, Santiago, and Tolouse - even as far out as Puerto Williams (the Chilean jump-off to Antarctica)). But this feels totally different.

The hotel I’m in is a subset of a larger building complex (called Boyu Nanshan Yuncheng Flagship Store per the maps translation). The complex is a little city all on its own: there are families and youths mixed into what must be thousands of apartments, the lower floors are filled with small businesses, cafes, restaurants, stores etc. The elevator ride is long enough that it’s worth considering when you estimate how long it’s going to take to get anywhere. Probably I am making myself sound like a country bumpkin, but I guess that’s just the sensation that is new to me. Normally I go places and I feel like I can kind of comprehend it: the scale makes sense and I am not really in awe of anything. This feels different, things are big and there is a real sense that the thing is all working together.

Walking around on the street level, I was struck also by the calm that it had. The air felt fresh, the cars are quiet (my current estimate is that only about 20% of the vehicles have tailpipes). The streetlights are not running - I don’t know if this is intentional or what - but traffic proceeds without issue, people giving eachother way and making it through relatively busy intersections efficiently. I stood for a while waiting for a walk sign until I realized that this is what was happening.

No streetlights to govern, no worries.

It’s unclear to me if this is planned or if they have simply not finished with the lights, having many other city-block-size projects to get on with. Another thing that I noticed a few days later is that these intersections are watched over with loving grace by many CCTV cameras. So, security and calm via the panopticon - whereas in the west our surveillance is somewhat more obscure and functions primarily for the purposes of ad sales.

On the sidewalk you share space with little electric scooters (many running deliveries, many are just personal transport), but there is ample space. Many streets (in my one-day sample) have dedicated lanes for them, something basically anyone who lives in a western city will be envious of.

Shenzhen scooter design is evolving faster than it is in the west. They co-habitate with pedestrians, trucks, cars, and tuktuks filled with Yams.

The scooters are everywhere - I wondered for a few days why traffic didn’t seem as bad as it aught to in a city of this scale, and then I saw enough of these giant fucking piles of scooters to realize that they’re doing a classic case of lots of little vehicles vs. lots of big ones, seems pretty effective and they look like a blast to rip around in. Two weeks of this later and I can say that there is sometimes a cacophany of scooter-honking but I witnessed zero collisions and only one near miss.

Scooters at SusTech, Yihua Market, somewhere between Yihua and Nanshan, and in Nantou. I would regularely see these on highways and major roads, resplete with helmetless riders.

Photos above: descending into our base camp at Chaihuo in a so-called design commune; a kind of brutalist sunken park / mixed use space. There is an amount of three-dimensionality to these cities that made my inner architect excited. Also a sense of planning: Vanke is a monster developer in China, they made this with the explicit purpose of building lots of small (and hip) little studios / offices for creative and “innovative” firms to start up in. There are cafes and bars and design firms and lunch spots and design stores all setup in here. Somehow it is a park as well, on the top/sunny side are children playing and weird little designer dogs out with their people, and a set of basketball courts.

Above: more from the design commune park: subterranean bbq’s, topside cafes, whatever this little courtyard nook is, and a little sports complex. All of this takes up about three city blocks in Nanshan with monumental-sized towers on the north and south sides, including the DJI Headquarters (above, right).

On that first morning it was the peace of it all, paired with regular city bustle, that really struck me. I picked up a coffee and wandered down towards the makerspace that served as our basecamp for the residence, Seeed’s Chaihuo makerspace (Chaihuo translates roughly to fire starter or fire kindling), our main sponsors / hosts. It lives in a design commune built by Vanke, the same developers of the hotel complex. It is sunken into a tiny stretch of park, and on my way in I was met with the street cleaning crew who were working on the sidewalks. I may have been projecting but I had a sense of their contentment: I though of the quip that my old boss and buddy Saul used to make, that when he retires he hopes to sweep the floors in a shop that he built while the next generation builds whatever’s next. This is the vibe I got from the cleaning crew: older men in dignified jobs, doing a service for a city that they feel is theirs.

Up at 6am, I was out with the morning cleanup crew: mostly older dudes sweeping with this style broom (yet to modernize: if it ain’t broke…).

I often feel that this simple sensation… the feeling that our work contributes to something that is real and worthwhile - is largely amiss in the western world. The vibes are off and it seems like everyone can feel that, but we can’t quite put our finger on what’s missing. I have always held that the economy has basically two faces: the real shit (building, inventing, maintaining, transporting, healing, etc), and the extractive: financial speculation (though I will allow that capital markets play an important role in allocating economic energy, they seem clearly (1) overcomplicated and (2) highly ‘skimmed’ by those who direct them) and all of the various forms of rent-taking, i.e. literally landlordship but also any economic activity that is predecated on the sole ownership of some exclusive property (especially those that require no maintenance, i.e. IP (also netflix and the like)), and the extraction of rents (subscriptions, licenses, and “locked-in-on-proprietary-tech” business models) on those properties. This second half of the economy is fundamentally not productive - it serves only to accumulate capital into bigger and bigger piles.

Economies where real growth is low overall (not a lot of building shit) naturally tend towards rent-taking: when there is less new output, it is much easier to make money by owning more of the pie than by participating in the making of more pie. Because rents can only be taken on things that are owned, the result is that prior ownership of assets becomes more important than “real” activities i.e. labour (of which we all own an equal amount of).

This is the broad-strokes driver of inequality and it’s vibes-based counterpart: the feeling that we don’t have meaningful stake in our world, in our city. For a properly enlightened read on the same, see Piketty’s Capital in the 21st Century, probably one of the most important economics works of the first quarter of this century and an exceptional piece of level-headed academia (it is also actually enjoyable to read).

The main dynamic that Piketty discusses in his Capital; where the growth rate (more real stuff) is larger than the return-on-capital investment, we get equality. Or the opposite. The second chart here is to 2010, and you will probably believe that it goes up and to the right from then until 2025. Notably missing is Chinese data: FWIW its Gini Coefficient sits somewhere between the USA and Canada.

For a simple example of the results of this dynamic, readers of my generation will have probably noticed that access to real wealth basically requires our (1) being born or married into it or (2) making some new and valuable firm (a startup) after having convinced other capital owners (VCs) to lend you a few tens of millions dollars - and subsequently convincing other capital laden folks to purchase it for the purposes of their future returns. Consider also the phrase you will own nothing and you will be happy and think about how many things in your life have tended in this direction (and whether or not that brings you joy). There was a moment there where we had enough real growth in the western economy (following the destruction of WWII, it’s accelerated industrialization of the west, the joining of women into the workforce and ~ the yolking of North America’s previously peacefully-at-rest vast natural resources) where many (not all by any means) folks could work an honest job and eventually enter the owning class (via home ownership). Today, this kind of salary requires that we get into the top few percentile of day jobs.

So that’s where I was at: thinking about the scale of the city (big!) but the simultaneous sense that everyone here is in it - is a meaningful participant. I won’t say that I am any kind of authority on the topic but I am here trying to tell you about how/why it made me feel this way, not really to convince you of anything. I had a seat in the little pit of the design commune and honest to god I shed a few tears about it. I will hedge again that I had just done about 24 hours of travel / wakefulness and only a few hours of sleep, and that I am personally really going through it these days, but these were not a woe-is-me set of tears, this was kind of… awe of the world, awe of this other culture’s accomplishment, sadness that my own culture has not been working on a similar project of massive growth via mass economic participation, and excitement that it is happening somewhere.

There is much other hedging to do on all accounts; relative measures of actual justice and equality in any of these large and messy systems that we build for ourselves to live inside of. China is only now having its reward on the backs of a few generations that absolutely struggled through it, and I won’t pretend that I completely understand their story. This was just my feeling and thinking after walking around about two or three blocks in Nanshan which is just a very small (and shiny) slice of a very, very large place. I absolve myself of any responsibility for (1) not having seen enough of any of this to properly comprehend it at this point and (2) bringing all of my prior bias and readings and thinkings to the topic. As noted we’re doing travelogue in this one, y’all. so it is vibes based economics for today.

To close this one out I want to try to articulate that scale again. Below are three images: the view from this hotel / apartment in the morning (facing south-west and then south: an infinity of towers) and then the slice of the city that I am estimating I am looking at. By the way that haze is water vapor not smog, the air quality index was excellent and my lungs felt great the whole trip.

From 2026 01 20 (on the flight to Boston)

16 days later and I still feel like those reflections above are a reasonable take. I want now to look a little more at what I saw of the city itself, and then aim at what has always been the beacon for me: economic empowerment through increased participation in manufacturing. For this I will share what I saw from the markets (inputs and outputs to / from the Shenzhen manufacturing ecosystem), and then some factory visits (transformers of the same) as well as some robots that represent top tier output. And we will get back to those street sweepers.

Canadian Architecture Student visits the Big City and Gapes at Buildings

I love a busy city for the same reason that I love to think about economics: these are real manifestations of collected human effort, cooperation and struggle. If the “economy” is the holy ghost (ephemeral, impossible to completely measure), the city is the body of christ (messy and human), and the father is nowhere to be seen (or maybe it’s the govt’, but I don’t want to give them that). They contain an infinity of little stories that are all linked together, they are systems of systems that all somehow work (mostly). They are all (mostly) guided by individuals who want to better their own world and that of their neighbours.

Your browser does not support the video tag.

Shenzhen is big; it should be possible to look at a map and understand this but of course it doesn’t really work. You may recall the feeling of hiking towards a mountain: it is large in the distance and then gets larger over the course of minutes, but slowly. Suddenly you are under / within it but you never really got there, it just eventually became your surroundings. There are buildings like this in Shenzhen.

The DJI HQ from ~ 500m away, 50m away, and right below.

To a foreigner it is also fantastically difficult to navigate. It feels that there are two extra layers of scale: I am used to finding a building on a street in a neighbourhood of the city. Here there is an extra terminal guidance step once you get to the complex of buildings where your goal is, and an extra layer somewhere between neighbourhood and city. Take for example maps provided to us to navigate to our hotel (itself just a particular set of floors in one tower of a four tower complex), and to navigate into the Chaihuo makerspace (nested in the belly of a three-block long “design commune” development (subterranean)).

Terminal Navigation to Boyu (our hotel) and to Chaihuo

There is a wonderful heterogeneity here as well. 20 minutes from the towering, bladerunner DJI headquarters is the Nantou “Ancient Village” (a tourist trap that I shamelessly enjoyed). Here is adhoc and small scale construction that architecture students dream of, buildings stuck on to other buildings and the dissolving of street / building boundaries. People seem to just be sticking structures in whichever places they can get away with to make more space.

I saw a lot of this in Hong Kong as well: shops, bars and restaurants where the interior of the space borrows from the street. Architects loves this blurry stuff. The climate here allows it and the density requires it.

Parts of Shenzhen are new and shiny but many of the towers are both monstrous and also well lived in, drying laundry heaps out of the balconies.

Many of the shiny new towers had a feeling that the wrapping hadn’t entirely been taken off, or maybe their first major maintenance cycles had yet to finish. I saw a lot of dirty windows in otherwise brand new looking buildings… the city being only forty years old means that it has been somewhat unburdened by maintenance so far - or uninterested in it since the new things are shiny enough by default. This is maybe part of a larger arc on technological leapfrogging - Shenzhen being “all new” enables them to make a better city than we can in the west because there isn’t so much cruft to deal with / old shit to maintain while the new activity goes on. For example it is famously difficult to build new transit in Rome because everywhere they dig is an archeological site. In modern western cities there are existing land owners to deal with, nimbyism and more, and planning authorities that lack the backing of a totalizing government body. These “encumberances” can be good; in Shenzhen, neighbourhoods are scheduled to be entirely demolished by the state and existing residents are simply made to move out. This is called an evacuation - as if it is the result of some natural force. The same kind of dynamic shows up in i.e. American power grids: the east coast’s is shoddier than the west’s because it was originally built many decades prior. Mistakes made the first go-round weren’t repeated, but the lessons and technologies were transferred. The same has happened wholesale in China, largely in part via grey-zone technology transfer from western companies (who put their factories in China while chasing a profit) that has many folks in the west upset. The mistake that we make is to ho-hum about it rather than get after our own improvements. There is no really good long term response to this other than to get good and dig back in ourselves.

The construction of the city continues; every morning I woke up around six am and by seven as I sat down to work on the 42nd floor I was met with the sounds of pile driving in the giant site to my south…

I also managed to get out for one run along the riverbank, which is done up like we would expect in a modern city and it was sublime to maplessly make my way down and back through it.

Your browser does not support the video tag.

Trucks of Shenzhen

All of this requires that a lot of mass move around: dirt, microcontrollers, fruits and fish. One of the moments that gave me chills was when I noticed one of the dump trucks from my window’s construction site moving around the city without emitting the whistle of a turbodiesel; instead it was the coil wine and gear-meshing sound of an EV. Rather than a tailpipe I saw an absolute monster of an electric motor bolted onto the rear axle of the truck. In the end I think about 40% of these particular model haulers I saw had been converted to EV’s.

Believe it or not, this is an EV.

I had been prepared to see lots of EV sedans and scooters, but I think I was still of the mind that workhorses were going to be trusty old diesels. This was not the case and it was joyous to see boxy, boring and dirty old vans and haulers rolling around via the magic of motor control rather than interal combustion.

The Markets: HQB and Yihua (and LCSC and JLC)

So far we have buildings (made of stuff) and trucks (moving stuff) but what about the buying and selling of stuff? For this we have Huaqiang Bei (HQB) and “Yihua Electron Plaza” (per the maps translation) and pitched to me as “the factory market.”

HQB is a famous landmark that most electrical engineers will have heard about at some point - a place where you can actually go to the Murata stall and buy a reel of 10000 resistors, capacitors etc. There are already really good write-ups and resources for the market and its navigation but I will relay my own observations.

I don’t think I’ve ever had my visual cortex as stimulated as it was walking through HQB. It is chaotic, disorientating and on first glance lacks any real organizing principle. It is a bazaar and I am still trying to piece together how this could be an efficient way to do business in an era where I can go on DigiKey and find exactly the component I need against some millions of SKUs. I would have to suppose that it is the speed: DigiKey still takes a few days to arrive - but surely if I need thousands I am planning at least those few days in advance?

In the market for Mil-Spec Connectors, LEDs, or Robot Actuators ?

From what I gather, HQB has evolved from a proper electronics-components market (which would have made more sense in the proto-internet era of its origin) to what it feels like now, which is half components market and half shopping mall for knockoffs of Shenzhen-manufactured goods. I guess I was hoping to find the inside of McMaster Carr, Digikey and AutomationDirect, but it was more like DigiKey + BestBuy + … You know what, it’s not like anything else, it is HQB.

From Wartime Goods to Knockoff Airpods and Dyson to Flying Cars…

Your browser does not support the video tag.

DigiKey but… not online. Shoutout to MeanWell who make a solid PSU. Should’ve taken one of me at the Murata booth, but missed the opportunity.

The part that I did enjoy were the tools: here you can get the worlds best tweezers for PCB assembly, as well as some very cute but dubiously spec’d handheld oscilloscopes, thermal cameras, etc. Also: spectacular microscopes. Third dimension? Never heard of it.

Your browser does not support the video tag.

I saw microscopes and BGA rework stations and countless fixtures and jigs. There is a block where you can get every part to make an iPhone and then also the tools to put it together with.

Steve !

I am a machines fan and I was hoping to find those components: linear slides, motors, etc. HQB was not the spot for this, but Yihua (about an hour’s drive west of HQB) was closer. This is the place to shop if you are building or running a factory: we have industrial motors of all sizes, knockoff (or maybe genuine?) festo cylinders / slides / valves, belts, bearings, metrology equipment, cutters, drill bits, literal heaps of nuts and bolts and screws, etc.

On the road to, and arriving at ‘Yihua Electron Plaza’ - an industrial market.

Yihua’s main atrium.

Belts, PLCs, and Cutters…

Compressors, metrology, motors…

Pneumatic actuation, grippers, and soft grippers.

Shoutout to the CNC Booth

Couplers, slides, and motors.

Maybe most interesting were the factory accoutrements - safety stickers, QC stickers, PPE, etc. The junk drawer of the factory / all of the things that you don’t remember you will need until you need them.

PPE (those are fingertip condoms), stickers, and ESD Flip-Flops.

Oh, and the nuts-and-bolts of it all:

Hardware, babey. Bags of it.

Working in the Markets

The thing I saw in both markets that helped me to understand them better was that people are working there - they are packaging and preparing their wares for shipping, modifying them for semi-custom orders, and haggling - age old market stuff.

Your browser does not support the video tag.

They were there with their children on the weekend and after school, and they were kind of living there… being middlemen and old school retailers. It all has an informality that has been washed out of the western world: people negotiating with one another to try to find the right price and arrangement that could benefit both. I think the thing that really struck me was that informality and that flatness - you can probably get anything there if you can find a guy, in fact two folks from our residency have people they call finders who are professional market navigators. In the west we try to stand up searchable systems to do this, i.e. the digikey hierarchy that I pointed at earlier. Those are great but lack the flexibility of negotiating directly with another human and when we need to put a new thing on that market it takes time to get into the system. As a result we have some less variety and some lag between offerings and availability.

Special shoutout to Terry for showing us around HQB!

E-Commerce and MaaS

Besides some wonderful hand tools, I did not find the ballscrews, loadcells or spindle motor that I am “in the market for” at either of these places. For those I will have to use online vendors as is my SOP.

Going to HQB and Yihua helps me to understand things like aliexpress: it is a bazaar and it is flat in the same way that the physical counterpart is. Searchable but not organized, and never static. One of the things that I leave Shenzhen thinking about is how much those markets will do what I think of as modernizing - moving towards more stable, organized systems that are perhaps more consolidated and (would seem to be) more efficient. I have a sense that what I see as organization may feel like an encumberance to the natives of HQB, but I am not sure.

On the horizon in this regard are vendors like LCSC (Shenzhen’s DigiKey) and JLCMC (McMaster Carr). These are more organized and maintain the low costs of the physical markets. TaoBao (Amazon but faster) would also be worth understanding: it is even flatter than the markets, is online, and can deliver most wares same-day.

Last thought on the markets (real and virtual) is that they are overall soupier than what we have in the west; my current model for them is that they are the chaotic broth of a manufacturing ecosystem that is itself diffuse, heterogeneous and messy in the same way that some of the buildings are: there is so much new happening that there is not any spare energy to clean up what is already there.

Engineers (and Artists) from a World of Manufacturing

OK: trucks and markets are a side effect of the main activity. Historically that main activity in Shenzhen has been the manufacture of goods designed in the west, but this part is also changing.

In my short jaunt I got to visit a few sites where design is happening in Shenzhen and I want to show you some of those. The pretext for this section is that one of the core assumptions that western businesses make when they offshore their manufacturing is that they can do away with the dirty, “boring” and low-margin parts of their business while maintaining their innovative edge: their talented engineers, their IP, and their new product pipelines. This has always seemed like shoddy boardroom logic to me, so I was curious to meet some Shenzhen engineers and their work.

Your browser does not support the video tag.

Firstly we have these exhibits at the UABB where the artists are toying with robotics. These two were beautiful and were expertly crafted, which I think is points-for the notion that the practice of robot building is sufficiently diffuse in the population. Being at the Media Lab at MIT I have seen my share of “at the intersection of art and technology” type sh-, what I saw here was ~ up to or above the level of most of what I’ve seen from the lab.

For machines-hacking in particular, take for example this knitting piece (below): they’ve added computer controlled linear slides to one of the OG hand-knitting machines, and a Jacquard system (Jacquard was an expert systems integrator himself, by the way), and then fed one into the other. Art! I think, but also a solid little piece of engineering, done casually for the sake of it.

Your browser does not support the video tag.

We visited also AIRS, the Shenzhen Institute of Artificial Intelligence and Robotics for Society. Here were numerous robots designed to serve infrastructure projects: basically a suite of inspection and maintenance robots. Two notes on this: (1) it is very forwards-looking and practical of this institute to work direclty on such obviously forthcoming issues and (2) that the funding model is not confounded by the requirement for academic novelty, they are simply making functional things that directly serve a purpose. The innovation happens as a byproduct of doing real things and (from the few controls posters I saw) is driven by the desire to i.e. build better tools and more generalizeable systems for the next practical thing. This in counterpoint to what feels like a common practice where “innovation” is the primary goal and then “applications” are invented to suit demonstration of the innovation. I think that a lot of research groups get lost in trap where novel systems, architectures, etc are pursued for the purposes of publishing. Grounding them to real applications is furiously difficult because we have to also make whatever new invention work and if it’s robotics etc, work well and reliably. When the grad students who are working in your lab don’t have any grounded experience making i.e. robotic systems, it is even more difficult to ground whatever novel invention in real applications.

Academic output from AIRS

Your browser does not support the video tag.

Pipe Inspection, Underpass Inspection, etc… there’s a robot for that.

I want to stick a note in here about composability of robotic systems: at AIRS were lots of projects that were in the vein of “this thing strapped to this thing, plus some new stuff.” It’s the kind of thing that happens when we have a genuine sense of urgency about systems development (and a small amount of personal ego involved) I.E. it’s exceptionally common in wartime, observers of the current trends will know. I’m not going to do a whole tirade on that, but I think it’s a sign of real systems familiarity, and is a practice I’m interested in enabling more broadly. The guts of my own thesis is basically “tools for strapping different embedded systems together.”

It’s a billet world.

One of my running fantasies is to live in the billet world - where machining is so easy and cheap that, for almost any part, we might as well just carve it out of a block (a “billet”) of metal. The design is simpler and faster that way, but the trade-off is normally cost and time to machine. The component in the robot above (zoom at right) aught to be an extrusion of some kind, or a square tube. It would be cheaper, it would be simpler. However, these engineers seem to be living in the billet world: where CNC machining is so abundant that we can just throw it at everything.

The output we saw at AIRS was impressive because it was immanently real, and clearly made by people who were versed in the arts - who had studied the blade of mechatronics. Same with the art pieces we saw. This is a difficult thing to measure or account for, but you know it when you see it. Over at MIT we have for example this decades-long push to bring projects based education into the mainstream - that basically aims at the same outcome: that we could learn our heady maths etc while also reckoning with real systems and messy debugging. The alternative might just be to soak the world in a milieu of making things, then surely at least some handfuls of pupils will have been raised in that broth and the schools can be centers for excellence where practical skills learned in the world (where they naturally belong) are honed and refined with more abstract studies.

Seeed HQ

On the Monday following I visited Seeed’s office / HQ to discuss the possibility of my partnering with them to produce some integrated motor controllers via this interface. I met there with Seeed’s CEO Eric Pan and three of his engineers who are working towards expanding their ecosystem of robotic components. I hope to spend more time explaining my hopes and dreams for that project (a family of motors, motion controllers, and other modules…) but this particular rambler / travelogue / brain dump is not the time for it.

I do want to recount two things about the visit to Seeed. The first is that their offices are large and filled with interesting work - one full floor of a full size office building. I arrived around 10am and I would guess that I saw perhaps 120 people at work there. It is in a modern building and had a cozy, lived-in energy. There is a big fish tank with plants growing out of the top, people were brewing tea at their desks as they chatted about their projects. Their desks were messy in the way that a productive desk is messy: they had their products strapped down beside their keyboards, a number of those were machines or robot parts that were moving around as they worked on their control systems, interfaces and mechanical designs. Some of them were busy taking apart their competitors products, some had circuit samples and board design tools open, some were reworking and probing circuits, some were locked in writing firmware, and some were writing documentation. There was a lot of real shit going on, and it was a joy to see.

Your browser does not support the video tag.

The Seeed Offices: Bustlin’ !

I will hedge that this trip / residency is largely funded by Seeed and that my lab has an existing relationship with them - they worked with my thesis reader Nadya way back in 2011 and with my old colleague and pal Zach Fredin to make some kits for How to Make Almost Anything in 2020. We consume probably a few hundred of their Xiao dev boards every year, etc (they are friends of the lab). However they are friends of the lab for a reason and that is this, the second thing about Seeed, which is that they have an intentional relationship with open source. If you are not on this blog for the first time you will know that it is a dear topic for me and so I am definitely doing the heart-eyes when I talk about this but! Enough hedging.

An Aside on Open Hardware

1000 words in this aside, skip to the next section it if you want to see more Shenzhen and less discussion …

From what I can tell Seeed’s attitude towards open source is that, once something is sufficiently figured out in the manufacturing ecosystem, there aught to be available a good open implementation of it that will allow novices to get their hands on it, use it, and understand it. They are also no-nonsense about it: they expect to turn a profit and they can do this because they are in Shenzhen where developing and releasing products is cheap.

Seeed runs a co-create program where they partner with open source designers to license and produce open designs and split revenue.

This program is actually a nice nexus for the whole of Open Harware’s quandry; Open Software can work because the cost of one additional software unit is zero (besides maintenance), hardware has fixed and recurring costs. We can make money by selling the things we have designed (not many people actually want to go to all the effort to DIY a circuit even if the design is free, also our costs at volume (plus margin) are about on par with the cost for a one off), but if our thing is successful enough surely others will start selling them too. This leaves us with two questions: (1) why doesn’t Seeed just start making and selling Open Hardware projects that they think would do well in the market (why involve and then pay 50% to the creators, other than for good will?) and (2) why get into the business of making products where, if the product does well, you have already made it easy for your competitors to try to make the same part and undercut you?

This is still an emerging model and it sits in the dangerous gulf between prototype-project and commercial project. Typically open hardware doesn’t make it to the commercial step and in fact lots of major open hardware projects that have gone there eventually either go closed source (i.e. odrive) or die (i.e. mechaduino). I think figuring this out is well worth everyone’s time, and there’s some evidence that Seeed is on the right track: the foldascope first launched with Seeed, and in robotics they are now collaborating with reachy / pollen robotics. Robotics requires a lot of systems integration, something that open source can be really good at, so it will be fun to see how this all shakes out.

In any case, back to our questions. Why does anyone do this?

The first hidden dynamic has to do with the project creators. If you read Coase’s Penguin or Working in Public (both excellent), you will see that open source is a top tier labour market. That is, the people who select to work on some project and give it away for free are probably really good at it, or at least uniquely passionate about it. This makes them the ideal “hire” to work on that project, and means that a company like Seeed (who might need a project manager anyways) might as well work with the original creator. This person is going to be best suited to help take the rough edges off of the project and figure out how to fit it into a broader market. They are also very likely already the leader in the community of people who would use the thing: this is a perfect sales channel. They are also going to be good at supporting customers who buy the thing, i.e. through those user communities and with documentation authorship etc.

The second is just plain old market competition: by building relationships within the open hardware community, Seeed is hooked up to a pipeline of potentially innovative products that their competitors might not see coming. This is why they fund the residency and were so (surprisingly to some of us) excited to help us out. If they were to turn around and run away with something without involving the project creator, the (small) community of open hardware folks would share that and they would loose this competitive advantage. On the side of the project creators, there is the same market of contract manufacturers and it would not be exceptionally difficult to ask one of them to do what Seeed can do and simply do it first without them.

Finally there is a unique attitude to copying and IP in Shenzhen, which is that it is inevitable (there is a stall in HQB where you can have someone reverse engineer a PCB in its entirety for you: every trace, every part, and as much of the firmware as they can extract). IP law is ineffective and slow, so the more surviveable option is actually to (again) get good - be the first mover, and be working on the improved versions or newer products while your competitor is spinning up a line to make the thing that you already have in the warehouse. If your competitor is following you they are going to have thinner margins (they have to undercut you and their input feedstocks come from the same ecosystem as yours) - and your relationship to the creator means that all of the offical documentation etc (which is always growing) points to your storefront and not theirs. For a few dollars of price difference, any busy engineer is going to prefer the trusted vendor over the knockoff.

Unfortunately this model doesn’t exactly scale: if the product becomes so well established that it becomes a fundamental piece of kit, it essentially becomes a commodity and other manufacturers will be eating lots of the pie (which is still growing, by the way). This happened to i.e. the pololu stepstick and to the arduino nano etc. But for creators this is actually kind of cool - the thing you made is an important part of the ecosystem now, and you probably made enough money to keep doing your thing, which is to design new things (not run a race-to-the-bottom business). Ben Katz’ Robot Actuator is a famous example of this phenomenon: that basic design pattern (hand-wavey) led to the emergence of i.e. Unitree Go2 (who perfected it) by way of a proliferation of aliexpress clones of his original design. When I got to ask him about his feelings on this (it was the aliexpress proliferation era, not yet the Unitree era), he kind of shrugged and said “I think it’s kind of flattering.” I think he currently lives in rural-ish Western Mass, probably pulls a mondo salary (due to his actuator fame), and builds cool shit in his shed. Not bad.

The subtlety I think is that open source works well for what I would call “core competency components” - actuators, sensors… modules that are re-used in lots of applications. People who develop these tend to do it for the love of it, not for perpetual profit. I could go on but I won’t, if any of this is interesting you should read this and stick around this blog for a longer update on the topic.

Western Firms in Shenzhen

I was also lucky enough to visit a Cambridge-Native engineering firm’s Shenzhen office. Their founder is an MIT alum and I’ve met many of their engineers during my time in Cambridge and it is an institution of its own right: time at this firm is the practical equivalent to an MIT masters degree.

This firm has long worked with contract manufacturers in the Shenzhen area and obviously they are impressed enough with the engineering talent there to have launched this office. It is in Nanshan and takes up one full floor in a relatively new office park. I probably saw about 60 folks at work there (a Tuesday around noon). It feels like a failing on my own bias that I would have to say this but it was totally… the same as what you would expect in an engineering office in any western city. Maybe skewing younger, but it was just a bunch of college grads (they even had lattes) doing bigbrain work in front of fancy monitors in an open plan office. It was nice - it had regular cash-flowey office niceties, trees, a solid cafe downstairs and tasty lunch spots nearby. Same as the Seeed HQ: despite being halfway around the world in a country that can seem kind of opaque from a distance, the vibes are astoundingly familiar.

Your browser does not support the video tag.

Regular day at the office…

For another seat-of-the-pants jaunt into economics town (shorter this time), the salary arbitrage value I have heard thrown around is about 30%. For a talented mechanical engineer an American startup might pay $220k USD, for the same in Shenzhen about $70k USD (source: am in this market). Since these businesses are all international anyways (and are probably already shipping at least part of their product out of the region), this amounts to being able to hire three Chinese engineers for every American - or one in China before you could hire any in America. Add to that the fact that finding a junior engineer in America who has actually laid their hands on or repaired a piece of industrial equipment is (anecdotally, source not disclosed) difficult. Add again that these businesses spend good money and time flying their American engineers into Shenzhen to liase with their Contract Manufacturers (including i.e. interpreters and finders types of middlemen).

So, a picture starts to emerge… the results of funding the living daylights out of your educational institutions (and doing a little bit of currency devaluation as a treat). Also the value of having raised a generation of engineers whose parents and neighbours were either building, maintaining, or working in factories (and the related mechanical arts) - that’s the broth.

Factories, Automation, Tools

OK. We have the markets (goods exchange) and the engineers (goods design), what about the business end, the muscle of it all - value adders ? If you are like me you probably came here to see some gd machines doing stuff. Finally we have arrived.

Seeed’s Flex Factory

The first visit was to Seeed’s factory, which is a small but flexible operation relative to what is typical in Shenzhen. They have one board assembly line: paste squeegee -> inspect -> smt (small parts) -> smt (large parts) -> inspect -> selective reflow solder (through holes) -> final inspection.

Your browser does not support the video tag.

PNP, Inspection, and Selective Wave Solder

Seeed’s SMT Line

For the one line there is a lot of supporting storage. For scale reference the SMT line fits into about 20x60 feet, maybe 10% of one floor in their (two floors) of factory. Much of the rest of the space is inventory: parts (for PCBs and other products that are assembled here, so i.e. injection moulded parts, antennas, hardware, other odds and ends).

Seeed’s stockroom

Also stored are PCB solder stencils (for that paste machine), and test jigs. Lots of test jigs. These are used to i.e. load bootloaders into microcontroller dev boards, and also run full tests on electronics modules.

Your browser does not support the video tag.

Stencil Storage and Test Jigs

Ahead of the SMT line is a parts sorting room / kind of clearing house which is properly chaotic at first glance but for which they have a fairly sophisticated stockroom tracking system. These are the boring parts.

One surprise for me was that for PCBA runs less than 500 or 1000 units (depending on the board complexity), assembly is done by hand - as in, someone gathers the parts, applies solder paste with the stencil (by hand) and then tweezers the parts onto the board.

Tweezertown.

This is what we do in the basement of the CBA for board fab at low quantities, and I really thought that when I was sending boards out to be assembled at i.e. JLCPCB, PCBWay or Seeed, they were running them through pick and place lines. I appear to have been wrong.

As part of the residency I sent a board through Seeed’s PCBA service. They invited me for a second visit to see this happen and indeed it was a team of folks with tweezers assembling my boards.

Parts, Docs, and Kitting.

It was a little bit strange watching these folks do this for me, but also cool to see how the pros do it… it is mostly the same as what I do but they are faster and more organized than I am. They also have the good tools: these tweezers are the same that I eventually picked up to take home from the Yihua market. They have hardened steel tips that are quite sharp and are a different material than the body of the tweezer, a little bit like a proper chef’s knife with a layer of hardened tool steel in the blade that is forged together with mild steel jackets for impact resistance.

Your browser does not support the video tag.

Stencil to apply solderpaste, then tweezering components. And those mf tweezers.

Other tools, like the jig they used to solder stencil the second side of the board (after the first was soldered) were ad-hoc, i.e. a stack of paper.

Reflow, Human Inspection, and Machine Inspection.

They also sent the boards through their inspection tool, which uses a telecentric lens to image the board as a flat scan.

Your browser does not support the video tag.

Machine Inspection and Flat-Scan Images.

This board is one of my knuckles set that is long overdue to get out into the world. Have a thesis to finish first, but the long and short of it is that it’s an IMU module based on a chip that Quentin found and that runs up past 7kHz sample rate. For machine things.

The PCBA in this case was done in Seeed’s flex assembly shop, a medium sized room on the first floor where they do the least automate-able step for short run products, i.e. final assembly. They are also working on this cute little robot with pollen robotics and huggingface.

Seeed’s Flex Assembly Room.

Your browser does not support the video tag.

Reachy Mini and Assembly.

This factory is in a little industrial park that’s on the near-periphery of the city. The park had a good energy to it, and when we left after the second visit (around 630pm) it seemed like most folks in the area were getting ready to wrap it up for the day.

Outside the Seeed factory.

A Flex PCBA Line

Cedric hooked the whole group up to see a Flex PCB Assembly line as well. This was further out of the city and a fairly small operation as well. Similar to Seeed’s line, but a little less sophisticated.

Your browser does not support the video tag.

Flex PCB Assembly line: automation + manual rework.

Was not surprised to see lots of manual intervention. I think that there is a lot of myth making about automation around lights out factories - which are in fact quite rare. Rather, most lines and especially flexible ones are still kept alive in large part by people doing spot fixes or a few of the tricky steps (or are busy reconfiguring and fixing the automation equipment).

Real !

Factory accoutrements, lunch, and the crossing into Shenzhen’s Special Economic Zone.

Injection Mould Making

We saw an injection moulding shop (I missed the moulding step because I was oogling the Sink EDM machines). This is Electric Discharge Machining - we take a copper tool and drive an arc between it and the workpiece (probably hardened steel) to ablate it. Great for making tiny, sharp pockets in moulds - and produces a solid surface finish.

Your browser does not support the video tag.

Sink EDM: drop, discharge, wash and repeat…

Sink EDM: CNC to make tooling, and an old school edge finder.

Sink EDM: tools, output, and fixtures.

Punching gates in a mould, and an in-progress unit.

The Motor Factory

As part of this potential motor project, I whipped up an interface proposal for mounting closed-loop controllers to stepper motors. Seeed hooked me up with a visit to this factory for a tour and to potentially work together on the project.

Short notes on that: I learned that you need to show up to this meeting with (as close as possible) a 1:1 of the thing you want to make. With the language barrier, drawings are good but mostly useless and it’s better to be able to take the actual thing apart and spec every relevant detail. This is obvious in hindsight and for me this was more of a see-if-it-is-possible activity.

In any case, it seems like this factory can make just about any electric motor smaller than a 355ml soda can. I actually didn’t see any steppers in their factory (this seems to be a shrinking market), but did see many BLDCs and DC motors being assembled.

Motors in the showroom. Last is the Dyson 100k RPM Fan that powers their modern lineup, this part and its knockoffs have become a commoddity.

Lots of CNC here to make motor housings, back irons etc. Lots of young CNC operators, this is a problem in the west where most operators are kind of old hat and filling junior roles can be difficult.

Your browser does not support the video tag.

Rows of CNC, and young operators making motor bell housings.

Your browser does not support the video tag.

Laminations and Windings, and soldering by-hand windings to inerposer / connector PCBS.

This factory was at a different scale than the others. They reported 40% growth year on year for the last two and were actively expanding, I saw a new test facility going up and some factory floors being epoxied in prep for new lines. I didn’t get an answer on their total sales volume but I would estimate the whole factory was producing about one complete motor at least every few seconds. Doesn’t take much imagination to understand why demand for these types of motor has exploded in the last few years.

Your browser does not support the video tag.

Rotors, their winding machine, and stacks of the same.

Hadn’t seen a coil winder in person before so that was excellent. Again there are many many humans here: moving things between machines, operating them directly, testing motors one at a time, and doing fine-dexterity work like terminating windings onto housing connnectors.

I’m always curious about who builds these pieces of equipment. Coil winders and other generalizeable tools are almost certainly produced by coil winder machine manufacturers, but a lot of the lines seemed to have been assembled for the specific lines, a-la these bits of kit.

Your browser does not support the video tag.

Brushes being mounted onto rotors, motor lead preparation, and stators being mounted to rotors.

This would mean that this factory has a (proably small) team of machine builders and maintainers. They, not the particular technologies, are where automation comes from and I think we forget about them and their relative abundance / sparsity when we think about how much automation could be going on but then notice that actually not very much is.

People and their HMI.

I doubt that these engineers / mechanics are not mechatronics grads or “tech” workers like we would imagine them to be in the west. They are hands on, greasy, inventive, and probably worked the lines that they later automated. I would be surprised to find a complete CAD model of any of these lines, but probably some bits and pieces.

Messy !

And some advanced testing equipment:

Your browser does not support the video tag.

Science !

There were also a good number of general-purpose three-axis gantries doing i.e. glue-ups and other odd jobs.

Your browser does not support the video tag.

XYZ Gantries for Goo Dispensing.

The orchestration of an operation like this is I think an immense task that takes many years to mature and maybe requires slow, smooth growth to come into being. This factory has been around winding copper onto laminations for twenty years: it’s hard to stand things like this up overnight. It is not sexy, but its existence as part of the ecosystem is giving direct rise to the explosion of humanoid and legged (dog) robot companies that are popping up in Shenzhen (each of which contains probably twenty-to-thirty of these types of motor, plus gearboxes). Not to mention those drones, and did you see the DJI headquarters at the start of this document? In a decade there will be a few more towers of that magnitude built by the winners of the humanoid robot race.

About those Street Sweepers

This has been fun to write down and I hope you can tell by this point that there is a point I am trying to snake out of my vacation-photos slideshow.

The scale in Shenzhen is huge and is very obviously real in a way that is undeniable. Though I only had glancing contact with it, it seems to be working well: people are ambitious and driven but are basically kind and fair with one another. A vast number of people make a living in the city and many of them are by now either as skilled as their western counterparts or they are tooled up to work jobs that simply don’t exist in the west. They are probably still working harder, too. The whole pie is getting bigger and everyone is scrambling to make their mark in the expanding territory rather than fighting tooth and nail over the scraps. There is an attitude there that I heard echoed a number of times by business leaders that competition simply involves doing better than your adversaries (rather than trying to build technological “moats” or manipulate other structural systems), and that is paying dividends. A huge amount of this was basically directed by the government so yes we can continue to say that some of it is “fake,” but it has been realized and fine tuned in a straightforward, rules based market economy.

There is obviously much missing from this ad-hoc attempt at explaining any of what I saw vs. what I have seen during my time in America. I will note that I have had the priviledge of seeing a number of factories there as well (special thanks to Manufacturing at MIT and the IPC for that). My Dad also spent his career in manufacturing and for the last twenty years I have watched as he has grown a small industrial bakery’s output from about 50 to 500 thousand pounds of bread per week. I have studied the nuts and bolts of automated mechatronic systems pretty intensely over the last eight years as a grad student. I am however not a real economist, I just read about it. I have not done substantial numeric analysis on the total size of these economies, markets, etc etc. None of this is either endorsed or sponsored by anyone, I am self motivated by the desire to understand where in the world my own craft will fit in - and whether it will be useful at all.

Back to the point: I think the boring parts of any project - economic or factory scale - are its bedrock. We traded those away for short term profits starting in the 1980’s and we also forgot about the most fundamental bedrock of all, which is in a well educated population. We won’t bootstrap our way to effective automation if the people on the factory floor can’t imagine how they might invent themselves out of their own jobs (and give themselves a better one in the process).

If we want to bring manufacturing back to the west, we shouldn’t be making moonshots for magical lights out factories: they don’t actually exist except for to make the most advanced and high volume products, and if you want to meet the people who put them together and brought them to life you probably shouldn’t go to the university and you definitely shouldn’t ask a software engineer (sorry to my software friends I still think you’re brilliant).

But what about those street sweepers ?

Your browser does not support the video tag.

New Age Sweeping.

NVIDIA stock holders reading this piece are thinking I might be a luddite that has never seen a large language model or reinforcement learning, or that I am doing the same trick where we mix up our linear projections with those nasty exponentials. I thought about this whenever I saw these urban roombas around in the city.

Your browser does not support the video tag.

Demographics Solutions ?

Just about every country in the developed world has some nasty demographic issues coming up in the near future. Not to mention debt issues. By the time those dynamics come home, Shenzhen may have built more roads than it has dudes with brooms. The same is already true for American infrastructure. An LLM can’t sweep the floor or grease a wheel but a robot can, and thanks to the last thirty years of hard work they will be flush with robots in China. If we don’t get our act together we will be buying them at a mark up or we’ll be getting the robot squeeze in the same manner as we’re currently being rare-earths squeezed: we’ll realize we need them but we won’t be able to make them. Not because we don’t have enough GPUs or AI researchers or datasets, but because we won’t have enough motor factories, magnet wire plants, neodymium, aluminum foundries and extruders, screw machines, etc.

We could maybe get the robots to build all of that, but whether or not a robot with an LLM for brains can debug or build a factory is a question that I hope this tour of the messy reality of manufacturing might help you to answer. I don’t even want to hedge against the possibility of AGI because planning on that rather than doing our homework is tantamount to economic suicide. It additionally proposes that the AGIs won’t need any machinery, warehouses, or feedstocks to work with.

It would also be worth asking about who will own those robots and tools, but if we want to participate in that discussion we aught to start by being able to build them.

The Blair Winch Project

Mon, 09 Jun 2025 00:00:00 -0400

Deployable robots in the woods!

Drawing in space!

Long exposure linework.

I got lucky again this year and was able to join in Haystack Labs, which is a craft school / summer camp that Neil (my advisor) and James Rutter (Haystack Fab Lab Director) temporarily repurpose (for six days) as a kind of craft x digital-fab experiment. The gist of the experiment is to ask: why is it that digital fab and craft seem mostly like oil and water, and what can we do to emulsify them?

I think that mixing craft and digital fab is a kind of generational task… many of my favorite collaborators and advisors (Ilan Moyer, Leo McElroy, Nadya Peek, and Quentin Bolsee) are making big contributions here. I have always hoped that my own work will contribute along this front, making it easier to make interactive tools with human-and-process feedback across multiple levels, but for all of my efforts, the tools I have developed have yet to hit the level of reliability and usefulness that they would be appropriate to deploy in these messy environments. I have resolved that I will have to leave grad school and gather a team in order to fulfill that dream.

It was my fourth time at Haystack - in the past I have made a clay 3d printer and another one… and this weird pots-and-pans machine. This time around, I decided to try to build something wondrous - to do machine building as a craft, in a shamelessly self indulgent way. Art, babey… and the simple joy of making things that move.

What / Where

Haystack is a gorgeous place: the coast of Maine, an island reachable via an island, and woods, moss crawling over rocks. I was programming in the woods and it was joyous.

I wanted to make a machine that was lightweight, that we could install in an hour or less, and take away with no footprint. So: the winches.

One such winch, lashed to an unsuspecting tree.

I made four of these: torque-controlled winches that use my latest driver, and one of my favourite design patterns for a belt reduction. The spool does not have any ‘tending’ hardware, i.e. the rope simply winds on, so we can’t be completely sure what the exact diameter is at any given moment (not good) but the thing is pretty coarse anyways, so I eschewed the complexity that would have entailed.

Rope enters the spool through a small teflon o-ring, so that the angle of entry doesn’t change the friction too much, and this entrance (… known colloquially as the butthole) is positioned about 150mm from the spool, in an effort to have the rope coil up nicely on the spool.

I also designed the orb - a sphere with four entry points, and interior mounts for some RGBW LEDs. We stuck an ESP32 in here and a 21700 LiPo cell, so that we can change the light’s color and intensity remotely over WiFi.

There’s a flap on each winch with a limit switch, which we use to home the machine… this little bonk might be my favourite part of the whole project.

The homing / geometry discovery sequence: last three motors are shown.

The orb has a ~ spooky ~ vibe, hence the Blair Winch...

Homing, and Learning Where the Winches Are

Credit goes to Quentin for figuring out how this should work, and implementing it in a routine that I could plug into the controller.

Homing the winches is simple-ish: we startup each motor, and then run them sequentially with three (idling motors) under some minimum torque (to keep lines straight), and the fourth (reeling motor) at a larger torque. Then we pull motor encoder measurements (which we can correlate to rope lengths), as the chosen motor reels the orb in towards the limit. Once that limit is hit, we swap motors so that the next is reeling and the rest are idling. We do this four times, so that each motor is the reel-er once.

After the homing routine is done, we have a big system of equations to solve. The main issue it that we don’t know where the winches’ rope-entry points are, we’ll call those $(w0 \ ... \ w3)$. At each time step in the homing routine, we have rope lengths (measured) for each winch $(r0_t \ ... \ r3_t)$. We also add a long list of unknown positions of the orb for each time step, $(orb_t)$

We assume that at each time step, the distance of the orb to each winch is matched to the measured rope length… i.e.

\[\| \overrightarrow{wN - orb_t} \| \approx rN_t\]

This gives us a pretty simple loss function: for each time step, some $orb_t$ should exist that is $ropeN_t$ distance from each $wN$ position. If we apply that loss to each time step, we find $wN$ positions that are consistent through time (… the winches don’t move). So, sort of oddly, to solve for the winch positions we also have to figure out what the whole homing trajectory looked like.

This is a simple loss, but there’s lots of data - normally 1000 or so points along the homing trajectory. So we throw that over to scipy.optimize.minimize to chew through it.

count = 0

def loss_func(x, data):
    global count

    n_motor = 4
    tot = 0
    n = len(data)

    # winch positions
    xyz = np.reshape(x[:n_motor*3], (n_motor, 3))

    # orb positions: one for each time step 
    p = np.reshape(x[3*n_motor:], (n, 3))

    # guess is good if distance from winches to the orb 
    # matches the measurement at that time step 
    for i in range(n_motor):
        dist_sq_i = np.sum((p[:, :] - xyz[i, :])**2, axis=1)
        err_i = np.sum(np.square(dist_sq_i - data[:, i]*data[:, i]))
        tot += err_i / (n_motor*n)

    count += 1
    if (count % 1000 == 0):
        print(math.sqrt(tot), flush=True)

    return tot

In the end, this solved pretty well each time we ran it. I stuck it in a subprocess so that we could run it (takes a few seconds) after the homing routine, with the rest of the controller still alive and talking to hardware. The solver doesn’t solve in any fixed orientation (although we tried that), instead it just orients the solution randomly. Once the hard part is done, Quentin simply rotates the solution around such that w1 is at 0, 0, 0 and the vector from w1 -> w3 is vertical along z and w2 is somewhere in the XZ plane. Good enough for a first pass, probably we could also look at torques over time to ascertain the z / gravity vector, but not today.

To get a qualitative check for the solve, Quentin popped output points into blender, which let us check if the tetrahedron was ~ about the right shape.

The solver also returns the tetrahedron’s insphere radius and location, which is a useful place to move target trajectories into: points in the insphere are guaranteed to have relatively nice reachability, whereas i.e. points outside the tetrahedron are unreachable (obviously) and points close to the corners require using lots of torque on one of the motors…

With some help from Sebastian of morakana, we validated our tetrahedron solves by measuring vertex-to-vertex lengths.

An Aside: Solving w/ Unknown Offsets

As the orb approaches each of the stops, the geometry changes such that the opposing three motors exert more force on the orb: i.e. we are pulling “more away” from motors 0-2 when we are close to motor 3. This is obvious when you look at the geometry, and raises an issue with the somewhat underpowered winches: in the first setup we did, the tetrahedron made up by the winches was pretty far from an “ideal” tetrahedron… this made for a situation where the homing motor wasn’t able to reel the orb in completely. We tried initially so solve this problem away, by having our solver learn these offsets as well as the rest of the geometry, i.e adding an unknown offset $roN$ …

\[\| \overrightarrow{wN - orb_t} \| \approx rN_t + roN\]

Neither Quentin or I could get it to solve consistently. Probably it is possible, but it was beyond us at the time (although Quentin certainly got closer).

This shows the homing routine w/o offsets, where we flip motor directions once the orb has stopped for ~ 200ms. It should be possible to solve even with this data, solving also for the final offset in each direction, but that turned out to be a second-spiral task. This video also shows an attempt at control using only rope-length-setting output, which results in odd shaped circles due to the ~ nonlinear force that each corner of the tetrahedron exerts (and non-infinite torques).

To solve this problem ad-hoc styles, we added additional stops on the rope to make sure the limits were hit, and measured that offset manually. The second tetrahedron we made was also much more regular, so this problem went away (i.e. the orb actually hit the stop in each corner).

Controlling the Orb

If only I had internalized more of Gil Strang’s class…

Controlling the orb once we had the winch positions was more involved than I had thought. My first naive attempt was to simply take the target position, measure what the rope lengths should be at this target, and then run PID on the motor torques to hit that. This worked fine but did not produce very smooth motion, and led to some instability when the winch positions were not very well worked out.

I eventually developed a net force calculation, to push the orb along a particular force vector at each time step. This involves a few steps.

I did manage to get the orb more-or-less under control, here it is drawing a circle in space - slowly, surely…

1: Where is the Orb ?

We have a target position at each time step, and we have rope length measurements. To figure out where the orb is, we need to do forwards kinematics (actuators to cartesian). This can be solved using sphere intersections: each winch is the center of a sphere, and each sphere radius is the rope length. With four lengths, this is overconstrained, but if we just take a subset of the motors it is perfectly defined always (so long as all spheres intersect).

def sphere_sphere_intersection(c1, r1, c2, r2, atol=1e-9):
    """
    returns (center, normal, radius)  |  None
    """

    d_vec = c2 - c1
    d = np.linalg.norm(d_vec)

    # Degenerate centre-coincident case
    if d < atol:
        raise ValueError("bad circles")

    # No intersection if too far apart or one inside the other
    if d > r1 + r2 + atol or d < abs(r1 - r2) - atol:
        return None

    # Unit vector along the centre-to-centre line
    e = d_vec / d

    # Distance from c1 to the circle’s plane (along e)
    x = (d**2 - r2**2 + r1**2) / (2.0 * d)

    # Circle radius^2
    r_c_sq = r1**2 - x**2
    if r_c_sq < -atol:      # Numerical guard
        return None
    radius = np.sqrt(max(r_c_sq, 0.0))

    center = c1 + x * e
    normal = e              # already unit length

    return center, normal, radius

def fwds_kinematics(tetra_vertices, rope_lengths):
    # intersect 0, 1 to get a circle betwixt, 
    circ_0_1 = sphere_sphere_intersection(tetra_vertices[0], rope_lengths[0], tetra_vertices[1], rope_lengths[1])
    if circ_0_1 is None:
        # bonk, return center ... 
        print("sphere-sphere intersect fails")
        center, rad = insphere(tetra_vertices)
        return center 

    # intersect circle and third to get two points... 
    pts = circle_sphere_intersection(*circ_0_1, tetra_vertices[2], rope_lengths[2])
    if pts is None:
        print("sphere-circle intersection fails")
        center, rad = insphere(tetra_vertices)
        return center 
    
    if pts.shape[0] == 1:
        return pts[0]
    else:
        # pick tallest... 
        return pts[np.argmax(pts[:,2])]

I learned this trick while helping the HTMAA Architecture Group finish their machine. Works great.

2: Move the Orb

Once we know where the orb is, we can apply some force to take it where we want it to go. In this case, I simply chose a ‘requested’ force vector that was scaled with the orb’s current distance to the target: when further away, we ask for more juice, and when closer we dial that back. PID control, the quickest of hacks.

Then there is the matter of picking which motors to exert which torques. We know the orb position and the winch positions, we have four rope vectors that can exert some force (but only in tension), so we have to find a set of tensions that sums to the desired direction (and magnitude).

To control using ‘net force’, I calculate a set of tensions that best combine to produce some desired force vector on the orb. This would be simple if we could produce unlimited torques, but we sometimes don’t have enough juice to reproduce the desired vector exacly.

Were our motors infinitely torquey, this would always be possible - but they are not. To add complexity, we always have to exert some minimum tension to keep the lines taught (otherwise our rope length measurements get wacky).

I tried and tried to work this out using i.e. vector projections, but couldn’t quite nail it. I tried to summon my inner Gil but failed. Instead I used a least-squares fit that minimizes error to the target direction, ye olde have-someone-else-who-wrote-the-python-routine solve the problem for me: praise to the tools-builders, for they are truly His disciples! To check how much trouble this was causing, I calculate the resulting force vector’s alignment with the target at each step, and report that if it’s eggregious.

# net force vector style control 
def do_control_net_force_dir(tetra_vertices, rope_lengths, targ_dir, tq_min, tq_max, debug=False):
    pos_now = fwds_kinematics(tetra_vertices, rope_lengths)
    dir_vec = targ_dir
    dir_vec_mag = np.linalg.norm(dir_vec) 

    f_des = dir_vec       # desired Cartesian force
    f_des_unit = f_des / np.linalg.norm(f_des) 

    rope_vecs  = tetra_vertices - pos_now
    rope_units = rope_vecs / np.linalg.norm(rope_vecs, axis=1)[:, None]  # (4,3)

    B = rope_units.T            # (3,4)
    t0 = np.full(4, tq_min)     # initial torques (all @ min) 
    f0 = B @ t0                 # initial net force 
    f_res = f_des - f0          # extra force we still need

    hi = np.full(4, tq_max - tq_min)
    sol = lsq_linear(B, f_res, bounds=(0, hi), method="trf")
    delta_t = sol.x
    tensions = t0 + delta_t 
    # tensions = np.clip(tensions, tq_min, tq_max)

    # to check 
    # project torques back along ropes 
    f_final = B @ tensions
    f_final_unit = f_final / np.linalg.norm(f_final) 

    # angle betwixt ? 
    cos_ang = np.clip(np.dot(f_des_unit, f_final_unit), -1.0, 1.0)
    ang_err = np.degrees(np.arccos(cos_ang))

    # Magnitude error
    mag_err = np.linalg.norm(f_final - f_des)
    rel_err = mag_err / np.linalg.norm(f_des)

    # track max-latest 
    global control_iters, ang_err_max, ang_err_max_at
    control_iters += 1 
    if ang_err_max_at < control_iters - 100:
        ang_err_max = ang_err 

    if ang_err > ang_err_max:
        ang_err_max = ang_err 
        ang_err_max_at = control_iters 

    # compare torque result w/ desired
    if debug:
        print("do_control_net_force ...")
        print("f_des (unit)     ", f_des_unit)
        print("f_final (unit)   ", f_final_unit)
        print(F"f_ang_err        {ang_err:.3f}\t (max {ang_err_max:.3f})")
        print(F"f_rel_err        {rel_err:.6f}")
        print("")

    return tensions, dir_vec_mag 

I found that this caused lots of problems with tightly bounded motor torques (i.e. if the minimum torque is big, lots of trouble), but when I deployed more motor juice and controlled with smaller force requests, it was mostly OK. Next go-around I will either build stronger winches or write a smarter controller.

There is clearly lots of improvement here: we can always exert some amount of force in the target direction, and probably direction alignment should be prioritized over magnitude matching - i.e. the higher level controller (a p-only controller, hah), should avoid requesting forces that would require the direction vector to skew.

This controller doesn’t do any lookahead either, a clear flaw. I am working on MPC style controllers for other projects, and their utility here is clear, especially since the kinematics effectively changes at each time step. Modeling the orb as a mass in space (and gravity) would improve things, etc etc. I think it is a good test case, and I hope I will get to wake it back up once more to improve. For now, the output is satisfying:

MIDI -> Orb

Besides light painting, we also ran a little demo / concert with the gang at Haystack. Char plugged an accelerometer into strudel (a live-coding synth environment), and piped accelerometer values to the orb controller via MIDI. The values were also used to change some beat arrangements and samples… I transformed this data into orb motion, and people gathered-round for a try piloting the orb / music.

What We Learned, What We Missed

More Power == Better

Despite what I thought was a pretty solid belt reduction and reasonable sized NEMA17 motor, we were still torque limited in many cases. I think there’s lots of room to improve the controller to overcome this, but it would still of course be preferrable to be able to absolutely throw the thing around.

The upper limit wasn’t actually the motor (we only ran it near ~ 60% available output) - it was the belt, a GT2, which would occasionally slip on the drive wheel (which is printed…). Moving to a GT3, and improving the printed geometry would help in this regard.

Control with Smarter Limits

When the controller was in a position when it couldn’t exert a net force in the right direction, it would apply the best effort to hit the target magnitude / direction combo, meaning the thing would wiggle off trajectory. A controller that scales up to to maximum magnitude, without violating the target direction, might perform better - but then we would be running PID with a nonlinear real output. A smarter version of this would involve some more of that direction-vector modeling in the main control loop… MPC for the orb would be very cool.

Learning with Torques to boot

These motor controllers can learn parts of the motor model, and this project showed me pretty well where the next step would be: with a good motor model, we could also model i.e. the friction at the rope input (as a function of entry angle), and the winch’s stiction properties.

We could also add data to our solver by looking at net torques - i.e. if we keep rope lengths matched, then dial one motor up (more tension), and run a controller on the other motors to hold length, we get some information that describes the motors’ positions w/r/t one another, since we can assume the force vectors along the ropes in these cases are summing to zero. We would need to account for stiction there, too, but it’s all more data to better constrain the solver.

Learning All The Time

Something I’ve been really interested in is developing a controller that is continuously learning… each new time step generates a new set of rope lengths, motor output torques, and positions… we should be able to continuously add those to our data pile and occasionally re-run / improve estimates of the systems’ properties.

Very fancy versions of that system would learn also where they are underconstrained and then try to operate in regions where data is sparse, to better characterize the whole system… and with enough data, also measuring i.e. motor and driver temperatures, we might learn about how i.e. heat saturation affects performance, etc etc.

Whereis Python Good for Control

In a lot of the work I do elsewhere on MPC control for fast machines, using python to build controllers can be a performance bottleneck. On the flip-side, those controllers are easier to develop. This system was controlled pretty slow (around 100Hz main loop), where python perf was absolutely good enough. The upshot is big in these cases: we can lean on our scipy.optimize and lsq_linear code blocks to quickly assemble relatively sophisticated controllers.

The real win would be if we could build portable blocks for the same: having developed a controller in i.e. python, quickly deploy it in an embedded build to bring the bandwidth up. This is one of the fantasies for the graph-style control architectures I have been perpetually trying to bring to life.

Linear Algebra is Neat and Sometimes Non-Obvious

I was surprised to find that even i.e. intersecting two lines in space can amount to ~ some 20 lines of numpy code - or i.e. intersecting two spheres, etc. Most of the linear algebra sub-problems in this project involved Quentin and I both reading lots of wikipedia pages, and then puzzling out how to implement those in numpy. Leo also pointed out the overlap between these types of solvers / routines, and his more generalized work on constraint solving.

Again towards more generalized controllers, I saw here the need to build good libraries for these commonplace tasks, and ~ geometry representations to feed them… yada yada

In Summary

An absolute blast of a project, and one that was on my bucketlist for grad school. A pleasure to do it with such an awesome gang of people around, who were interested in helping with each of the system components, and knowledgeable.

Major thanks to Haystack and to Neil for bringing me along. Go there if you can, it’s a wondrous spot filled with really good people.

Field Oriented Control with (almost any?) Stepper

Mon, 24 Mar 2025 00:00:00 -0400

What feels like ages ago when we were both at the CBA, Sam Calisch pointed out to me that it’s possible to control a stepper motor as if it were a brushless motor, using field oriented control, aka FOC - everyone’s favourite motor control algorithm. This was an exciting idea and I quickly built a circuit to carry it out and I made a decent pass on the project back in 2021 although I never managed to completely integrate it into a motion controller.

Since then I have some new hardware with better current measurement, and a better idea about how to integrate the thing with the rest of the motion control problem, so I am back to learn again.

In particular, I want to learn how to do this across generalizeable hardware - i.e. slap a controller onto whatever motor is lying around and transform it into a high performance servo. To do this, I need to understand and implement FOC, and then use the controller hardware to learn enough about whichever particular motor it finds itself on in order to control it successfully. In this post, I mostly accomplish the first task, and learn lots about the second.

The result of this effort: a NEMA14 stepper motor in closed-loop position mode, reporting torque in real-world units, using a motor model that was (partially) learned using the motor itself.

The closed-loop stepper board, and one of the same mounted on a NEMA17 motor.

Whether or not this is a good idea has been the subject of many a heated debate in the ~~dungeon~~ basement of the CBA. Proponents argue that steppers might have more low end torque than ‘normal’ brushless (BLDC) motors - probably due to their high pole count. This seems likely to me, but only in an intuitive way. It seems that they are almost definitely less power dense than BLDCs, but again all I can stand on there is intuition… steppers are built with a lot more magnet steel than seems necessary. Hopefully I will find time to look into (and write up) some more fundamental motor physics that would explain these trade-offs, but in practice I think they are often more subject to the availability of components for our project than first principles physics, and stepper motors are downright ubiquitous - and downright cheap. Hence the almost-any-motor framing for the project.

I have to make a shout-out to the mechaduino project, which is the first implementation I saw - I think there are many such projects out there, and I know this is not terribly novel of me. FWIW, I suspect that the real lack of closed-loop in open-source has to do more with the systems we use to integrate motors with motion controllers (and tune servo loops) than the availability of motor controllers themselves. I have done lots of work building higher-level motion controllers / planners (synchronization, look-ahead, and even model-predictive motion lately), but I wanted to come back to the low level and see how that might all connect to i.e. position PIDs.

So, in this post:

a brief explainer on FOC and its relevance / application to steppers
a tour through the hardware and some of the low-level controller components
- ADC aligned sampling
- current control in rotating reference frames
- encoder calibrations
- a simple position / velocity / acceleration observer
some motor data collection and modeling to find (kt, ke, kd)
learning how to use the motor as a sensor even when we are running open loop
some notes on integration with motion systems

Stepper FOC vs. Brushless Direct Current (BLDC) Motors

Electric motor architectures are proliferating, especially due to interest in EVs - but all share the same principle: the rotor has a magnetic field - normally provided by permanent magnets made of those geopolitically sensitive rare earth metals - and the stator has electromagnet windings. We then use a computer to switch current through the stator windings, so that we can “point” the stator’s magnetic field in whichever direction we would like.

In all cases, what we want to do is drive the stator’s magnetic field so that it is 90’ out of phase (left or right) from the rotor’s magnetic field. Since the rotor’s field wants to align with the stator, the 90’ phase advance effectively pulls the rotor in that direction. Moving the stator’s field so that it advances ahead of the rotor’s field is known as commutation, and we call the reference frame that rotates with the rotor the dq frame, with a direct axis that is aligned with the rotor’s magnetic field, and a quadrature axis that is 90’ offset from that axis.

In a motor with three phases, like BLDC motors or most induction motors, we have three magnet windings in the stator that make up a kind of set of basis vectors. By varying the current on these three windings, we can effectively point a current vector within in the stator.

The best explainer on stepper motor operation that I could find.

Even though they are fundamentally the same as any other motor, steppers can be confusing since they are ‘hybrid’ architectures, combining normal permanent magnet motor design patterns with ‘reluctance.’ This basically means the field generated by the permanent is carefully guided through patterns in the rotor’s laminations. If you’re unfamiliar, the video above is the best way to learn.

When we’re doing control, it’s almost better to not think about those finer details. What we really want to know is that the stepper has 50 electrical rotations for every mechanical rotation - that is, every time we swing the shaft around once [0-2PI] radians we rotate the magnetic field around inside the stator fifty times. There are four ‘full steps’ in each full rotation of the electrical phase, hence 200 ‘steps’ in a motor.

The image below via allaboutcircuits shows an effective way to think about a stepper motor, for control purposes. We have two windings - one ‘up and down’ (the ‘B’ winding) and one ‘side to side’ (the ‘A’ winding) (sometimes X and Y, depending on the parlance).

If we only drive current from left to right on the horizontal winding, we are putting all of our current at zero degrees. If we only drive current ‘up’ on the vertical winding, we are putting current at ninety degrees… and so on: right to left = 180 degs, down = 270 degs. We combine components of these to drive current at whichever angle we would like.

We can effectively think of stepper motor stators as an electromagnet with two windings: one pair side-to-side (A) and one up-down (B). These are basically basis vectors A, B (or X, Y depending on who you are asking) that we can use to control the direction and amplitude of the magnetic field within our stator.

So, besides the particular layout of the hardware, FOC applies just about identically to the stepper. In fact the algorithm is smaller, since our component vectors are a simple orthagonal set (rather than the 120’ separated phases in a BLDC). That said, FOC in a stepper probably needs to commutate faster than in a BLDC (since the phases are so tightly packed), and steppers tend to have higher inductance, meaning we might want more voltage to swing the things around.

The Hardware

My FOC-stepper board, with current sensing and PWM pins bodged out for debugging. Microcontroller (SAMD51) and an interface for comms is on the other side. It’s a busy fit on a NEMA11, but well sized for NEMA14s, and can probably drive up to NEMA23 windings.

Some specs for you:

MCU: SAMD51J19A, 120 ~ 200MHz, ARM-M4F
Encoder: AS5047P, 14 bit, on-axis magnetic
Drive: 2x DRV8251A, 4.5-50v, 4.1A peak, 450mOhm, current sense out
I2C Port
Fan Port
Motor Thermistor Plug
Limit Switch Plug

The board is part of a family that I am calling knuckles - that share physical link hardware with a swappable PHY spec, TBD on releasing that but it basically means that we can put whatever connectivity we want onto this thing: USB + USB PD for development is pretty handy, I also have an RS-485 adapter, and should develop CAN and BLE versions, etc.

Front and Back, from KiCAD. The group of 12 labelled pads is the interface for a modular PHY layer that I am developing.

Some of of the’knuckles’ family of modular circuits, showing also the small embedded message passing and power distribution device (top center).

This fits ish on a NEMA11, and fits well on NEMA14 up to 23, where I suspect that the drivers would start to be a little bit heat challenged. I did put a small fan driver stage on the board for active cooling, and with the right enclosure / heatsink I think NEMA23 is totally manageable - at least the shorties.

TODO: pics of the circuit mounted on motors !

The board is four-layers, the two internal layers were aspirationally all GND plane but I had to cheat a little and route some lines under the microcontroller. This layout was tricky because I was really trying to pinch the thing onto a NEMA11, but I think I worked it out well. I do love a good routing challenge even though it normally takes me ~ four days instead of the two that I budget.

The layout. This one was tight, and it could probably use more bulk capacitance, but in the end it all fits OK. I did have to get a little fancy with the internal layers - only time will tell if I committed any GND plane sins.

The schematic… thanks to the DRV8251A, and to the modular PHY, it’s pretty simple: encoder, 2x hbridge, voltage regulation for the MCU, thermistor, some plugs, and the fan driver… RGB LED as a treat. I’ll publish the harware properly someday, I promise.

I picked the DRV8251A because of it’s specs (50V, 4.1A, nice!) but mostly because of it’s feedback. The bridge contains a current mirror output on the IPROPI pin that outputs 1575uA/A of current flowing through the bridge. By sticking a resistor on that output (to make a voltage that is proportional to the output current), we can measure that on our microcontrollers’ ADC.

The DRV8251A block diagram. Mostly a standard hbridge, but including this current mirror, which we can use to measure current in the motor stator (and so, the magnetic field). Note that the one output pin IPROPI combines inputs from two measurement devices ISEN1 and ISEN2 … this will cause us to have to do some tricks later on.

The FOC Algorithm

The controller is really two main loops. Key inputs are encoder position and current readings (left top, bottom). We use encoder readings to correct for position disturbances (or hit target speeds / trajectories etc) (top loop). We also use it to commutate the motor (bottom).

FOC’s power is that it works well across a wide range of speeds, and uses the entirety of the motor effectively… it is smooth, torque-ey, and silent when implemented well. However, it requires that we have an accurate reading of the rotor’s position and readings of the current in the windings - both of which require additional hardware (as opposed to i.e. an open loop driver). FOC also requires that we tune a few different controllers: the PI controllers in the current loop, three gains in the rotor position observer (more on that later), and PIDs for position or speed loops that will get rolled on top of the core FOC driver. These are some of the things that we want to tune either automatically or with some information from a motor model.

The Magic: Current Control in a Stable Reference Frame

As mentioned, FOC works by placing target current on the quadrature axis of the rotor. We can’t directly control current in the stator, only voltage. This means that we need to sense current, and write a controller that dials voltage up and down in order to hit the right current. This would be enough to get started, but we would end up with rapidly changing current targets - as the motor spins, the target current vector rotates as well. This is where the magic of FOC is: it lets us do that current control in a reference frame that rotates alongside the rotor.

Via ResearchGate ‘Review on FOC of Induction Motor’ - the dq frame is aligned with the rotor: d axis current doesn’t do any work, but q axis current generates torque. Open loop steppers blindly rotate current in circles and take for granted that the rotor follows pretty closely, but lots of energy is wasted because only the misaligned current does work.

Via ‘FOC Implementation using STM32F103…’ (doi). This diagram depicts FOC for a three-phase BLDC, our two-phase stepper doesn’t require the Clarke transforms, just the Park transform and its inverse. FOC basically lets us control motor currents in a stable reference frame, i.e. the frame that moves alongside the motor’s rotor. Without these transforms, our current controller’s setpoint would be rapidly swinging around, making tracking much more difficult - and probably impossible at high speeds.

While I was working this out, I made a small test code in python to help make sense of the transforms before I wrote them into the firmware.

steps = np.linspace(0, 3 * 2 * np.pi, 1000)

x_in = np.cos(steps)
y_in = np.sin(steps)
# t_in = (np.arctan2(y_in, x_in) - np.pi * 0.6) % (2 * np.pi) - np.pi 
t_in = np.arctan2(y_in, x_in)

def xy2dq(theta, x, y):
    d = np.cos(theta) * x + np.sin(theta) * y 
    q = -np.sin(theta) * x + np.cos(theta) * y 
    return d, q

def dq2xy(theta, d, q):
    x = np.cos(theta) * d - np.sin(theta) * q 
    y = np.sin(theta) * d + np.cos(theta) * q 
    return x, y 

d, q = xy2dq(t_in, x_in, y_in)
x_out, y_out = dq2xy(t_in, d, q)

My transform test. At the top, two example ‘measurements’ ‘x_in, y_in’ and the reference angle ‘t_in’… these are transformed into the stationary dq frame (middle) and then back (bottom).

I called these transforms simply xy2dq and dq2xy, (or ab2dq …) but they are properly known as Park and Inverse Park. In FOC for a stepper, we don’t need the Clarke components since we already have two orthogonal basis vectors.

Current Sensing

I did the usual hardware setup to read the encoder and write some basic PWMs into the hbridges, and then got into measuring current. My ambition was to simply spin the thing around in open loop voltage mode and then look at currents coming back, to start to piece together the rest of the controller. In particular I was curious to see what the phase delay between voltage and current was, since I figured there was lots of motor physics to learn with that data.

First trace I saw on the scope, CH1 (yellow) is an inactive debug pin, CH2 (blue) and CH3 (purple) are IN1 and IN2 on one of the hbridges, and CH4 (green) is IPROPI

This was encouraging: with previous hardware, current sense output was really noisy, and this is nice and clean. There’s a brief switching time (2us here) when I first switch the hbridge on (CH2 goes HI), then we see current rising in the winding, before it goes off again. But we see these nasty current spikes at startup and at the end of the pulse.

I hadn’t looked very carefully at the hbridge’s ‘truth table’ though, and at this point I was actually driving the thing into high-z state during the PWM off times. This would brake the motor excessively - it’s also known as fast-decay - it totally shuts the stage off, and causes the windings to generate lots of voltage… basically, we spun them up, and now there is a magnetic field in the windings that can’t just instantly turn off - so voltage rises as the energy tries to fight its way out of the winding. What we really want is to let the little current pulse that we created during the on state to keep looping around in the windings, it’s free real estate. This is kind of similar to using a flyback diode on a low-side switch: we give current in a winding a place to go during PWM off-times.

The hbridge ‘truth table’ - when we switch IN1 and IN2 to LO, we put the bridge in ‘fast decay’ mode - instead, we want the thing to always circulate current through the two low-side fets, and then turn one-or-the-other high-side fets on momentarily to generate more current.

The hbridge current paths, corresponding to the IN1/IN2 states above.

So the real strategy here is to set both pins HI when the ‘voltage’ request is LO and then flip one side LO when the voltage request is issued in that side of the bridge’s direction - this little inversion is one of many small gotchas/headaches in this kind of firmware and basically requires that we write the PWM driver by hand. So it was into the datasheet for me.

In this firmware I’m using the SAMD51’s TCC module - Timer/Counter for Control which is a relative joy to use, but probably only because I am already familiar with the chip. Here I simply flipped the polarity of each output channel on IN1 and IN2 and then I operate them based on a signed pwm request value.

void apply_pwm_hbridge_a(int16_t duty){
  // no -ves into the TCC 
  uint16_t abs_duty = abs(duty);

  // flip dir. based on sign 
  uint16_t cc_in1, cc_in2 = 0;
  if(duty > 0){
    cc_in1 = abs_duty;
    cc_in2 = 0;
  } else {
    cc_in1 = 0;
    cc_in2 = abs_duty;
  }

  // write to hardware 
  TCC0->CCBUF[HBRIDGE_CC_CHAN_A_IN1].reg = cc_in1;
  TCC0->CCBUF[HBRIDGE_CC_CHAN_A_IN2].reg = cc_in2;
}

CH2 is PWM into IN1 and CH3 is PWM into IN2 - CH4 is the IPROPI pin at the ADC. I am driving a triangle wave -ve -> +ve, and we see positive current measurement in either direction. The small aberrations are me rotating the shaft by-hand - this changes inductance in the stator (and generates some BEMF voltage), so we can see the effects on the current trace.

As above, but zoomed in. We can see the short ‘on’ pulse happens when IN1 goes LO - the current rises during this interval. Then, in the slow-decay state, the current gently decreases, before it is lifted again at the next pulse…

The plot above is what we want to see with a motor controller: the motor is ‘basically a big inductor’ that is filtering the pulsed voltage (blue) into a steady stream of current (green).

PWM Aligned ADC Sampling

Looking at the current on the scope is cool and all, but we want to read that into the microcontroller to do control with it. So here comes one of the tricks we can pull with a microcontroller that simply isn’t possible on more general computing devices: we configure the ADC so that it samples exactly when one of the TCC channels is triggered.

This SAMD51’s TCC has 8 channels in total, and I am using four already for the PWMs (they are configured as outputs). I can setup an additional channel to trigger one of the SAMD51’s event channels - these are like little hardware dataflow lines that we can use to communicate from one peripheral to another. That event channel is then connected to the ADC, where it triggers a sampling event.

This is a bit of a PITA to setup, but the timing is important. Firstly, it helps us to sample away from any noisy spikes on the line during the switch, and it lets us sample when we know what states the gates are in - this last part is more important.

CH1 is a debug pin that I toggle in an interrupt activated by the event channel that triggers the ADC reading… so, it flips when the ADC is triggered. Here, I trigger it just after the IN1 gate goes LO.

I spent a while trying to get this right.

Originally I was triggering the ADC in the middle of either side of the PWM phase, i.e. halfway through the ‘off’ part of the cycle or halfway through the ‘on’ part of the cycle - I thought that due to the slow decay, I would always be measuring valid current and the precise timing was just useful to avoid noisy transitions.

The current reading is unsigned, as in we don’t know which way it is flowing through the hbridge when we sample the IPROPI pin if it’s in slow decay because the measurement is taken from whichever of ISEN1 or ISEN2 is largest… recall from the earlier block diagram of DRV8251A. I had figured that, looking at previous states from the system, it would be easy to reason about the direction of the current - but I soon realized that I needed a way to deterministically ascertain the sign of the reading…. the current loop is far too fast to be messing about with observers etc.

The real solution is somewhat counter-intuitive: during every pwm cycle, we need to flip the hbridge on in both directions for some amount of time. For example, if we are driving the bridge FWD at 20% duty over a 1040us period we drive:

IN1=HI, IN2=LO (forward) for 220us, sampling the ADC (reading is FWD signed)
IN1=HI, IN2=HI (coast) for 800us,
IN1=LO, IN2=HI (reverse) for 20us, sampling the ADC (reading is REV signed)

Basically, we insert this minimum on-time for both directions, and sample the phase at the instant where we know the sign of the current measurement. This required that I re-write the PWM driver and the ADC sampling events.

Traces from a Saleae logic analyzer, showing PWM to IN1 and IN2. The bridge here is driving in reverse: IN1 spends more time ‘down’ than IN2 does, driving current backwards through the bridge on average. We additionally pulse the bridge FWD for a short duration, during which cycle the IPROPI output (red) flips to show us how much current is flowing in that direction. The ADC is sampled during the middle of both pulses (sample timing is illustrated with the ISR line toggles).

With two measurements, one in each direction, I could finally ascertain the signed current measurement from hardware, no fooling required. In these plots, you can see the phase delay between i.e. pwm_a and cur_a…

To set this up in hardware, I flipped the PWM’s WAVEGEN mode to use dual-slope pwm - this is a mode where the TCC counts up until TOP, and then back down to ZERO (figure below). I then flip the polarity of the second output channel, so that it goes LO around the TOP of the wave, and the first channel goes LO during the bottom of the wave (eliminating any overlap of the two LO signals, which would put the bridge into fast decay mode).

Then I have to trigger the ADC at both pulses. For this, I generate an event out of the TCC’s overflow (OVF) event, which (in this pwm mode DSBOTH) is fired at both the top and the bottom of the wave. That even triggers the ADC sample, and then an ISR picks up the readings and stashes them (and reads the TCC’s COUNT register to ascertain which side has just been sampled) - they’re consumed in a subsequent ISR that runs the current loop.

The SAMD51’s Dual Slope PWM mode, (datasheet: 49.6.2.5.6). I inverted pins such that IN1 to go LO at TOP and IN2 goes LO at ZERO - this eliminates a fast-decay state that would occur during the small overlap that would occur if I used ‘normal’ PWM. I also use the TOP and ZERO overflow events to trigger ADC readings at those specific moments.

I tried to set this up with the minimal number of ISRs, since switching contexts is costly. Both ADC0 and ADC1 are both setup on the D51, with ADC0 sampling one hbridge and ADC1 sampling the other. Since I am using one TCC to control both sides of the motor (via four output channels to two hbridges… two channels each), the samples (and ISRs) are aligned across hbridges, meaning I can use one ISR to collect both samples.

Interleaving Other ADC Readings

For one last gotcha, I had to get the other ADC readings to happen in the middle of all of these events. Most microcontrollers only have a few real ADCs, that are ‘muxed’ onto many different input channels. For example in an arduino with nine ‘analog in’ pins, the MCU might only have one actual ADC. When we tell arduino to sample on (1) and then on (2), it actually reconfigures the ADC when we call the function, then triggers a sample and waits for it to complete. This all takes up valuable time, since the conversions don’t happen in one clock cycle (it can be 50 or more clock counts).

To avoid these hangups, I added a modal switch that swaps the ADC channels to my three other analog inputs on the board (on-board thermistor, off-board thermistor and a voltage divider on VCC). Right now this happens only two or three times a second, which is plenty fast for those readings. I should improve this later, as it briefly disables the current readings during those intervals (the controller just uses the previous measurement…). To do better, I can sneak a third sample in without interrupting the main current sampling instants.

The thermal information is slow, but the VCC might be interesting to watch at higher frequencies - doing so could be a good indicator that more bypass capacitance needs to be added, for example, and precise input voltage is important to measure if we are going to dynamically fit motor models.

Calibrating the Encoder

OK: we have current readings, so we can roll a current controller to point the magnetic field in whichever direction we choose. However! We need that pesky reference angle $\theta$.

This is why we have the encoder: it reads [0, 16383] (14 bits) and it’s absolute, so the readings are indexed in the same manner when we restart the controller. However, we need to know how mechanical angles correlate to electrical angles. Recall that we have 50 full electrical rotations for every mechanical rotation, and we have some offset. The full 14 bits are useful here, since it means we have ~ 328 counts per electrical phase. Some other common encoders for BLDC FOC have only 11 or 12 bits of resolution, i.e. 41 or 82 counts per electrical phase.

To gather data for this, I spin the motor around in open loop voltage mode - I sweep forwards and reverse at a few different speeds, and ingest time stamped readings of the speed, voltages, current, and encoder readings. Using that raw data, I can turn A, B voltage and current readings into phase angles, and then plot them against the encoder angle, that’s below.

The electrical phase of the current is the best estimate of the rotor’s actual position during these samples, since that’s where the magnetic field is pointing, and we are assuming that the rotor is tracking that. However, it’s easy to spin the field too quickly and loose rotor tracking, so the calibration routine needs to discriminate stalled rotors and eliminate those data points. It’s also important that we use an equal number of samples from each speed and direction, since the rotor actually lags the current vector by some amount - more when the thing is moving faster.

Encoder readings potted on x, with voltage and current phases (grey, black) plotted on y. We want to fit a function that returns the rotor’s electrical phase for any given encoder reading.

This is a sensible way to look at the data, since it’s basically the function we want to fit: for any encoder reading, return the electrical angle. But fitting it in this modulo’d space is insane, although I will admit that I tried to do so for a minute. Quentin ended up helping me out, and showed me np.unwrap(...) which unrolls both sets of periodic data, where they look like more normal functions. For convenience, we also do the fit in [-pi,pi] on both axes (i.e. normalizing the encoder readings). Then we can fit them with a little scipy.optimize.minimize …

# returns electrical phase, given encoder phase 
def fit_func_linear(params, enc_phase):
    return enc_phase * params[0] + params[1]

def loss_linear(params, enc_phase, tru_cur):
    fit = fit_func_linear(params, enc_phase)
    err = fit - tru_cur 
    return np.sum(np.square(err))

x0 = np.array([50, 0])

res = scipy.optimize.minimize(loss_linear, x0, args=(encs, curs), tol=1e-12)
xf_lin = res.x 

Errors after a linear fit - looks a lot like CORDIC error.

The linear fit is pretty good, but has this odd looking error… there’s a sinusoid in here at twice the frequency of the encoder phase, and one at the frequency. As it turns out, this is probably mostly error from the AS5047’s CORDIC unit, which is an old computational / hardware library that approximates the atan2 function - something I learned as we investigated this error.

Via wirelesspi.com - Coordinate Rotational Digital Computer (CORDiC) is a set of lightweight trigonometry approximations for embedded devices and ICs. The atan2 function is particularely common / useful for translating between cartesian and polar coordinates… but it has this error, which looks about identical to the error term that drops out of a linear fit from our encoder.

Block diagram for the AS5047 encoder, showing the ATAN CORDIC block, which is what the encoder uses to translate A, B magnitudes from its hall sensors to an angular reading.

Probably the best way to do this would be to draw up the actual CORDIC error pattern as a function and then fit parameters for offset and scale of that to this data, but instead we just stuck two more sinusoids in to fit: one at 2x the encoder frequency and one at 1x. This worked great, or at least well enough for now.

# returns electrical phase, given encoder phase 
def fit_func(params, enc_phase):
    lin_fit = fit_func_linear(xf_lin, enc_phase)
    major_period = params[0] * np.sin(2 * enc_phase + params[1]) + params[2] 
    minor_period = params[3] * np.sin(enc_phase + params[4])
    return lin_fit + major_period + minor_period

def loss(params, enc_phase, tru_cur):
    fit = fit_func(params, enc_phase)
    err = fit - tru_cur 
    return np.sum(np.square(err))

x0 = np.array([1, 0, 0, 1, 0])

res = scipy.optimize.minimize(loss, x0, args=(encs, curs), tol=1e-12)
xf = res.x 
final_loss = loss(xf, encs, curs)

Including two more sinusoids into the fit (one at 2x encoder frequency, and one at 1x), we get rid of most of the rest of the errors. Those noisy little spikes are a bona-fide source of noise in the system: the SPI transaction with the encoder seems to occasionally suffer from one-flipped-bit… I haven’t tracked that down yet.

Now that we have our fit, we can stick it back in hardware. For now what I’ve done is just had numpy pretty-print a table into the console that I copy and paste into a lookup table (LUT) style thing. I do the modulo arithmetic in python as well - and re-scale everything to the units that I’ll be using in the embedded system.

// in lut_encoder.h 
const int16_t LUT_ENCODER_2_ROTOR_ELECTRICAL_PHASE[16384] = { 
  //...
  //...
};

Periodic data as-above, now with 'le fit'

LUT Trigonometry for the Transforms

Speaking of units and data types, we now have to do a bunch of arithmetic in embedded, and so we are going to add even more lookup tables. If you recall the controller outline above, you’ll notice that i.e. native data types for encoder and motor phase are [0-16383] …

Note elec. phase LUTs are [0-16383].

Well! That’s because we are doing trigonometry in here with more LUTS. This is a pretty common trick so I won’t go into too much detail, I did find a neat new LUT generator here to use…

For those unfamiliar, the gist is simple: we take some function $f(x)$ and then discretize it across some integer range… in this case mapping $x \in [0, 2\pi] = [0, 16383]$ and then we calculate all of the possible results ahead of time and simply write them into memory, in a lookup table. Using i.e. const ... will tell the compiler to write that into Flash (not RAM), saving some valuable space. Then to evaluate, we simply say $f(x) =$ LUT[x].

It is particularely useful with periodic functions aka sin/cos, where the input can wrap (i.e. $[0,2\pi] == [2\pi,4\pi]$) to set the span of the input at some power of two: i.e. in this case the input is 14 bits. This means that we can use a bit mask to replace the modulo operator we might otherwise need…

/** Generated using Dr LUT - Free Lookup Table Generator
  * https://github.com/ppelikan/drlut
  **/
// Formula: sin(2*pi*t/T) 
const float_t LUT_TRIG_SIN_LUT[16384] = {
  ///
  ///
};

// ingests theta = [0 -> 16383] == [0 -> 2PI] 
// returns float [-1.0 -> 1.0]
float lut_trig_sin(int16_t theta);

// ingests theta = [0 -> 16383] == [0 -> 2PI] 
// returns float [-1.0 -> 1.0]
float lut_trig_cos(int16_t theta);

typedef struct{
  float d;
  float q;
} lt_dq_t;

typedef struct {
  float x;
  float y;
} lt_xy_t; 

lt_dq_t lut_trig_xy_2_dq(int16_t theta, lt_xy_t xy);

lt_xy_t lut_trig_dq_2_xy(int16_t theta, lt_dq_t dq);

// we can wrap 14 bits using a mask, meanig we do periodic maths 
// implicitly / bitwise - fast ! 
#define LUT_TRIG_THETA_WRAP_MASK 0b0011111111111111

float lut_trig_sin(int16_t theta){
  // wrap for 0-16k 
  theta = theta & LUT_TRIG_THETA_WRAP_MASK; 
  return LUT_TRIG_SIN_LUT[theta];
}

float lut_trig_cos(int16_t theta){
  // same wrap, shift phase by 90' 
  theta = (theta + LUT_TRIG_HALF_PI) & LUT_TRIG_THETA_WRAP_MASK;
  return LUT_TRIG_SIN_LUT[theta];
}

Then the transforms themselves are pretty simple - using a custom struct for dq, xy frames - and (since the SAMD51 has an FPU) I do most of the rest of the maths using floats - nice and simple.

lt_dq_t lut_trig_xy_2_dq(int16_t theta, lt_xy_t xy){
  lt_dq_t dq;
  dq.d = lut_trig_cos(theta) * xy.x + lut_trig_sin(theta) * xy.y;
  dq.q = -lut_trig_sin(theta) * xy.x + lut_trig_cos(theta) * xy.y; 
  return dq;
}

lt_xy_t lut_trig_dq_2_xy(int16_t theta, lt_dq_t dq){
  lt_xy_t xy;
  xy.x = lut_trig_cos(theta) * dq.d - lut_trig_sin(theta) * dq.q;
  xy.y = lut_trig_sin(theta) * dq.d + lut_trig_cos(theta) * dq.q;
  return xy;
}

Filtering Encoder Readings

By now all of the nuts and bolts are together, but the larger challenge of controlling the motor well is still with us. One of the things that has caused trouble for me in the past is de-noising the position reading from the encoder, and also estimating the thing’s velocity (and acceleration, if possible) using that data.

Position is obviously useful for closing position loops, but velocity and acceleration are great to have. For example in FOC, we can use the velocity to predict the rotor’s position in between encoder samples - so i.e. if we are sampling our encoder around 10kHz but we want to run our current controller at 30kHz, we need two extra predictions. We might also want to replace the position PID with a proper LQR controller, and velocity measurement there would be critical… The PID controller is also making a continuous estimate of the position error derivative which is ~ at least highly related to velocity, so we might be able to filter once and use the information twice - etc, you get the picture, it’s useful to know how fast you’re going haha.

Taking discrete derivatives of position in order to calculate velocity is famously difficult to do without making really noisy signals, since we typically sample at high frequency and most of the change between measurements sample-to-sample is due to noise, unless we are going very quickly. We basically need to do some kind of averaging or filtering to get a sensible output, even with pretty good encoders.

Kalman4Idiots

I was originally trying to crack this by writing a Kalman filter and, while I could get ChatGPT to spit out a good implementation of one, I still don’t entirely understand what it is doing. This is a real kind of moment where I wish I had eaten more of my maths vegetables in the past.

I then wrote this weird filter that turned out to have really good performance - it keeps a velocity estimate and uses that to improve position estimates (taking i.e. 20% of the estimate and 80% of the reading, depending on a setting), and conversely uses the position estimate as a basis for the velocity estimate (which is filtered using a normie exponential filter).

class Kalman4Idiots:
    def __init__(self, dt, alpha_pos, alpha_vel):
        self.dt = dt 
        self.alpha_pos = alpha_pos 
        self.alpha_vel = alpha_vel

        self.vel_est = 0.0 
        self.pos_est = 0.0 

    def update(self, measurement):
        vel = (measurement - self.pos_est) * (1 / self.dt)
        self.vel_est = vel * self.alpha_vel + self.vel_est * (1 - self.alpha_vel)
        
        pos_predict = self.pos_est + self.vel_est * self.dt
        self.pos_est = measurement * self.alpha_pos + pos_predict * (1 - self.alpha_pos)

        return self.pos_est, self.vel_est 

This actually worked really well, in fact I could tune it to about the same performance as the Kalman implementation (with the bonus that I had an intuition for the knobs I was dialling). These plots are coming from tests using data that I generated in numpy with some artificial noise - so I’m not actually sure how well this would work on real data.

Performance of the Kalman4Idiots filter was surprisingly good.

A neat trick that I will probably stash for later - and that could presumably be extended into acceleration… but it works like shit when we have substantial changes to acceleration - unsurprising since it doesn’t have internal prior that acceleration might happen.

The same filter, but subjected to big changes in acceleration. The position estimate basically lags the real position by some amount that is proportional to the acceleration.

“Alpha Beta Gamma” Observers

I wound up learning (again) about observers - I originally saw these when I took Jacob White’s beginner controls class (IIRC it was 6.302)… I probably should have finished this project right after that class, but at least the seed was planted!

The basic idea is that we have a model of our system and we track its internal states. This one in particular is simple: we have pos, vel, accel, and the model just says that acceleration turns into velocity and velocity turns into position. At each time step we make a prediction, and we compare our prediction to the measurement. The difference is called innovation (unpredicted change!), and we can update our estimates using some amount of that innovation - i.e. three filter values (the alpha, beta and gamma) that are ~ related to our trust in the measurement.

class AlphaBetaGamma4Idiots:
    # alpha = position gain, beta = velocity gain, gamma = acceleration gain 
    # we assume position is slow to change (small gain), 
    # velocity is faster (larger), 
    # accel is nearly instant (arbitrarily big) 
    def __init__(self, dt, alpha, beta, gamma, vel_max, acc_max):
        self.dt = dt 
        self.alpha = alpha 
        self.beta = beta 
        self.gamma = gamma 

        # very-useful abs. bounds:
        self.vel_max = vel_max 
        self.acc_max = acc_max 

        self.pos_est = 0.0 
        self.vel_est = 0.0 
        self.acc_est = 0.0 

    def update(self, measurement):
        pos_pred = self.pos_est + self.vel_est * self.dt + 0.5 * self.acc_est * self.dt * self.dt 
        vel_pred = self.vel_est + self.acc_est * self.dt 
        acc_pred = self.acc_est # ... all change in accel is due to new info 

        # unpredicted change... 
        innovation = measurement - pos_pred 

        self.pos_est = pos_pred + self.alpha * innovation 

        self.vel_est = vel_pred + (self.beta * innovation) / self.dt 
        self.vel_est = max(-self.vel_max, min(self.vel_est, self.vel_max))

        self.acc_est = acc_pred + (self.gamma * innovation) / (0.5 * self.dt * self.dt) 
        self.acc_est = max(-self.acc_max, min(self.acc_est, self.acc_max))

        return self.pos_est, self.vel_est, self.acc_est 

I had a really hard time getting this to be stable until I added the vel_max and acc_max values… which are pretty easy to guess using system knowledge (and they don’t have to be too close to reality, just guesses with lots of overhead). Without them, what can happen is this:

we make a measurement that causes us to make a big acceleration and velocity innovation
the next cycle, our prediction is well ahead of the measurement, and now we dial it back in the other direction (overcorrecting)
etc, until we hit np.inf

This one I was ablt ot tune up pretty well to my test data, and the addition of an internal acceleration state (while noisy…) is cool.

Output from this alpha-beta-gamma filter, showing good tracking for position and velocity, with noisy (but available!) tracking of an acceleration state.

I went ahead and implemented this in cpp on the device, so now I have states available in hardware, although I haven’t properly tuned it, which is kind of future work at this point. That said, the gains should be about the same from motor-to-motor, since they are only related to the noise that comes from the encoder, and that shouldn’t change drastically. Tuning the vel_max and acc_max is similarely simple: we should basically cap them such that they don’t allow the system to explode, which can be calculated easily if we know the time step taken between measurement / update steps.

Current Controller PI Gains

To get to the next steps, we need to tune the current controller gains. I learned that this is possible if we know the motor’s coil resistance $R$ and inductance $L$ - both of which are on the datasheet for this motor, and can be measured using this hardware, although I haven’t implemented that step yet.

$k_p$ is the simplest, check it out: $k_p = R$ - seems too good to be true, but consider the units $V / A$ - i.e. how many volts should we apply to get one amp of current correction? That’s just ohm’s law.

$k_d$ doesn’t exist, basically beacause there’s no / negligible inertia in the system, thank god.

$k_i (V/A \cdot s)$ is messier - we want to pick a bandwidth for the controller first. We can use the electrical time constant to understand what a maximum bandwidth might be:

\[\tau = \frac{L}{R} = \frac{0.01}{6.8} = 0.00147s\]

The time-constant is how long it takes for the system to hit ~ 63% of its final state, after a step input, so $\frac{1}{\tau} = f_c = 680Hz$ represents the frequency that we can expect to swing the thing around. So, an upper bound on how fast our current controller can expect to operate. What still confuses me here is that this somehow applies in the rotating frame even though the underlying hardware is rotating much faster. As will appear later, my non-understanding of frequency-dependent impedance of the motor-as-and-inductor is causing some problems.

In any case we can pick this as a target bandwidth, and use $k_i = 2\pi f_c \cdot L = 6.8$ (!)

I will say, I don’t completely understand this step, but it works terribly well so I am leaving the comprehension for next time.

A Position PID Loop, and Loop Timings

Once the core FOC layer and observer was working, this step was pretty uncomplicated: position estimate comes from the observer and the PID controller decides how much torque to put on the q axis of the FOC controller.

Tuning the gains is another question: it is easy to make the thing work, but difficult to make it work well. I get the sense that tuning in-general (of the current controller PI gains, of the observer gains, etc) is the main challenge for closed-loop motion systems, and (besides cost, which is actually not that great) is probably why most simple systems use open loop motors, or ‘closed loop’ motors that only use their encoders to recover lost steps.

Here I’m periodically updating a position PID’s setpoint, and plotting torque that the motor is using to hit that target. The gains aren’t execellent, and the inertia is huge relative the actuator, but it at least closes the loop across the whole system.

I also tightened up my loop timings at this point. The current controller bandwidth is way higher than I think is probably necessary, but it is pretty cheap to run. The spline interp / position control is fast enough, but ideally we would actually poll the encoder at extremely high speed, since commutating the many-pole-pairs of the stepper motor requires really high frequency position information… in fact, in a couple of cases I saw current sinusoids on the oscilloscope that were only two or three PWM periods long - meaning the motor rotated through one-half electrical rotations in as much time.

30kHz PWM
30kHz Current Control
15kHz Spline Interpolation
15kHz Position Control

Via app note 103 / 102 from Oracle Robotics. Shows ‘typical’ loop speeds for closed loop servo control - although this is likely to vary application to application and across motor sizes, with smaller inertias requiring more responsive control. These intervals are substantially larger than those that I implemented, probably typical of larger motors with less pole pairs, but also means that I could probably get away with less Hz if I could tune these loops like the experts.

To give myself some more overhead for all of this, I am running the SAMD51 at 180MHz - a little more than the ‘spec’ 120MHz. Using Platformio and Adafruit’s M4 Feather Core, this was a really easy config change. Internet suggests it can run at 200MHz, but I think I have it in a somewhat noisy environment so I will leave it here.

I have two ISRs here, and interleave operations. Basically the whole controller runs in these ISRs besides the interface, and it takes about 25% of the processor’s time to do so. Not pictured are the ISRs that handle the SPI transactions for the encoder. Each sequence is triggered by the TCC’s overflow or underflow, which sends an event to the ADCs to start a sample. That event also triggers the first ISR (the tiny slice), which records pin states during the sample time. The second ISR is the ADC’s conversion-complete signal, where we wake up again and pick up the new data. Then we either (1) go back to sleep if we are still waiting for the second ADC sample, (2) run the current controller, the interpolator, and tell the encoder to start acquiring a sample, or (3) run the current controller and position controller, using the new encoder reading. The interpolation step is the longest (up to 10us) since it is evaluating a basis spline and potentially incrementing through a buffer of control points.

Measuring Motor Parameters, and some Modelling

OK, with hardware it normally takes a long time to sort out the nuts and bolts, but then we get to do the fun stuff. That is the point we are finally at here; we get to (try to) learn some parameters for the motor-under-test (MUT)’s model, so that we can report torques while it is in use… so that we can model our machine systems / motion systems while we operate them! Modeling / understanding motor parameters also helps us to tune some of those nasty gains without thinking too hard.

Robots measuring their own forces is sometimes called proprioception, a term borrowed from biology for ‘the sense of self-movement, force and position.’ - i.e. the sense you use when you’re reaching inside the cheeze-its box to find that last morsel is proprioception.

In the case of most electric motors, output torque is more or less directly proportional to the current that is active on the q-axis, but we need to fit a few parameters in order to know how this current relates to torque.

I want to add a caveat here that this is something I am still learning about - so … don’t trust my maths, basically.

The Canonical Motor Model

It’s beautiful, but lossy.

\[\ddot{\Theta}J = k_tI - k_d\dot{\Theta}\] \[I = \frac{V_a - k_e\dot{\Theta}}{R}\] \[V = RI + L\dot{I}\]

$V_a$ is the voltage applied to the motor, i.e. v_a = vcc_supply * pwm_duty
$\dot{\Theta}, \ddot{\Theta}$ velocity and acceleration (rads)
$Q = torque = \ddot{\Theta}J$ is the identity for torque (Nm)
$R$ is coil resistance (Ohms)
$J$ is inertia of the rotor and whatever else is attached (kg \ m^2)
$k_d$ is damping due to friction (force / speed)
$k_e, k_t$ are emf and torque constants
$L$ is motor inductance

This model is what you’ll find if you’re studying DC motors, i.e. the brushed kind. With FOC, we do all of this work to commutate the motor… so that we can ignore that part when we make these models. To disambiguate, in these models the $V, I$ terms are basically assumed to lie on the quadrature axis.

Once we fit each of these, we should be able to use $k_tI$ to measure output torque - since we can measure current directly. Eventually I want to measure each of these automatically, but for now my process to find the physical constants was thsi:

Measure $R$ with a multimeter (it’s easy)
Measure $J$ by drawing the motor rotor in CAD, and having that software calculate it
Measure $L$ by reading the motor datasheet

A real measurement of $L$ is probably the most important thing to improve the readings’ precision, same with actual $R$ measurements, since the datasheet sepcs are for $\frac{+}{-} 20\%$ (!). Even $R$ changes by about 10 percent coil-to-coil in the same motor.

I also found that adding some real inertia to the motor is basically required, since the rotor’s own inertia is normally so small as to be negligible in the face of the torque it can produce. That is, when we spin up just the rotor, the change in direction is so fast that we can’t really sample any of the dynamic states that happen during that turnaround. So: enter big disk of PLA, the world’s simplest dynamometer.

The motor’s own rotor has some inertia: for this tiny NEMA14, that’s about 0.000001878 kg m^2. Because this is so small, it was nearly impossible to get meaningful data for a model fit (although I did try): the rotor accelerates to its no-load speed in about 10ms, not enough time to gather very many samples. I bolted this giant PLA disk on (0.002888 kg m^2) to slow things down.

As an interesting aside, a common rule of thumb for motion systems design is to pick an actuator whose reflected inertia is about equal to the inertia of the component it is pushing around. The inertia that I printed to make sampling simpler is ~ 1000x bigger than the actuator’s rotor.

Spin-Up Data and Filtering

To fit the rest of the parameters, we want to make some time-series data of the system operating and then fit that to a model. For this, I put the motor in direct_q_current mode, where we can just tell it how much current to try to put on the q axis. Then we just… swing it around: apply maximum torque in one direction, wait for about ten seconds, and then flip directions, waiting for another 10-or-so. All the while we are sampling a set of the motor / controller states and stashing them for analysis.

Data collection is simple - I just spin a known inertia around using maximal applied current, flipping directions after some number of seconds and taking a few passes in either direction.

Time-series data from the test spin-ups. Note that while vcc_q moves rail-to-rail, the amount of current it generates is highly dependent on the rotor speed - this is back emf in action. Also interesting is to watch the vcc_rail signal (bottom trace): this is the power supply sagging when we pull current (it’s a USB-PD brick…).

The observer’s velocity estimate is noisy (and it’s acceleration estimate is almost useless), so I filtered velocity using some newly acquired scipy magic.

Once it’s filtered, we can take a discrete derivative of the velocity estimate to get acceleration data. These are still a little noisy, but serviceable.

We do a little filtering, etc (thanks scipy!) and then we can basically plot accelerations vs. speed, i.e. in ‘torque-curve’ land.

Looking at data from the spin-ups w/ velocity (x) and accelerations (y), with the rotational symmetry because we are spinning in either direction.

Fitting

To fit this, I’m just using scipy’s built in optimizer. We write a model with some params, then provide it with data and a loss function and if things aren’t too hairy it tends to work pretty well.

R = 6.8                 # ohms
J = 0.000001878         # kg m^2 (rotor) 
J = J + 0.00361 * 0.8   # adding le-diskus, plus print density ... 

def model(params, vcc_app, vel_now):
    ke, kd, kt = params 
    # assert that ke works against us regardless of speed dir. 
    ke = np.where(vel_now > 0, ke, -ke)
    cur_now = (vcc_app - ke * vel_now) / R
    acc_now = (kt * cur_now - kd * vel_now) / J

    return cur_now, acc_now 

def loss(params, vel_data, cur_data, acc_data):
    # hmmm 
    cur_guess, acc_guess = model(params, 20, vel_data)

    alpha = 0.0
    w = 1.0 + alpha * np.abs(vel_data)

    cur_error = np.square(cur_data - cur_guess)
    acc_error = np.square(acc_data - acc_guess)

    return np.sum(cur_error + acc_error)

# outfit an initial guess 
params_x0 = [0.001, 0.001, 0.001]

# run the minimizer, from scipy.optimize.minimize 
res = minimize(
    fun=loss,
    x0=params_x0, 
    args=(vel_data, cur_data, acc_data), 
)

I had also tried a scheme where we reshape the data into input and output states across time steps, then use the optimizer to tune variables so that an ODE of the system would correctly predict the next step, but I found that pretty ineffective, which is a shame since it’s a cleaner way to articulate dynamics in-general.

A fit! Although some wisdom says that k_e and k_t are equivalent, I let this model fit them independently because I suspect that in this case k_e also captures losses due to phase lag in the commutator - i.e. at high speeds the q-axis is not exactly aligned with the rotor’s real position.

What’s up with the Mismatch ?

There’s obviously a lot missing in the model. What initially confounded me is that the model suggests that when the applied voltage is positive but velocity is negative, (or vise-versa) we should see much more torque, since the effective voltage is greater with EMF working ‘with’ the power supply: $V_{effective} = V_a - k_e (- \dot{\Theta})$

The big mystery to me is how the power supply acts when faced with regenerative loads. For these tests I was using a USB-PD brick with a decoy, which is probably a sin in power electronics. In the data traces where I measured supply voltage during the tests, it doesn’t change meaningfully during these reversals (although it does dip when we use lots of current). In my understanding, even regenerative power supplies will only increase the supply voltage marginally during regeneration, doing so to drive current back into a battery (or dumping it into a resistor), so the back half of this plot should at least be flat.

I suspect a few things are happening here:

(1) The commutation speed is not fast enough to keep up with high angular velocities, so ‘on-axis’ current as measured is not totally on-axis, since the rotor’s electrical angle changes so quickly. This is especially true for a stepper motor, where we have 50 electrical rotations in each mechanical revolution. On the scope, I could see instances where one electrical rotation occurred over the course of only four or five PWM cycles.

(2) There are lots of losses that aren’t modeled here. Motors make a lot of heat in the form of $I^2R$ and we also heat the stator and rotor iron as we spin it through the magnetic hysteresis loop. This is probably the most obvious contributor to the fact that the output torque drops off irregardless of the sign of speed w/r/t the sign of the applied torque… most of the power in these states is consumed by simply swinging the magnetic field around, rather than turning into useful current. I think on this front I need to understand the motor-as-an-inductor’s frequency dependent impedance… that pesky inductance - actually it is staring at me right in the face: $V = IR + L\dot{I}$ - the term I didn’t stick into this model.

(3) The motor may be saturating its magnetic field. Current is equivalent to torque so long as we haven’t reached saturation, but the relationship is nonlinear once we are making lots of field meaning that as we drive higher and higher currents, torque stops increasing. IIRC most magnet steels saturate around 2.0 Tesla of field strength - this is one of the fundamental phyiscal limits to electric motor performance (and transformer performance!). This might contribute to the fact that the torque seems to ‘level off’ around zero velocity (i.e. is bell-shaped).

Via Wikipedia, also see Encylopedia Magnetica. The magnetic hysteresis loop - magnetization (m) is related to field applied by an electromagnet (h), but saturates when h is big. The area inside of the loop represents loss.

So, modelling is hard and the physics become complicated once we really dive into it. I think it is cool at least to be getting data at this point, and fitting some simple models, although I am weary to trust them too much.

To continue testing and learning, I think that I should more carefully monitor the power supply’s behaviour. USB-PD bricks are obviously not meant for this kind of thing, and I was hardly even using that much bypass capacitance in these tests.

I also suspect that losses and winding impedance at higher frequencies are the primary missed-model-components here, so I should endeavor to model them next.

I should also check to see what the effect of changing the drive mode is in the hbridges: at the moment, when the applied voltage is ‘off,’ current is allowed to recirculate in the low-side of the bridge. In prior controllers, this was different - ‘off’ time used a fast-decay mode. It might be the case that fast-decay is advantageous when we are slowing down, and recirculation is better when we’re accelerating.

Finally, I should try to improve the commutation speed. Using the observer, I should be able to update the rotor’s position estimate more often than I am able to sample the encoder - which I am already polling as fast as that hardware allows me too.

Measuring Torque + Trajectory Following

So far, none of this is actually directly useful to me, besides being a really great learning exercise. As soon as I realized how difficult it would be to tune good PID gains for the motor (esp. since changing dynamics…), I was trying to think of a way to measure active torque while running in open loop, since that control mode is so dead-nuts simple, and the PI gains for the current controller are at least easy to tune if we know the electrical values (R, I) which in this case we can get from the datasheet.

In practice, I have machines with ~ 8 motors total, and tuning gains for each of them remains too-difficult to be worth it, especially since open loop steppers already work well enough - instead, I want to be able to use these motors to make models of the system so that I can i.e. intelligently tune my lookahead controllers, i.e. set maximal accelerations and speeds.

Here, the motor is following an open-loop s-curve trajectory generated with my motion controller. It’s using the current controller stage for this, and reading the encoder just to ascertain how much of the current is on the d-axis (in blue) vs. on the q-axis (red) (y-axis is scaled incorrectly, whoops!). Only current on the q-axis is responsible for generation motion, the rest is waste - this is typical of a stepper motor. The oscillation in both signals comes from the motor’s rotor ‘springing’ around to align with the magnetic field.

For a while I thought this would require the position controller, using the output from that controller to measure how much ‘effort’ was required to move the motor under certain tracking conditions - but really it is pretty simple: we can always measure on-axis current to know how much of it is actually responsible for moving the rotor around.

  // in controller.cpp, current control loop 
  case CONTROLLER_MODE_CURRENT_FOLLOWER:
    {
      // we track active currents w/ the rotor-tracking projection, 
      active_dq_currents = lut_trig_xy_2_dq(enc_2_elect_phase_latest, active_ab_currents);
      // but we control current on a target angle that is moving in open-loop, 
      // following our interpolator blindly... 
      follow_dq_currents = lut_trig_xy_2_dq(follower_target_angle, active_ab_currents);
      active_dq_voltages.d = pi_d_axis.run(follower_d_current, follow_dq_currents.d);
      active_dq_voltages.q = pi_q_axis.run(0.0F, follow_dq_currents.q);
      // transforming that back and writing voltage / pwm as usual
      active_ab_voltages = lut_trig_dq_2_ab(follower_target_angle, active_dq_voltages);
      hbridges_write_pwms(active_ab_voltages.a * HBRIDGE_PWM_REQ_MAX, active_ab_voltages.b * HBRIDGE_PWM_REQ_MAX);
      break;
    }

This is fundamentally the same to any other open loop stepper driver / controller, but we are leveraging the FOC hardware to generate a signal that we can use to see how much force is actually being exerted on the system. The modeling above, also, helps us in setting our feed-forward parameters like maximum acceleration and velocity… soon, I’ll also be able to write about how I’ve been using those models more completely with a model predictive controller deployed for lookahead planning on a machine system.

What’s Next

Well, I set out to make a controller that could turn (almost) any stepper into a high performance servo - a goal shared by many - but I spent most of the allotted time in the nuts-and-bolts.

I’m glad to understand the problem a little more in-depth, and to have seen all of the gains we have to tune:

the encoder calibration
- done, reliable, repeatable, and easy to sample
current controller PI gains
- easily done with known R and L, but somewhere beyond my current comprehension
observer alpha, beta, gamma gains
- should be purely functions of the present noise, but hand-tuned for now
position PID gains
- a strong function of the motor’s context
model kt, ke, and kd
- made a reasonable fit, but clearly missing the inductor’s impedance

So I would say that at this point I have learned where all of the problems are, and where the holes in my understanding lie - honestly a great success. I would dive back into the impedance issue right now, I think having figured it out as I wrote all of this down, but I have to get back to the main activities.

I have also been thinking a lot about how to hook this up to the rest of my motion controller. In my machine systems, I stream splines to motors using a fixed point represtentation and a basis spline (aka b-spline) formulation. These have been really productive, and they have the nice property that we can use them to evaluate not just position, but also velocity and acceleration at any time (within a window of available control points).

I commonly use basis splines to transmit trajectories from planners to motors. $P(t)$ is position as a function of time - the first, second and derivatives are also well defined: velocity, acceleration and jerk.

I saw a note recently on feed-forward terms in position controllers like this one, which is that it’s productive to feed forward some gains from a trajectory’s acceleration and speed curves.

Via app note 103 / 102 from Oracle Robotics - shows how acceleration and velocity terms from a known trajectory can be fed into a position control loop - it also notes a gravity compensation term (towards bottom right).

The splines obviously provide a handy set of accel and velocity values to feed-forward, and the gains I would bet we can find using a more complete motor model and some knowledge of the inertia that the motor is working with… it might even be productive to try an LQR controller with multiple inputs (target accel, vel, pos), rather than i.e. the PID with feed-forward gains.

I also have lots of work to do in the firmware to i.e. remotely write encoder tables, and work to do in the calibration scripts to integrate it all into one or two steps: plug in a board, get the encoder table, run some tests and fit the rest of the system. Then there is the question of how to cleanly represent all of that to higher level motion planners.

For a litmus test, I would like to verify the torque measurements with calibrated loadcells - one of the knuckles boards is a CS5530 loadccell amplifier that can run up to 4kHz, it would be awesome to see time-series of the motor in action reporting torques and compared to direct measurements with one of those.

So! I have probably just scraped the surface on this one. I should endeavor to graduate soon and become free to work more intensely on these things; I would really love to provide high-quality, closed loop, comprehendible open source motion control systems that are easy to use… soon!

A Thesis Proposal!

Thu, 12 Dec 2024 00:00:00 -0500

I’ve been at the CBA for seven and one half years and it is time to scoot. I have learned very many things, met many exceptional people, and built many circuits, fimwares, machines, systems, etc.

A ‘thesis proposal’ is what you do when you think you have figured out what you will do to finish a PhD. Academia is weird because you are inventing new things, and so no-one (besides a collection of your peers / advisors) can decide if you have done “it” or not. The proposal is to say: I will do this thing, and that will be enough - if your committee agrees, you do the thing, and then you are done. A bit of a contract.

I am proposing to outline (and develop!) a systems architecture that could replace GCode. The thing is ostentatiously titled:

The End of GCode: Machine Architectures for Feedback Systems Assembly

I made the document into a website using an awesome tool called quarto that I would recommend to anyone who has to write papers and thinks that LaTeX is kind of insane. The website is here. There is also a pdf (auto generated) - but the website is vastly superior, and if you are intrigued by the idea you should read it and tell me what you think. The slides that I used for the presentation are here.

Most of the world’s 3D Printers and CNC Milling machines’ realtime controllers know very little about what they are doing, they simply consume and execute series’ of low-level instructions called GCodes. Those codes are generated in software that knows little about the machine (and has no way to verify what it does know).

GCode is known by many and loved by few (besides cases of trauma bonding). It is an archaic ~~language~~ script / layer / interface / bungled mess. It gets in between us (programmers, users, people) and machines, which are doing very cool physics but we would hardly know how to ask them about it.

A systems overview of the contributions made in this thesis. I refactor a mostly feed-forward workflow and replace parameter selection with online optimization. In this architecture, machines can continuously improve datasets and update models of their own physics, which they use in order to control their outputs.

The core contribution I am making is basically a system architecture that re-casts machines as networks where path planning, hardware control, and sensing are all bits of the same distributed system. Instead of GCode getting in the way anywhere, we can freely connect various software blocks into one another, running as they are on various pieces of hardware: motor controllers, sensors, ‘big-computers’ (aka PCs: things with lots of compute power), etc. This lets us easily move pieces of the controller around, recombine them, take them apart, inspect them, share them, etc.

Machines that can measure what they are doing: here, an FFF printer measuring nozzle pressure and filament flowrates during printing.

This seems kind of obvious (aren’t all machines already made of various blocks of compute?) but state-of-the-art controllers are strictly partitioned; parts of the control (CAM, path planning) happen ‘before’ the GCode, and the rest (motor control, acceleration control, etc) happen ‘after’ the GCode. The partitioning is largely, we think, due to historical mistakes, inter-firm boundaries, and limited embedded compute availability.

The partition has made machine controllers very difficult to modify, measure, and repurpose. It also poses a real problem for high-level planning algorithms (like CAM), whose fundamental constraints (how fast can I push an endmill through this material, how fast can I extrude this filament, and what temperature should I use?) are functions of the machine’s physics - but we can only measure those physics by interacting with the low-level devices!

I develop lots of new bits of tech for this: a very simple, flexible and decently performant networking scheme, some simple link layers and data presentation layers, and of course a new approach to motion control. The controller is composable (easy to edit and configure), and inspectable - it exposes lots of internal states to programmers.

Motion controllers’ internals are normally hidden from high-level programs (like CAM tools). However, low level motion states and optimizer outcomes are important for their performance. In this example, I show the results of rotating the infill in a 3D print so that it aligns with a machine’s anisotropic acceleration.

But why? Don’t machines work well enough? Well - have you ever tried printing a new filament on a printer that didn’t have a preset set of parameters for it, or tuning the motion system for a new machine you have built, or programming a toolpath on a CNC machine? Or re-purposing a 3D printer’s controller to do something silly or novel? These things are not easy, but they should be. They are difficult primarily because machines don’t know what they’re doing - they are just following very simple instructions very faithfully: once we have exported GCode, the smarts have left the system.

So! I then use my fancy systems architecture to develop (1) machines that can tell what they’re doing (sensors!) and then (2) a controller that can use that data to get smart, a model predictive controller for FFF printing.

Distributing controllers lets us more rapidly develop fancy controllers, like this model predictive controller for an FFF machine’s motion and flow systems, which are optimized together, online.

For more, visit the site, and give me about three or four more months to finish the project and get stamped… after which I will take on the project of trying to explain and share all of the goodies into the commons.

Modular Code for Modular Hardware w/ Metaprogramming

Mon, 29 Apr 2024 00:00:00 -0400

Remote Procedure Calls (RPCs) are a great pattern for interfacing with embedded codes - especially when they replace serial string parsers - but they can be tiresome to generate by-hand. In this post, I demo an automatic RPC generator that pulls function signatures directly out of compiled code on embedded devices, and rolls up python interfaces for them.

Automated RPCs and Templated Proxy Classes

A few weeks ago I finally finished the automatic Remote Procedure Call (RPC) class for arduino, which lets us automatically build interfaces for embedded devices using most basic programming concept: the function call. We write a function on the device, we tag it with an RPC macro, and it becomes available to call over the network.

This post is about the other half of that workflow: generating software interfaces - “proxies” - that represent embedded devices in higher level programming languages. In particular, previous work required that we use the auto RPC pattern within a standalone code editor - in this work we can deploy the modules more fluidly.

I’ve been interested in this kind of work because I think that Open Hardware doesn’t have a great set of tooling for systems integration, and I think that good, modular interfaces can lead us into the commons of Open Hardware.

Automated RPCs in the Wild

So… what is this for? Well, for one fun example, we have this little robot xylophone project that I finished with Quentin. It runs hardware / firmware modules of my design, whose APIs are exposed using the system I’ve described here. Quentin, the CBA’s resident CV lover, used those interfaces to stack an interactive controller w/ a camera and MIDI keyboard onto the machine:

Building good interfaces to our hardware helps other developers build applications on top of what we’ve done - i.e. this robo-xylophone serves an API that Quentin used to turn it into a MIDI keyboard output device (including CV to track finger position to preempt the key presses).

I use these also to build bigger systems, like this FDM printer that is collecting data on printing physics while-it-prints. Doing this in embedded alone would be obviously cumbersome, but being able to partition between big-compute (to collect and analyse) and embedded-compute (to listen and actuate) makes it pretty easy.

Good embedded interfaces also helps us de-clutter big systems, like this FFF 3D printer that captures process data while it operates.

The printer’s controller is mostly built in python, using interfaces to hardware devices built using the strategy described in this post. Devices are simple and relatively stateless: motors know how to follow spline trajectories (but don’t manage i.e. acceleration control), sensors can generate readings, but high level program state is managed and modified without editing firmware.

For a clear example of digital twinning, when we virtualize our controllers, it becomes also very easy to build parseable virtual models of them, and pipe data back and forth, i.e. to debug kinematics:

In the world beyond our desks at the CBA, many new off-the-shelf scientific devices already serve python APIs, making the language a productive site for systems integration; it’s the party that everyone is already attending. We want to be able to quickly bring our own ad-hoc systems to the same venue, and share tools with device OEMs, whose costs are increased by developing these interfaces. We discussed all of these bottlenecks at some length in April 2024 at Nadya’s POSE Workshop at the University of Washington.

Replacing Serial String Parsers with RPCs and Proxies

One of the most painful, and most often re-engineered components of embedded projects is the serialport parser: a layer that connects to a device and writes / reads characters to the port. In many instances, these characters are ASCII string encodings, and developers craft interfaces on either end of the port (one in embedded, one in i.e. python, processing, or javascript) to develop whatever interconnect they need: for data collection, for sending commands to an embedded device, etc etc.

This pattern is all over the place, but it’s cumbersome to build and maintain, it sucks space in Flash, and it’s hard to document. I complained about these over here as well.

Replacing this with object oriented hardware is an ongoing effort in my research group at the CBA; see Ilan and Nadya’s theses. What I’m writing about here is basically an extension of modular-things, which I started with Quentin Bolsee and Leo McElroy.

The gist is that we turn devices into software objects that I call proxies; i.e. the stepper motor that controls the x-axis of a machine becomes a proxy object called stepper_x that has multiple rpcs attached. In our program, and then we can write higher level codes with that, like:

# for example, with a nice low-level API to a motor, we can write 
# higher-level routines (like homing) in a language that is easier 
# to inspect and edit (like python)
# meaning less firmware updates, more flexibility, etc:

# our interface (next code snippet) 
import SimpleStepper 

# we instantiate each instance by passing in a device name 
# the dname is a circuit-specific string; this is basically an addressing step 
stepper_x = SimpleStepper(osap, "stepper_x")

def home_x_motor():
  # turn the motor on and head towards the switch 
  await stepper_x.set_current_scale(0.25)
  await stepper_x.set_velocity(-10.0)

  # wait until the limit is hit, 
  while (await stepper_x.get_limit_state()) != True:
    await asyncio.sleep()

  await stepper_x.stop() 

The pattern is useful mainly because embedded development is hard(er) than development in high level languages like python. We still need embedded codes to manage the really high-speed, low-latency actions required for i.e. motor control (for example, a stepper motor needs to ‘tick’ at nearly 100kHz), but it’s a pain to also assemble complex logic in embedded cpp. By bundling the device as a software object, we can interact with it at a high level without sweating the details. If we develop those device APIs well, we can re-use modular hardware across a number of projects.

async codes are a particularely useful tool in this regard: we can issue requests to multiple devices at once, and wait for them all to complete, before moving on to the next action for our system. Doing this in embedded code alone basically requires an RTOS (or some similar homebrew RTOS-like scheduler), but we get it for free in python.

What’s in a Proxy

Writing proxies is straightforward: we build one rpc_caller for each function, which gets network addresses to the rpc_implementation that we wrote previously in firmware. The rpc_caller does all the work of serializing data, sending it to the right network coordinates, and deserializing return values. The proxy simply wraps all of these into a class that represents the device, and presents python-native function names and type hints.

This might look a little clunky, but don’t worry because we don’t need to write it by hand.

from typing import cast 
from osap.osap import OSAP 

class SimpleStepper:
    def __init__(self, osap: OSAP, device_name: str):
        self._set_position_target_rpc = osap.rpc_caller(device_name, "setPositionTarget")
        self._set_velocity_target_rpc = osap.rpc_caller(device_name, "setVelocityTarget")
        self._set_position_rpc = osap.rpc_caller(device_name, "setPosition")
        self._get_error_message_rpc = osap.rpc_caller(device_name, "getErrorMessage")
        self._get_position_rpc = osap.rpc_caller(device_name, "getPosition")
        self._get_velocity_rpc = osap.rpc_caller(device_name, "getVelocity")
        self._set_current_scale_rpc = osap.rpc_caller(device_name, "setCurrentScale")
        self._get_limit_state_rpc = osap.rpc_caller(device_name, "getLimitState")
        self.callers = [
            self._set_position_target_rpc,
            self._set_velocity_target_rpc,
            self._set_position_rpc,
            self._get_error_message_rpc,
            self._get_position_rpc,
            self._get_velocity_rpc,
            self._set_current_scale_rpc,
            self._get_limit_state_rpc
        ]

    async def begin(self):
        for caller in self.callers:
            await caller.begin()

    async def set_position_target(self, position: float, maxVelocity: float, maxAccel: float):
        await self._set_position_target_rpc.call(position, maxVelocity, maxAccel)

    async def set_velocity_target(self, velocity: float, maxAccel: float):
        await self._set_velocity_target_rpc.call(velocity, maxAccel)

    async def set_position(self, position: float):
        await self._set_position_rpc.call(position)

    async def get_error_message(self) -> str:
        result = await self._get_error_message_rpc.call()
        return cast(str, result)

    async def get_position(self) -> float:
        result = await self._get_position_rpc.call()
        return cast(float, result)

    async def get_velocity(self) -> float:
        result = await self._get_velocity_rpc.call()
        return cast(float, result)

    async def set_current_scale(self, duty: float):
        await self._set_current_scale_rpc.call(duty)

    async def get_limit_state(self) -> bool:
        result = await self._get_limit_state_rpc.call()
        return cast(bool, result)

Hardware systems can be complex; when we have to remember what all of our RPC function names are while we’re programming, we can easily make mistakes. The interface class basically gives our IDE the ability to auto complete and auto correct for us, and the type-hints are (IMO) the best form of documentation we can ask for.

Proxies are invaluable tools when programming, for the simple utility of providing auto-complete and type hints in an IDE. We can also name each proxy to make sense of which device we are addressing, making semantic sense in big systems with many devices.

The ‘Metaprogramming’

So, the proxyy is rad, but writing it is a time suck - especially if we’re looking at someone else’s firmware. But, all of the information we need to write the interface is already available via the auto-rpc class - that is, the embedded device knows what its functions are called, what their type signatures are and so on, and we can query whenever they are connected to our network in order to learn these data.

To turn those function signatures into callable proxy classes, I used jinja - a python package for string templating. It’s not dissimilar to liquid, which Jekyll uses (and which this site is built with). String templating is a pattern I became familiar with thanks to Leo, who showed me lit-html a while ago and got me thinking about this notion of programs writing programs.

This is what I mean by ‘metaprogramming’ - although I think this is like soft-core metaprogramming - the really wild stuff in this domain is i.e. programs that optimize themselves at runtime, etc.

Jinja can look kind of gnarly, since we are making statements (in jinja) about making statements (in python), here’s the code that writes modules:

from typing import cast 
from osap.osap import OSAP 

class {{ class_name }}:
    def __init__(self, osap: OSAP, device_name: str):
        {% for func in signatures %}self.{{ func.pythonic_name }}_rpc = osap.rpc_caller(device_name, "{{ func.name }}")
        {% endfor %}self.callers = [
            {% for func in signatures %}self.{{ func.pythonic_name }}_rpc{% if not loop.last %},
            {% endif %}{% endfor %}
        ]

    async def begin(self):
        for caller in self.callers:
            await caller.begin()

    {% for func in signatures %}async def {{ func.pythonic_name }}(self{% for arg in func.args %}, {{ arg.name }}: {{ arg.type_name }}{% endfor %}){% if func.return_type == "void" %}{% else %} -> {{ func.return_type }}{% endif %}:
        {% if func.return_type == "void" %}{% else %}result = {% endif %}await self.{{ func.pythonic_name }}_rpc.call({% for arg in func.args %}{{ arg.name }}{% if not loop.last %}, {% endif %}{% endfor %}){% if func.return_type == "void" %}{% else %}
        return cast({{ func.return_type }}, result){% endif %}

    {% endfor %}

To use these, we connect all of our embedded devices, and then query them for their function signatures, device names, etc. We then feed that data into this structure, and write the output as a .py file.

Naming and Case Conventions

At this point our system is authored in two languages - probably cpp (in the embedded device, or rust if we get our act together and learn that new language that everyone is on about…), and in python, where our interface and the rest of our high-level system lives.

As a minor annoyance, style is language dependent: javaScript folks tend to prefer camelCase, python’s pep8 advocates for ClassNames and function_names, and cpp is something of a free-for-all.

Mostly for Quentin’s sake, I added some utilities to translate betwixt them all - so if we use camelCase functions in the embedded device, they are served to us as snake_case functions in python.

Actually, ChatGPT wrote these utilities - regex was always for robots anyways.

def to_camel_case(name):
    # Split the string by non-alphanumeric characters
    words = re.split(r'[^a-zA-Z0-9]', name)
    # Capitalize the first letter of each word and join them together
    camel_case_name = ''.join(word.capitalize() for word in words if word)
    return camel_case_name

def to_snake_case(name):
    # Replace dashes with underscores
    name = name.replace('-', '_')
    # Insert underscores before capital letters (considering initial caps and acronyms)
    name = re.sub(r'(?<!^)(?=[A-Z])', '_', name)
    # Convert the whole string to lowercase
    return name.lower()

Writing a Boilerplate `main.py`

Finally, the code generates an example of how one would deploy these modules - the main.py - this is of course optional, but I thought it would be nice to have as a quick-start tool. It initializes the proxy objects, connects the networking drivers, and wraps it all in async nuts and bolts.

import asyncio 
import traceback 

from osap.bootstrap.auto_usb_serial.auto_usb_serial import AutoUSBPorts 
from osap.osap import OSAP 

# import each interface / device proxy 
from proxies.stepper_cl_d51_proxy import StepperClD51Proxy

async def main():
    try:
        # init networking service 
        osap = OSAP("auto_modules_main") 
        loop = asyncio.get_event_loop() 
        loop.create_task(osap.runtime.run()) 

        # connect via usb 
        ports = AutoUSBPorts().ports
        for port in ports:
            osap.link(port)

        # collect an image of the system 
        system_map = await osap.netrunner.update_map() 
        system_map.print() 

        # instantiate each of our modules, 
        stepper = StepperClD51Proxy(osap, "stepper_cl")
        
        # and set each up: this block additionally verifies that proxies 
        # are matched to implementation signatures discovered at runtime 
        await stepper.begin()

        # insert test code below... 
        #
        #
        #

    except asyncio.CancelledError:
        print("bailing...")

    except Exception as err:
        print(traceback.format_exc())

    finally:
        # put shutdown codes here, 
        # i.e. powering down current sources, etc... 

if __name__ == "__main__":
    asyncio.run(main())

The resulting workflow is one of the most productive tools I have written for myself during my PhD - we do:

(1) tag some functions in your embedded device using a one-line macro BUILD_RPC(func, arg_names)
(2) connect the device to a PC and run a small script, that:
- (2.1) reads function signatures from the device
- (2.2) writes a device_proxy.py for each embedded device, and a main.py file that initializes and connects to them all
(3) add whatever test codes to the main.py file,
(4) run the main.py … modify it, move the interface to some other project, etc etc

The bottom line for me is that I don’t have to hand-craft interfaces to embedded devices, even the ones that I am actively developing: when I add a new function to a device, or a new device to a system, I simply re-run these scripts and get back to development. Device names, function names and types, all stay consistent, and matched to hardware implementation. For a mild dyslexia sufferer / maker of small mistakes, this has vastly improved my development cycles.

Future Efforts: Protocols, Standards

Like I mentioned, the motivation for these types of systems is to enable a commons of open hardware. One of the current impediments to open hardware’s success is that, while many contributions exist (devices), we still struggle to make systems out of those building blocks.

Probably the best integration system for open hardware is still Arduino’s ecosystem, but it is fraught with strange gotchas - i.e. while combining many sensors into one project, we might run out of I2C resources (address space, or time)… and as we add parts and libraries, we eventually eat up all of the RAM and CPU cycles available on our project’s main microcontroller.

With strategies like this one - where we combine many devices by networking them together - we can scale systems more rapidly. However, we want to minimize the burden on device developers and on systems integrators. If I am going to use a circuit module that someone else designed, I should not have to understand how their firmware or their build system works in order to deploy it in my own application. I want to plug and play.

The proposal here is that device firmwares themselves can contain all of the information required to interface to the device, in a semantically meaningful way, using off-the-shelf programming paradigms. We want to be able to extract that information from a single source of truth and use it natively.

To improve systems integration in open hardware (and beyond), we should make more explicit use of an OSI model for mechatronic devices. The strategies discussed in this post take place in the presentation layer (near the top); they concern how we represent data and devices to application developers. We also need to work on the rest of the stack.

Besides the networking step (where many people are already sticking USB, BLE or i.e. CAN interfaces on their devices), we still need to make sense of the packets sent between our applications and our devices - and from device to device. Doing so across open hardware projects would be an effort in standards development: how to encode function signatures, type signatures, and serialize / deserialize data transfers. While I am independently working on each of these things, doing it on my own is not really a viable path towards such a standard - in the future we will all need to agree on their shapes, forms and styles.

Some Discussion

I am in no way proposing this as a be-all end-all solution, but I think it’s an interesting step, and has been hugely productive for me personally - so I wanted to discuss it.

I think that interesting future work would address the issue of versioning and duplication: i.e. a system containing two of the same devices, but with firmwares of variable lineage. There is also still a burden of documentation: functions should probably contain docstrings as well, but storing those all in a microcontroller seems cumbersome - instead, they could perhaps return URIs to expanded docs served online, or some mixture of both.

The approach also requires really good serialization / deserialization routines, and can lead to lots of manual type-checking, since embedded devices tend to have less flexible native types than i.e. python, so strange things might happen when we send i.e. python’s native int64 into a function that declared a uint8_t argument.

This approach is also of the flavour where we basically “throw it all into python” and then hope that there’s enough performance there, but .py is not a terribly performant language, so for the integration of high-speed systems, we probably need a better low-level tool that lets us avoid operating systems entirely. To that end, I have been developing graph based tools that let us pipe data directly between embedded devices (but visualize and edit configs at a high level). If I survive this PhD there will be more on that later.

In the end, what we want is for things to ‘just work’ - we want to build the complex bits, the high level controllers. No-one should be out here re-engineering motor controllers and motion systems just so that they can do science with a robot, etc etc. Systems integration is the stuff that plugs individual engineers into the world’s economies and hooks their contributions up to teams, firms, and society, man … we should spend more time making it beautiful, functional and fun.

MUDLink: a Modular UART Duplex Link

Sat, 09 Mar 2024 00:00:00 -0500

MUDLink on GitHub / Arduino

I built a new library for Arduino that lets us promote any Serial port (UART) into a competent link / transport layer with guaranteed delivery of framed packets.

MUDLink adds packet framing, delivery guarantees, and flow control to just about any UART Port. It also records stats on the link’s performance, to enable a litle bit of feedback systems assembly.

I have wanted to release something like this for a while since just about every microcontroller has at least one UART port, and they are well understood in the OSHW community - making them a good candidate for link layers that could help to enable a commons of interoperable hardware. In order to get from UART to Reliable Link Layer, though, we need to add a few things:

Packet Framing

UARTs send bytes only, and have no native packet delineation as is the case in i.e. I2C, CANBus etc: this makes it difficult to frame and decode efficient bytecodes of any real complexity.

Bit and Byte Errors

UARTs are not fault tolerant: each microcontroller simply sends bytes at whatever configured baudrate, and recieves at the same. If bytes are lost or bits are flipped, but programmers don’t anticipate or catch these errors, things can go wrong. This means that folks are left either with unreliable systems (throwing intermittent errors, the worst kind!), or are left re-building fault tolerance when they might rather be finishing the interesting parts of their project.

Flow Control and Feedback

Since UARTs are mostly used with only TX and RX lines (no CTS/RTS etc), they are not normally flow controlled. This can be troublesome in applications where i.e. one participant can write significantly faster than the other can read (leading to lost bytes).

It can also be difficult to tell what an appropriate baudrate is: lower rates are typically less prone to errors, but that stability comes at an obvious performance loss. In some cases, we pick what seems like a stable baudrate only to change our environment slightly, and find that it is actually quite error prone… for example we might turn on a high-power motor (injecting a lot of noise) and have our comms break down.

These kinds of phenomena typically means that (as is the case in much systems assembly), we end up using lower performance parameters than we might be able to, i.e. opting for baudrates that never fail, even though they may be up to one quarter the speed fo baudrates that fail only very occasionally.

Enter MUDLink, a Useful UART Link

So, the MUDLink is a software bolt-on that effectively promotes any Arduino-exposed Serial object from a PHY (layer one of the OSI model) into a simple Transport layer (although it roughly eschews Layer 3: since it only sends datagrams between two devices, there is no addressing or routing… it also misses segmentation which is meant to live in Layer 4).

Longer discussion on any resemblance to the OSI model is mostly useless anyways, as many have stated (most recently, Nathan made this point in his Machine Class lecture) it is a good common understanding or theory of how we might delayer networks, but is rarely followed carefully in practice.

In simple terms, MUDLink lets you quickly establish a packetized, reliable and inspectable connection between two microcontrollers.

Packet Framing with COBS

Firstly, we need to make streams of bytes into chunked packets. This is like networking 101: if we don’t know where any given byte goes w/r/t any other byte, we can’t reliably do much (unless each of our messages is just one byte long).

From the Wiki Page on COBS: COBS transforms an arbitrary string of bytes in the range [0,255] into bytes in the range [1,255]. Having eliminated all zero bytes from the data, a zero byte can now be used to unambiguously mark the end of the transformed data.

COBS is Consistent Overhead Byte Stuffing and is already in most embedded developers’ toolbox. It has the excellent property that it requires only two extra bytes of transmitted data for any packet smaller than 254 bytes (and only one more byte per chunk of 255 thereafter) - and requires those bytes consistently, meaning that we can have more deterministic systems.

COBS is also simple and fast: about 64 lines of c code (with comments). MUDLink deploys the wikipedia page’s implementation directly (unfortunately I wouldn’t be able to tell you who wrote that, though - thanks to them!).

Error Catching with CRC16-CCITT

Once we have rx’d a sequence of bytes, we need to check that they are valid, since things may have gone awry while they were en route (flipped bits due to noise or missed bytes due to slow interrupts being the most common culprits).

For this, MUDLink uses a Cyclic Redundancy Check, particularely the CRC16-CCITT polynomial. For a really well done explainer on CRCs, check out Ben Eater’s video on the topic.

A CRC is like a really fancy checksum: we use a carefully selected function that takes our packet (as a big number) and generates a smaller number that is guaranteed to change if some changes to the packet occur - for example the CRC16-CCITT sis guaranteed to change if there is a change to a single bit within a packet of 8000 bytes - so, pretty good. It’s kindof like a hashing function but different in important ways. Real big brain math stuff going on in there. I would like to point out that I understand the topic only glancingly, and you should refer to others for more detailed and correct explainers (i.e. Ben Eater above).

CRCs are additionally designed to be quickly implemented: in theory they are basically big long divisions but in practice they are calculated using bitwise XOR operations and bit-shifts.

Flow Control and Delivery Guarantees

So! We can catch packets and make sure they are legit. Now we want the real magic: we want to know from our end (1) that our packet was successfully received and (2) that it has been consumed by our partner, so that they are clear and free to recieve the next message. We accomplish these goals together with an ack-and-retransmit scheme not dissimilar from TCP itself, which is again explained really well by Ben Eater.

The MUDLink code is essentially four tiny state machines that signal one another, which I’ve tried to represent here. The left side is application facing states and actions: the programmer can tell if the port is open (not in a failed state), and if it is clear to send or clear to read a packet. On the right side we have, basically, hardware buffers. These send data out into the world and are responsible for catching it. Either side can generate and consume events from the others: for example when we finish RX’ing a packet that passes the CRC, it sets the user-facing CTR state. When they finally read the packet, the CTR state transitions to EMPTY, and an ACK is generated and written into the TX buffer. Keeping a clear separation of these layers normally helps us separate ISR’s from user code, but in this case it just helps clarify the system operation.

TCP is a lot more complex than this (MUDLink is only 350 lines of code as of this post), but they use similar principles:

We keep a rolling sequence number that increments when we generate a new message.
- We always transmit our most recent segment number, as well as the sequence number of the packet we have most recently read from our partner (as an ack).
When we transmit a message, we wait until we have recevied an ack from our partner before we transmit any new data.
- We may, however, retransmit the same message if our timeout for that runs out.
- After each timeout, we increase the timeout interval exponentially.
- When we have retried the packet a number of times, we enter a failure state.
When we recieve a new packet, we present the data to the application by setting Clear To Read (CTR) high. We generate an ack for our partner only when the user has consumed that data.
When we recieve a re-transmitted packet (that we have already successfully caught and read) we generate a new ACK, because the sender likely missed the previous one (hence their attempt to retransmit).
Finally, we run a keepalive timer: if we haven’t transmitted anything in some interval, we ping our partner to make sure they know we’re still open. Similarely, if we don’t hear anything for a while, we consider the port to be closed.

Statistics Tracking

So, I had mentioned that it’s often difficult to measure what’s going on in our networks, which makes it difficult to design systems. It turns out this is a bona-fide problem in industry, and especially in safety critical systems: great engineering bookeeping efforts are expended making sure that there will always be enough network bandwidth available in, for example, a 747’s avionics system.

In OSHW, which currently builds more ad-hoc projects, we rarely take on that level of bookkeeping, so we are left in the relative dark about how much bandwidth we are using in our networks, how fast they are really running, etc.

MUDLink adds a little to ameliorate this in that it tracks some stats, we have these (self describing) available:

typedef struct MUDLStats {
  uint32_t rxSuccessCount = 0;
  uint32_t rxFailureCount = 0;
  uint32_t txSuccessCount = 0; 
  uint32_t txFailureCount = 0; 
  uint32_t txTotalRetries = 0;
  // the longest timeout issued since startup 
  uint32_t outgoingTimeoutLengthHighWaterMark = 0; 
  // maybe the most important, average message trip time, 
  float averageTotalTransmitTime = 0.0F;
  // comparable to the wire time (bits / baudrate) for most msgs 
  float averageWireTime = 0.0F;
  // and the num of tx retries normally required 
  float averageRetryCount = 0.0F; 
} MUDLStats;

These are available to the programmer, to assess their choice of baudrate, quality of underlying link, etc.

API and Protocol

MUDLink ingests any Serial instance within the Arduino ecosystem, meaning it can be deployed in just about any Arduino project. We instantiate it using a a template like this:

// 2nd argument is the baudrate 
MUDL_Link<decltype(Serial1)> mudl(&Serial1, 2000000);

It has a pretty simple API, we can check if the link is clear to send or clear to read, and we can send and read packets.

uint8_t appRx[255];
uint8_t appTx[255]; 

void loop(void){
  // MUDLink runs on an event loop, 
  mudl.loop();

  // check-then-read,
  if(mudl.clearToRead()){
    mudl.read(appRx, 255);
  }
  
  // presuming we have stuffd appTx with a message: 
  if(mudl.clearToSend()){
    mudl.send(appTx, msgLen);
  }

  // and we can get those stats, 
  MUDLStats stats = mudl.getStats(); 
}

The message spec is pretty simple as well, beneath our COBS encoded packets we have this block:

Bytes 0 … N-5	Byte Len-4	Byte Len-3	Byte Len-2	Byte Len-1
… Payload	ACKD_SEQ	TXD_SEQ	CRC:1	CRC:2

So, pretty small overhead, and the payload is simply missing when we send ack-only or keepalive messages.

System Limits, Future Work, and What I Learned While I Did It

MUDL has a pretty clear limit to performance, which is that it has an overhead floor of 2x: we will always use roughly half (or less) of the available bandwidth in a duplex link, because we are always waiting for our partner’s ack before we can send another. So, on a scope (with TX and RX lines tapped) we see this kind of pattern:

This is ameliorated in grown-up protocols with transmission windows - each announces how long their receive buffer is, and then we can transmit as many messages as are in the buffer before receiving an ack. TCP does this, and hopefully MUDL will in the future… I have written that protocol out before, but it was a little shaky. Then again, it didn’t have CRC at that time, so I should bring my new learning back to that project.

MUDL is also limited by the underlying performance of whichever block of Arduino Core code is running the Serial implementation. In this case I was using Earle the III’d RP2040 Core, which is wonderful (thanks Earle!), but which doesn’t seem to service the actual transmit interrupt fast enough. So, if we send 1MBaud of 0b01010101 we see this:

But if we config at 3MBaud:

So - what’s up? Well, we are transmitting at a speedy baudrate, but every time the RP2040’s TX FIFO dries up, it takes too long to refill (hence the inter-byte gap). Since a 3MBaud Byte (10 bits for UART framing) is only 3.3 microseconds long (at 200MHz, 660 clock cycles), we can run out of ISR time if we don’t write tight enough code. In practice, this means that while the RP2040 can run a UART up to 3MBaud, we only ever effectively see about 1MBaud of real performance.

This kind of thing is why the pros use DMA, etc, and why I am thinking I aught to return to that practice. With DMA, our app writes a big TX buffer, and then each byte is ferried into the UART without the processor’s intervention at all - normally triggering an interrupt when the transmit is totally complete.

High-speed hardo embedded code trades off with portability though, of course. For now, I like the flexible, code-only transport / link layer as it is, but I can look forward to writing custom implementations for some of my favourite microcontrollers. One of the nice things of this scheme is that we can have a sloppy implementation still pair up with a high-speed implementation, and everything should still work.

Odds and Ends

One other thing I learned during this cycle is that the Arduino I2C library is blocking - so when we use these OLED displays (see the first picture here), which require a big chunk of data in the screenbuffer, we can get big blocking calls in our Arduino code. In the pictured demo, the display write is over 20ms.

This could also be ameliorated by some hardo embedded code, but it’s actually been a good foil: MUDLink works despite the blocking call that hampers performance servicing the transmit routine and also causes packets to be entirely missed (when the blocking call happens during a packet receive and we can’t run our loop, we miss bytes!).

What about Busses !

One last note is that I’ve come up with a pinout that suits this protocol (over RS485) as well as a bus topology. I would love in future times to build the protocol to service point-to-point as well as bus topologies, perhaps dynamically switching between the two.

Last trick to add would be dynamic baudrate scaling, to make these ports truly plug and play across devices and without any config.

Delete Your '\n' Delimited Firmware Interface!

Fri, 09 Feb 2024 00:00:00 -0500

TL:DR, I wrote an RPC protocol and compiler tool for Arduino that lets us quickly turn hardware modules into software modules. I think it’s a good candidate design pattern for replacing newline-delimited string-parsing protocols.

Remote Function Calls

aka RPCs - are a pretty common design pattern in distributed systems. The basic idea is simple: we use a network connection to ask another computer to run a function for us. In the case of modular, embedded devices (like those in Modular Things), it’s a great way to centralize systems assembly: each of our modules exposes an API of RPCs and we string those function calls together into new programs - building systems that are collections of modules becomes as simple as writing programs that coordinate multiple software objects¹.

Discovery

One of the main troubles with RPCs is that we need to know what functions we have available on our network. In many systems, we simply have to know this information before we start programming: what devices are attached, where are they located, and what functions do they contain … and then, what are the names, types, and argument lists for those functions. This kind of “feedforward systems assembly” can cause all kinds of headaches when we go to run our program, because if we are just a little bit misaligned between what our code expects and what is configured in reality, we run into crashes and failures.

So, the RPC code I’m presenting here additionally exposes a discovery layer - just a small addition to the protocol that lets other network participants ask for function signatures (names, types…). This means that we can do some measure of “feedback systems assembly” - we can inspect our remotes and check our local codes against their reported functions at runtime.

Discovery == Lower Documentation Requirements

Discovery is also a great way to handle documentation: rather than write datasheets about what functions our devices expose, we can rely on API discovery to handle that… basically, I am proposing to put the documentation in the device itself. I suspect that this could help a great deal with version control as well, since each firmware build can report exactly what it contains (whereas it is common for docs and codes to be out of sync with one another).

Auto Rollup

Older versions of Modular Things also basically deployed RPCs, but the serialization was left up to the module author. As I have discussed, making it easy for authors to include new modules into a commons is important, so the major effort I’ve made with this code is to make the function-to-rpc rollup really straightforward (for module developers).

Previously, authors of new modules needed to write matching .js codes for their firmwares. While that pattern is still present in Modular Things (and is useful for more involved device design), it is liable to cause errors when firmwares and higher-level descriptions don’t match up. The new pattern automates this process, and means that we build systems with less sources of truth.

Prototype Protocols, Serialization and Deserialization

Any RPC code relies on some protocol - for this project I threw together a semi adhoc protocol since I think a protocol is better designed once we have some experience with a system… and I know that my buddy Quentin is working on a more robust protocol design for a similar system.

At the core of this protocol we have a small set of type-keys for float, int, boolean and string - then we use those to furnish calls to get function signatures and names, and to serialize arguments (when calling functions) and reply types (on the way back up).

At the moment, this all works within OSAP, which has transport protocol built in; I simply instantiate a new OSAP port for each RPC class, and they are each individually addressable in the network… I’ll write more about OSAP hopefully in the next few weeks.

Type Keys and Data Serialization

Type Keys
void	int	bool	float	string
0	1	2	3	4

Serializations of these are straightforwards, i.e.:

template<typename T>
void serialize(T var, uint8_t* buffer, size_t* wptr){}

template<>inline
void serialize<int>(int var, uint8_t* buffer, size_t* wptr){
  buffer[(*wptr) ++] = var & 255;
  buffer[(*wptr) ++] = (var >> 8) & 255;
  buffer[(*wptr) ++] = (var >> 16) & 255;
  buffer[(*wptr) ++] = (var >> 24) & 255;
}

template<>inline 
void serialize<bool>(bool var, uint8_t* buffer, size_t* wptr){
  buffer[(*wptr) ++] = var ? 1 : 0;
}

template<typename T>
T deserialize(uint8_t* buffer, size_t* rptr){}

template<>inline 
int deserialize<int>(uint8_t* buffer, size_t* rptr){
  int val = 0;
  val |= buffer[(*rptr ++)];
  val |= buffer[(*rptr ++)] << 8;
  val |= buffer[(*rptr ++)] << 16;
  val |= buffer[(*rptr ++)] << 24;
  return val; 
}

template<>inline 
bool deserialize<bool>(uint8_t* buffer, size_t* rptr){
  return buffer[(*rptr ++)];
}

Protocol Message Types

Function Signature Request
B0	B1
PRPC_KEY_SIGREQ	MSG ID

Function Signature Response
B0	B1	B2	B3	...	...	...
PRPC_KEY_SIGRES	MSG ID	Return Type Key	Number of Args	... Arg Type Keys	Function Name	... Arg Names

Function Call
B0	B1	B2 ...
PRPC_KEY_FUNCCALL	MSG ID	... Arg Data

Function Return
B0	B1	B2 ...
PRPC_KEY_FUNCRETURN	MSG ID	Return Data

Spec / Protocol Based Systems allow Interoperability without Open Source

This is probably worth a lengthier discussion, but I think that we often reach for Open Source when all we want is Open Interoperability. This may be especially true in industrial settings, where closed protocols prevent interoperation in order to lock customers into particular ecosystems.

This is also a current site of tension in Open Hardware: folks are struggling to understand how to make money while exercising their craft as hardware engineers. Personally, I think the overarching goal is maybe more aligned with a theme of “Collective Intelligence,” of which Open Source is just one flavour… What we want is to be able to put our heads together, and build big complex systems using contributions made by many individuals. Interoperation achieves that, and allows individual contributors a clearer path to profitability - and we shouldn’t be afraid of that.

Towards Defeating the Serial String Parser

All highfalutin discussions aside, there is one nasty design pattern that I think this design pattern can help to vanquish, which is the \n newline-delimited string parser. By which I mean any (mostly Arduino-based) firmware that contains some snippet like this:

void loop(void){
  int received_bytes = Serial.readBytesUntil('\n',input_buffer,INPUT_BUFFER_LENGTH-1);
  if(received_bytes > 0){
    input_buffer[received_bytes] = '\0';
    String command = String(input_buffer);

    /* .... */

    // basically just a function call to "move_axes:" 
    if(command.startsWith("move_rel ") or command.startsWith("mr ")){ //relative move
      int preceding_space = -1;
      long displacement[n_motors];
      EACH_MOTOR{ //read three integers and store in steps_remaining
        preceding_space = command.indexOf(' ',preceding_space+1);
        if(preceding_space<0){
          Serial.println(F("Error: command is mr <int> <int> <int>"));
          break;
        }
        displacement[i] = command.substring(preceding_space+1).toInt();
      }
      move_axes(displacement);
      Serial.println("done.");
      return;
    }

    /* .... */
    // basic getters 
    if(command.startsWith("ramp_time?")){
      Serial.print("ramp time ");
      Serial.println(ramp_time);
      return;
    }

    /* .... */

  }
}

This is a snippet from the sangaboard; the firwmare that runs the open flexure, an exemplary piece of Open Hardware. I don’t include it here to pick on them (this is a really well-written parser), but just to point out that these string-based protocols (that are prevalent in OH), are often re-written project-to-project.

String Parsers are often shaky, ad-hoc protocols, that suck time and FLASH from our codes, and make difficult to maintain, read, and extend embedded systems.So, if you’ve a hardware project in need of a comms layer, perhaps give modular-things a shot. At the moment, this works just in the browser in Modular Things, but next weeks’ update will deploy a design pattern that makes it easy to drop these into native JS, TS and Python projects.

Limits and Future Work

Compiler Caveats

This works right now where c++17 or newer is available; of my favourite microcontrollers (RP2040, SAMD51, the Teensy, and the SAMD21) (all deployed on various arduino cores), that leaves just the RP2040. So, that’s a problem.

AFAIK, this is mostly due to my use of std::apply and also some fold expressions, both of which are c++17 onwards. However, the core utilities that I use (std::tuple, decltype and parameter packs) are all available from c++11 - so I should be able to hack this back to thence, using recursive techniques.

There is also the trouble of templating as-well against the full mixture of void-void, void-args, ret-void, ret-args - for those I currently use if constexpr (c++17).

Anyways, that’s all a TODO. AFAIK, c++14 is available across the SAMD core (which makes up very-many arduinos), and the Teensy core, so bumping down to that level would at least satisfy most of my constraints. IDK what other popular MCUs use, i.e. the ESP32 series would be awesome to include. AVR is probably another question.

Stronger Type Serialization / Deserialization

Alongside the templating tricks, the core of this protocol is a prototype of a set of serialization (and deserialization!) functions to turn all of our typed values into bytes-on-the-wire and back. These are valuable across protocols, so I would like to have a core set that work well across platforms. Requirements would be like:

hot
fast
flexible
self reporting
small code size
highly portable

There are many systems out there that do this type of work, including:

cbor: Concise Binary Object Representation
msgpack: “It’s like JSON but Fast and Small”
capn’proto: ex ProtoBuf engineer
protobuf: the serdes tool from Google

AFAIK, capn’proto and protobuf don’t support discoverability, and the other two have been on my homework list for a while. Anyways, we need good serdes for this kind of stuff.

Embedded Device to Embedded Device

The current pattern works well to centralize embedded modules in one higher level code, but we can’t chat between devices. It would be awesome to be able to configure an “RPC Caller” as well as the implementer, in embedded codes. This would be an odd way to configure systems though, because the whole “view” of the system would be distributed across multiple firmware source codes…

In my Masters degree I made a totally bonkers system called squidworks where embedded devices talked directly to one another and systems were represented in a graph view.

Co-ordinating and designing large distributed systems can be a major PITA, especially when it comes to orchestrating direct communication between multiple embedded devices. In my own prior work (above), squidworks used a graph representation to do so. RPCs on the other hand are typically used distributed systems where most control is centralized within one computer and ‘other’ devices are simple service workers, as is the case with Modular Things. However, it may still be useful to be able to implement RPC Callers in firmwares (the current build only builds RPC Implementers).

Graphs make for an excellent overview tool (to grep the whole state of some distributed computer program), but are sometimes clunky programming tools (or perhaps they just expose the spaghetti that is fundamentally inevitable). In any case, some “heavy mixture” of code- and graph-based programming seems like a likely candidate for sense-making of modular embedded worlds, and I would like to return to that question.

.py to .js, and vise-versa.

Along the same lines, I simply haven’t written all of the code with-which a .py code would announce itself to a .js code, etc etc etc.

Everything in time, perhaps.

This is an idea with a long legacy at the CBA, notably in Ilan Moyer and Nadya Peek’s theses. ↩

Some Link Layer Speed Testing

Thu, 04 Jan 2024 00:00:00 -0500

A lotta charts and graphs in this one, friends!

Low Level Link Speed Tests on the CBA GitLab

I’ve been developing an FDM 3D printer that can produce oodles of data. Our collaborators at NIST are interested in this because it will allow them to build better models of the FDM process, and we are interested in that because it will allow us to better optimize FDM slicing, machine control, etc - and I probably don’t have to tell you that consuming realtime data is the first step towards doing realtime closed-loop control.

For a systems redesign, I laid out a table of data sources and rates, and found that (with an amount of packet framing and overhead), I’ll be producing around 5Mbit/s total from the machine. Perhaps more on this in a later post.

That’s a 5MHz clock, each cycle transmitting one bit - it seems a measly number of bits to fit inside of each second (given our acclimatization with “Gigabit” ethernet and the like), but (as we will see) getting this amount of data off-chip and into a useful high-level context is not something that I found easy.

USB Serial: RP2040

So, this post is also something of a challenge: many may cast their gaze here and think “this lad is an absolute Hack, I just plugged my 12Mbit/s (specified) embedded USB device into my PC and wham, it worked!” - well, of course I tried that as well and here’s the results: histograms record inter-packet spacing (in microseconds) for some packet length, and overlay that with equivalent realized bandwidth in Mbit/sec (bits per second, not bytes).

pyserial -> USB -> RP2040 Earle Core SDK USB

Overclocking the RP2040 to 240MHz does help, organizing the distribution into a tighter spike at the fast end:

pyserial -> USB -> RP2040 @240MHz Earle Core SDK USB

USB Serial: SAMD21

And the SAMD21, at 48MHz clock, does much worse (notice the stretched axis):

pyserial -> USB -> SAMD21 XIAO

All in, these results were dissapointing, so I elected to circle back with the Teensy 4.0, a 600MHz bad-boy that advertises a 480Mbit/sec USB implementation.

USB Serial: Teensy 4.0

OK, these were all upsetting and I tested Ethernet and the PI next, but then I circled back to try the Teensy, since I think that Paul is an absolute wizard and I know that his Teensy core is spectacularly well engineered - I was actually not dissapointed:

Pyserial -> USB -> Teensy 4.0, with 32 Byte Packets (note the x-axis change)

Pyserial -> USB -> Teensy 4.0, with 128 Byte Packets

This is wonderful - and a relief honestly, I wish I had tried it earlier. We are up to, easily, our spec of 5 Mbit/sec, and then well past it. Those deltas are wild: tens of microseconds.

Ethernet

python -> Ethernet -> Switch -> Ethernet -> WizNet -> SPI -> RP2040

Eyeing that sweet sweet 100/10 Ethernet link, I collected the WizNet W5500-EVB-PICO, (an RP2040 Pico with the Wiz strapped on) and connected it to my PC.

A few notes on this setup; (1) the Wiz connects to the RP2040 via SPI, which (as we will see later) means we are basically SPI bottlenecked, in addition to the Ethernet’s own packet time, etc. (2) we connect via a switch to our local network, i.e. our PC. So, the setup goes from the RP2040, over SPI to the Wiz, over Ethernet to a switch, back over Ethernet to our PC.

Setting this up also required some static IP configuration: in Windows, I configured the Ethernet interface to have a static IP on the local network, and configured the RP2040 to aim packets at the same (it also got a static IP).

So, the results:

RP2040 -> WizNet -> Ethernet -> Switch -> Ethernet -> PC; one-way, Python ingesting.

RP2040 -> WizNet -> Ethernet -> Switch -> Ethernet -> PC; round-trip (packets down and back up), Python, 64 Bytes / Packet

Ethernet does better than most USB here, and we can strap mega-sized packets through it. While packet size increases delay, total bandwidth goes up:

RP2040 -> WizNet -> Ethernet -> Switch -> Ethernet -> PC; round-trip (packets down and back up), Python, 256 Bytes / Packet

RP2040 -> WizNet -> Ethernet -> Switch -> Ethernet -> PC; round-trip (packets down and back up), Python, 768 Bytes / Packet

We still have a real bottleneck around 2Mbit/sec, much less than we would surmize on a “100Mbit” link, so what’s up? Well, if we look at the RP2040-to-WizNet SPI link on the scope, we get a sense:

CH1: Transaction Flag, CH2: Chip Select and CH3: CLK - we can see that the single Ethernet transaction is made of multiple SPI transactions between the RP2040 and the Wiz…

CH1: Transaction Flag, CH2: Chip Select and CH3: CLK - … and that within each transaction, the actual data transfer (on CH3, where the CLK is active) are pretty well spaced out!

I.E. even though we are running 12.5MHz SPI to our 100Mbit Ethernet device, whatever in-between is happening (the Arduino Ethernet library, on top of abstracted SPI interfaces, etc etc) is introducing up to 5x overhead, so we are left with tops 2Mbit/sec of real data rate.

Raspberry Pi, and SPI

RPi 4 -> ? / SPI -> RP2040 Hardware SPI SDK

Given that the WIZ communicates to the uc via SPI anyways, I thought I would hook it up directly. Being able to do this is basically what SBC’s are for anyways (their CPU cores having direct access to some GPIO, just like an MCU).

This has the caveat that we are now in-context on this SBC (rather than, i.e, our beefy-ass laptop with a hella GPU), but maybe we need this middle ground. So, the setup is: RP2040 via SPI directly to the RPi 4.

The RPI/SPI Setup…

About the TTL levels here, traces look solid up to around 20Mbit, where I suspect we would genuinely loose some bits in the PHY:

At 20MHz, we start to see some shaky TTL…

In any case, this tests pretty well. In these cases, we are “transacting” - SPI out begets SPI in - and counting bytes that are out-of-sequence (missed). IIRC, I did all of these tests at 240MHz on the RP2040, to help with ISR time.

RP2040 @ 240MHz -> SPI -> RPi 4 Python w/ spidev: 10Mbit/sec

10Mbit underlying rate gets us into 5Mbit of realized bandwidth, which at this point was some great news for me. It falls off of a cliff after this though (check the missed-bytes ratio in the plot titles), probably the MCU running out of time to service ISRs:

RP2040 @ 240MHz -> SPI -> RPi 4 Python w/ spidev: 11Mbit/sec

RP2040 @ 240MHz -> SPI -> RPi 4 Python w/ spidev: 15Mbit/sec

We also come up against a bandwidth limit around 5Mbit. There’s an observable delay between transmissions on the line, around 10us, which is the RPi chunking between python loops. It would sometimes stretch to 50us or so…

10Mbit/sec, CH: Chip Select, CH2: DO

But, all-in, SPI-direct is a promising approach, or the best performing so far: it shows big bandwidth and small loop times, I mean, that’s 100us period for cyclic transactions on these plots, which is killer, that’s potential for 10KHz of action (if we can hook everything else up…) between a Python process on a decent-sized CPU and some embedded worlds.

What’s the Point ?

Latency and Throughput Both Matter

When we’re building machines, robotics, or whatever mechatronic devices, we are always interested in how fast we can get data into- and out-of the world; see also our Ring Test Page where we track how fast we can get between an MCU’s computer-part and its physical-part (the pins). In most systems, we are interested in pure throughput, but with machines are interested in both: data freshness as well as total data per second. Given that most “off-the-shelf” links report just their maximum throughput and not latencies, or actually-realized throughput, I thought it wise to test.

Many Links are Not As Advertised

I started doing these tests because I wanted to sanity check the USB link, which I assumed to be easily within range of 5Mbit/sec, given that USB 2.0 operates at 12Mbit/sec, and I was dead wrong. So, that’s a lesson learned: measure your systems.

This is because we normally talk about a link’s “underlying speed” when we discuss; it’s easy to measure how long it takes for one bit to be transmitted. However, with all of the rest of the framing (software on either end, protocol, etc), measurements get messier.

Even the Teensy’s USB Serial, which is wonderful compared to the rest, gives us 20Mbit/s on top of a 480Mbit/s link (others use, AFAIK, the 12Mbit/s USB spec).

For a common-day example of this, observe that most SD cards advertise above 80MB/sec, but we typically see ~ 20MB/sec in practice.

Even on (what I assume to be) a highly optimized system: an SD card (80Mbit/sec advertised rate) plugged directly into my XPS 15’s SD card reader, we only realize, normally, 1/4 of that rate.

Simple and Device-Specific

I also had a bias of mine confirmed (so, beware the validity of this statement); simple, low-level systems, custom-built, often out-perform fancy, layers-of-abstraction codes. In embedded, where we all want to have portable code, but basically can’t really, this feels especially true. It’s also not that difficult to (for example) spin up a UART or a SPI peripheral from scratch, it’s just a rare expertise.

This also means that many of the results above could be improved with other folks’ better code, though I’ve started with what I can tell to be the best in the open-source business. There is a larger point to be made here about open source ecosystems vs. code portability vs. performance; the gist of which being that device-specific engineering is typically the route to performance, but the open hardware community is focused (at the moment) on trying to make code portability work in embedded systems (see Arduino and Micro/Circuit-Python, for example). IMO, this is not the way, but I will discuss that at more length at some other time.

Interrupt Handling is Often the Bottleneck

This is something I’ve seen again and again: we aren’t really bandwidth limited when we use simple embedded links: UART and SPI, for example. The drivers can drive it, the rx’ers can catch it, and the noise is “not so bad”. Under heavy loads, though, we become limited by the time it takes for our MCU to service the associated interrupts.

This has another hairy connotation for open hardware: it means that, “normally,” we need to know how hard the MCU is going to be working before we can appropriately engineer the networking links. If the MCU is terribly busy handling SPI comms to a sensor, then we come along and ask it to please deliver those data over a network link at some absurd rate, we may very well lock it up, miss some readings, or miss some packets. So! More trouble for layer-agnosticity.

What Next ?

(1) These codes are not optimal (implementations) and might be worth trying again with more focus/time, esp. the RP2040 PIO, which seemed so promising to me. Also the WizNet SPI library could probably be done up all-fancy like, to delete some of that blank time.

(2) We can/should have more-better high level protocols (CRC, flow control, perf measuring) for these “generic” low-level interfaces (SPI, UART) that would make it easy to bootstrap new MCUs, and promote their peripherals to proper link layers. A goal of that development should be that they can auto-test themselves routinely, so that we can do this kind of work in-situ.

(3) There are many more to test: Teensy’s Ethernet implementation probably rippeth, the SAMD51 (another favourite MCU of mine), the ever-popular ESP32’s Bluetooth and WiFi links, etc etc.

That’s all for this one, TTFN.

Searching for the Commons of OSHW

Mon, 16 Oct 2023 00:00:00 -0400

Open Hardware has the potential to change the way we make and maintain our world - but we need to learn how to put our heads together to do it.

This post contains ideas that I’ve been bothering my friends about in conversation over the course of many years, and whose inputs (and prior works!) have been hugely valuable, so, shoutouts to Nadya, Sam, Anna WB, Quentin, Leo, Ilan, and many more - I am lucky to have found myself in this community of passionate folks!

The Question of Open Hardware’s Sustainability

Open Hardware is facing a small crisis; we have long standing incumbents who feel ripped off and a simmering of similar sentiments in the community. Many companies that were “open by default” are reconsidering or switching to hybrid and closed-source models as they discover that cost-competing with global manufacturers is difficult to do whilst also maintaining R&D, Documentation and Support roles. Indeed, many of these companies find themselves working to support users who have purchased their designs from some third party vendor.

At the same time, it was always the point that Open Hardware meant Open Hardware: unprotected designs that could be replicated elsewhere. Until now, OSH businesses were relying mostly on a community ethos to survive; hoping that buyers would choose to purchase from the firms who published the original designs¹. This was based on a shared understanding that everyone benefits from open designs and interoperability, and in the FDM world that ethos worked for a long while, ushering in a “next generation” of machines that are radically more reliable, easy to use, and faster than the prior art.

Many of the hard-won insights uncovered by the Open Source FDM community have been recently deployed in the hugely successful (but closed-source) Bambu P1P, causing some to question the viability of open development practices in a field where incremental costs are non-zero. In this write-up, I want to show how we can have our cake (shared knowledge, community of practice, and Open Source) and eat it too (making profits, making complex systems, and making them easy to use and understand).

This level of refinement is the kind of engineering that normally takes place within “real” firms and behind closed doors and with centralized coordination and control of development. With the newest FDM generation, we saw that Open Hardware can do what Open Software has been doing for a while: replace centralized design of complex systems with distributed groups of individuals working asynchronously with no real coordination other than a common goal. The result of the FDM community’s work is a profound drop in cost (towards $200) and radical improvement to the quality of machines that just about anyone can buy and use in their own home to build designs of their making… it is worth a moment to appreciate what a great outcome that is.

But in some way, the FDM community has been done in by its own success: the dominant design patterns and technologies that have emerged from their hard work have become ubiquitous enough that the average FDM user doesn’t recognize where they came from, and so they simply purchase the best and the cheapest incantation of these designs that they can get their hands on. Manufacturers are responding, and it seems that as soon as one participant breaks the game-theoritic trance and develops open designs in closed source products, the ethos threatens to dissolve.

On the Flip

At the same time, Open Hardware is showing huge promise: during COVID-19 we saw collective action where individuals turned their FDM machines into PPE factories. Digital Fabrication is continuing to make its way into STEM education, and a new generation of young engineers is growing up in a land filled with Arduinos, Breadboards, Fritzing and FreeCAD. Every year it seems like one or two new Open Hardware Machine Companies emerge; this year we got an excellent pick and place machine, 2024 may be the year we finally get ‘the prusa of milling’.

Organizations like OSHWA and Hackaday / Crowdsupply and Hackster are expanding, and scientists and academics are increasingly turning to OSH in order to increase accessibility of experimental hardware. GOSH has emerged around Open Hardware in Science and organizations like BOING are building open instrumentation, an effort that the CBA (where I work) is also contributing to… we imagine a future where scientists who publish data also ubiquitously publish designs, algorithms, and controllers for their experimental hardware - and where that hardware is easily sourced, assembled, and commissioned so that others can re-create and expand the frontier.

What’s the Vision, and What’s the Point?

What’s at stake is, in my opinion, the most important quality of the whole human project; our ability to put our heads together and make meaningful stuff - to build off of one anothers’ intellectual outputs as we pursue our personal and common goals.

For a long time, the dominant narrative around the design and production of complex things (engineering artefacts, let’s say) was that it took place within firms, was mostly proprietary, and that the outcomes were protected by patents. This has never been completely true², but Open Source Software blew it entirely out of the water as it came of age in the late 90’s / early aughts. Open Software completely changed the way we write code, and to have similar impacts in the realm of real stuff would be meaningful in ways I probably do not have to explain at length: with hardware we’re talking about healthcare, housing, agriculture, transportation, energy, science… not just apps.

What made Open Software so powerful was a reorganization that started as recently as 2008; it transitioned from a focus on monolithic, end-user facing products (like LibreOffice and Linux) to a commons of functional modules that are rapidly re-assembled into application specific tools. I think that Open Hardware is on the brink of a similar transition, and that it can lead us to a world where we can have our cake (interoperability, shared community of practice, open designs) and eat it too (making profits, making complex systems, making them easily understood and replicatable, and making sustainable hardware companies).

What’s in This Post

In this post, I want to start exploring this idea of modularity in Open Source, looking in particular at well-developed theory from the sphere of Open Source Software. The goal is to learn what we might apply from that domain into ours, in Hardware. This breaks down into a few sections:

(1) Some motivating thoughts, exemplary failures, and historical lessons from interoperable computer hardware.

(2) Insights from economists and practitioners of the (hugely successful) Open Source Software ecosystems: how do they work and how do they emerge?

(3) Understanding key technical properties of the commons.

This review should elucidate the core theory around Commons Based Peer Production, and ground it in a discussion of hardware modules. Over the next year, I want to follow this up with two or three more posts; one that provides an overview of the “state of the commons” in open hardware, a second that proposes device-to-device interoperability as a way forwards towards an even more productive commons, and perhaps a third that goes into more detail on the foundational economics and imagines business models that could thrive in the commons.

1. From Monoliths to Modules

“Hardware is Hard” is an adage that gets thrown around a lot in the Open Hardware community, but I don’t think it needs to be this way. The utterance normally comes from folks who are accustomed to writing software and then find themselves in the wee hours of the morning trying to reconcile their circuit schematic with a sensor’s datasheet and someone else’s pinout diagram. In software development, we don’t have to sweat these details: libraries and lower layers are successfully encapsulated into (mostly) well-designed APIs, and we simply invoke them with function calls in our favourite language. The result is that we get to use tens or hundreds of other developers’ expertise in our own projects at a low economic and intellectual cost: we don’t pay for them, nor do we have to invest hours of learning in order to understand how to use them.

With a little bit of effort, I think that Open Hardware can become this easy. We are reaching a critical mass of expertise and energy, the cost of embedded computing continues to drop, and more and more valuable systems components and design patterns are emerging. There is a chance to transition from monolithic to modular contributions, enabling open hardware to cross a threshold into the production of complex, systems-scale artefacts. This would have ramifications wherever we design and build hardware; from toy-building to energy infrastructure and medical equiment. Success could mean that:

Individuals and firms would be able to build new mechatronic equipment as easily as they write new software today (in weeks, not years).
More individuals would be engaged in the active making and maintenance of our world: more meaningful jobs and more ‘hands on deck’ to face meaningful challenges.
Society as a whole would be better equipped to face radical changes: economic upheavals, pandemics, existential (warming) threats - by producing real, useful artefacts and infrastructures at a lower engineering cost.

When we take a cold-hearted look at the state of open hardware today, these three outcomes can seem fairly pie-in-the-sky. On the other hand, when we think of how many engineering-hours are saved with Open Source in software systems, and how many large firms have adopted OSS as a core practice, it is hard to deny that open access to a commons of hardware designs and design patterns would represent a vastly more efficient economy than the one we see in action today, where re-engineering is the norm.

I would also call upon the doubtful reader to consider that as recently as 2004, Butler Lampson reported that “a general library of software components […] is unlikely to work”³ - just a few years later, node.js emerged - which currently plays host to over 1.3 million such modular components used by millions more developers. And if Open Hardware seems ‘too dinky’ to face ‘real engineering challenges’ in the future, I would also point out that the worlds’ best cryptographic algorithms are Open Source (more eyes see more potential pitfalls), and that the whole Personal Computing revolution was founded by small groups of hobbyists working “for fun” on systems that basically just blinked.

1.1 Scoping the Discussion

All of this is background to work that I do mostly in the area of machine building on modular motion controllers, feedback programming models, and networked systems assembly tools. When we talk about hardware it’s very easy to go off the rails, since the whole world is made of it. In this discussion, I want to stay focused on functional blocks of hardware, and since the world is filled with (and mostly operated by) computing these days, I’m mostly taking about mechatronic modules. These are components that we can imagine re-using in multiple contexts like motor controllers, encoders, computer visions solutions, end-effectors, etc; the things that are often the most valuable parts of a project, but are the most difficult to copy from someone else’s contribution. The Mechaduino is a great example:

1.2 Mechaduino: in Search of a Systems Integration Tool

when do we get to the point where, if I plug something in, it just works? 
  - Joe Jacobson (Lab Lunch, 2021)

Getting to this level of modular interoperability probably needs a new kind of systems integration (SI) tool, something like an NPM for Mechatronics. To explain what I mean by this, I want to remind readers of the Mechaduino. This is a closed-loop driver that represents a clear improvement in positioning accuracy, speed and reliability for one of the most common components in open machine design (the NEMA 17 Stepper Motor).

The mechaduino was an early OSH module that represents a significant improvement (open to closed loop) on motor control. However, since its launch in 2016, no open motion controllers can integrate with it appropriately.

This is a great example of a hardware module: it packages the developers’ expertise in a niche area (closed-loop control of a stepper motor) into a functional block for an application that many other developers are interested in (motion systems). But the Mechaduino was missing a larger architecture to live in; users were left to customize and re-compile the device’s firmware in order to integrate it into their application, and it didn’t serve a clear interface with which it could be “plugged in” to other systems. Although it was released in 2016, the mechaduino has not yet made its way into any other substantial open hardware project, nor has its core design pattern (although many clones exist, even I have designed a few!).

This is perhaps because closed-loop drivers are not natively supported by open motion controllers, or because (more broadly) we don’t have a good framework for projects that deploy on multiple microcontrollers (i.e. a motion planner talking to multiple modular motor controllers simultaneously). It may also be that the mechaduino is simply more expensive than its alternatives (at $65 vs ~ $20), despite my own anecdotal evidence of seeing many machine builders over the years spend above $250 per motor for proprietary closed-loop solutions. It seems that the mechaduino was a great module without a modular architecture to live in.

Rendered from the NPM graph viewer, the NPM package express - a server middleware code - easily bundles tens of others` modular contributions into a new contribution.

If we look instead at Open Software, interoperability is the name of the game. Contributions are made of other contributions, and are all integrated seamlessly within some shared language: JavaScript or Python, for example. I think this is worth a second thought: programming languages are systems integration tools. Any function written in Python can be called natively in anyone else’s Python script. This is obvious, but it is also Open Software’s super power: between high-level languages (with standardized self-documentation practices to boot), and the operating system (which abstracts hardware heterogeneity away from systems assemblers), anything written anywhere can be deployed anywhere else. Open Hardware will have to work harder for interoperability.

1.3 For Reference: IBM’s Open Hardware Mistake Was a Silicon Valley Catalyst

Interoperability is also the basis of computer hardware at large - servers, PCs, etc are all made from components that work together based on well defined standards. It’s an economic pattern that we take largely for granted, but which has a worthwhile origin story.

In the 1970’s, IBM’s flagship mainframe was the Systems/360 - and it represented the first computing system to intentionally generalize across markets: the 360 was used in business and scientific applications alike. It was also one of the most complex systems IBM had ever developed. A modular systems architecture helped them take on both challenges.

Images from IBM. System/360 was an early mainframe computer that paved the way for modularity throughout the IT sector.

Internally, modularity was a wonderful engineering-labour-coordination trick: it meant that when someone in the user-input team made a change, it wasn’t coupled to any design choices being made in the memory-control team⁴ (for example). Today, this kind of separation of interests is commonplace in systems design, but the 360 was emerging in a world where prior art was mostly monolithic, and where the first nuggets of information hiding theory and abstracted programming languages were still just emerging⁵.

Image (and much of this sidebar) from Design Rules Vol. 1. Figure is of a Design Structure Matrix^ref that relates tasks (and designers) to one another. Tasks are enumerated on both axes of the matrix, and where two tasks constrain one another we mark an x. A design task is modularized when we can diagonalize that matrix to some high degree.

Externally, modularity allowed IBM to sell their mainframe to a wide selection of customers. Consumers of computing at the time were highly sophisticated users (banks, universities), who hired groups of staff to run their machines - not unlike the current state of the art with industrial Digital Fabrication - these users wanted to be able to modify their equipment, and understand it. So, IBM also released the 360’s instruction set, bus architecture spec, and the busses’ plug design.

Image via WikiPedia contributor Tom94022. System/360 was an early “plug and play” architecture.

This has some real inlkings of the so-called platform economy,⁶ that emerged decades later, but IBM forgot to do the lock-in part, bless their hearts. As a result, their market share tanked - as tens of other high-tech competitors emerged.

Image from Design Rules Vol. 1. When IBM released the System/360, their open architecture led to the emergence of many competitors, who ate up much of their market share in later decades. IBM`s share is in the back of this image, in blue.

This was bad news for IBM, but it was a major contributor to the historical bifurcation that made the Silicon Valley ecosystem into a world center for innovation in the following decades. Baldwin and Clark describe the resulting situation well:

“Our theory is that modular designs evolve via a decentralized search by many designers for valuable options that are embedded in six modular operators.”⁷

Basically, we got into a situation where all of these (now big-and-fancy, then small-and-scrappy) firms were effectively working together to improve one modular system: the modern computer. Parts were interchangeable, individual contributors could focus on highly specialized parts of the whole, and the systems still clicked together and worked - even though very few of the specialized engineers even knew who the others were, or how their counterparts’ contributions worked.

The System/360 modularity lives on in today`s (mostly consortia-standardized) interoperability layers. Engineering Rules covers their development, maintenance, and dissemination.

Over time, the underlying architectures mutated as well, and new ones popped up. The particular design of the System/360’s architecture didn’t survive beyond the mainframe’s existence, but the design pattern survived and improved, and it continues to do so⁸. It is still possible to build your own computer using graphics cards from AMD or Nvidia, CPUs from AMD or Intel, memory from tens of different vendors, etc etc etc - all of this happens because this modular architecture is still with us, today in the form of tens of different specifications: DDR, PCIe, USB, … Standards that are developed and maintained by numerous consortia and working groups, which (if you’re into that kind of stuff) is explored in depth in Yates and Murphy’s Engineering Rules⁹.

Economists all basically agree that this situation is universally beneficial: there are minimal re-engineering efforts, there is less monopolization, and we end up building better-performing systems overall. However, it seems difficult to get “into the groove” of it, so to speak: it requires large activation energy or, in the case of the Systems/360, it requires that a major incumbent makes a strategic misstep.

There’s of course more to the info-tech story than I am presenting here, especially in the decades of work spent building the technologies that bring modularity to computers (compilers, languages, ISAs, etc) - as well as this story on ‘unbundled software’ - inall, I think that the resulting economic setup is fascinating - even some kind of a miracle - and it provides a good backdrop for the rest of this discussion.

2. Commons Based Peer Production

On top of all that interoperable hardware, we eventually built Open Source Software (OSS), and since the late 90’s, economists and social scientists have been trying to figure out how it works. Not how the code works, but how the people coordinate themselves to work on the code.

The first mystery was that highly talented individuals would give up their valuable time and energy to build free things, and the second (larger) mystery was how they were able to build complex things without central coordination. Software like Unix, Apache and LibreOffice was normally built using hierarchichal coordination within a firm, but then suddenly there were these FOSS nerds doing it on message boards without any real bosses. Sounds cool, but it didn’t make any sense in the old models.

The mystery has since been mostly solved, though the landscape continutes to evolve. Economists¹⁰¹¹ came to a few conclusions:

(1) People are naturally motivated to build stuff that they’re interested in.

(2) People are excited to share those things that they’ve built, and see their contributions realized in the world.

(3) Radically open “labour markets” like those existing in OSS projects (where anyone can work on anything) do a better job of matching interest and talent to problems that need solving (or that require invention, etc).

So, Open Source turns out to be a really, really good way to build things that require specialized skillsets - like Operating Systems or, perhaps next up, robotic systems. It also requires good plumbing so that individuals’ niche contributions can easily be assembled into complete systems.

The most revelatory book I read on the topic (Working in Public by Nadia Eghbal) ¹² compares OSS to social media, showing that folks are also motivated by clout, and that most software these days is “open by default” (whereas earlier ethos were more evangelical¹³¹⁴¹⁵). It also cites Commons Based Peer Production (CBPP, a theory developed by Yochai Benkler ¹⁰) as the working method behind OSS. CBPP is “a form of collective intelligence”¹⁶ that most software developers will these days be well acquainted with. In CBPP, there exists a large collection (a commons) of generalized, functional building blocks (libraries and packages) that can be easily integrated into new, specific applications (using APIs and high-level languages).

Commons Based Peer Production is an economic mode of production linked to a practice of systems development (pictured here), where specific applications are made from a Commons of generalized functional modules. CBPP is prevalent (and powerful) in Open Software, but relies on technical infrastructures that are somewhat difficult to replicate in Open Hardware.

These commons are wonderful repositories of human intelligence, but not without their flaws. In particular, it is difficult to regulate the quality of their contributions, especially when contributions are made in multiple tiers (dependencies of dependencies, etc…). There are horror stories of singular, simple packages breaking entire swathes of the internet. That said, packages are chosen by intelligent developers, use-base consensus tends to emerge as to which are most useful, and safety and stability is part of that selection process.

XKCD: Dependency and XKCD: Python Environment. The systems that make up CBPP have many flaws, but its overall usefulness and value is clear to anyone who writes software.

CBPP also stands in contrast to earlier Open Source ethos’ in its positioning; proponents of the 1990’s era movement supposed that legions of OS developers would build prevalent end-user facing products (word processors, operating systems etc) that would liberate end-users from for-profit software companies. “Why pay for code when incremental costs are zero?” was a sensible and courageous stance that didn’t pan out: Microsoft Word still Reigns Supreme.

While LibreOffice developers and their ilk were on a valiant mission to deliver free software to the people (but sort of failed), the current paradigm of CBPP has just about everyone on earth consuming Open Source contributions without their even knowing it - and it seems to have done it accidentally: as a by-product of developers just building better components for one another to use.

Early models of Open Source (left) supposed that large groups of developers would work together on large, end-user facing products like word processors. The dominant, emergent model (right, Commons Based Peer Production) is one where small groups of developers work on small, functional building blocks (libraries and packages) that are accumulated in commons (like PIP and NPM) and subsequently re-assembled by other developers into end-user facing products. This has the effect of freeing Open Source developers, who are normally motivated by personal interest, from building “the boring components” of any given system (like UIs, user manuals, etc) and allows them to make contributions on “fun” parts like advanced algorithms, architectures, data structures, etc.

Open Source turns out to be better at producing infrastructural codes (server operating systems, network protocols, security algorithms) and generalized software modules (databases, mathematic routines, graphics tools, form validation…). This is partially because those are the fun problems - and OS developers prefer the fun problems - and also because none of them require OS developers to author much in the way of user-facing documentation. Or, rather, that their users are other developers, and a community of practice and common knowledge means that documentation can be minimal.

This is another moment worthy of a second thought. At the moment, most Open Hardware projects need to be documented up the wazoo. When I deployed clank! for HTMAA students in 2020, I had to explain to everyone what a socket head cap screw was, what a NEMA 17 motor is, etc etc; we have students who have never held a power drill. Documenting machine kits to a level that everyone could be successful was a huge undertaking, one that I haven’t had the time to do since then, even though I have improved designs. Were it the case that we had a recognizeable community of machine builders (and it is emerging), who shared contributions in a collective practice, the documentation burden would be much smaller. This is all just to say that speaking to a topic expert about their topic requires far fewer words: and that is effectively what happens in Open Software and what could be happening in Open Hardware. Two programmers need only explain their APIs to one another, not how to instantiate an object, how to call a function, etc etc…

… OK, we were talking about how Open Source is consumed like a public good. Well, those infrastructural codes and modules we were talking about, those are widely used by other developers who package them up into application-specific user-facing products that they often make a profit on. This might seem like a devilish take for the Open Source evangelicals among us, but they should consider just how little most software costs, and just how many niches it is able to fill. Good software would be more expensive and less widely applicable if it were not written on the back of the commons. As Nadia Egbhal relates, “Open Source is produced like a commons but consumed like a public good.”¹²

Along this tangent is a field of research on information commons writ large (like Wikipedia) of which OSS is one of many, and of which OSH would be one as well (itself not really being made up of hardware, but of hardware designs). For background in that domain we have i.e. Elinor Ostrom¹⁷¹⁸ and Lawrence Lessig¹⁹. There is also fascinating work on how open-ness (as in, information sharing) helped to advance the design of steam engines in the very early industrial revolution²²⁰, as well as Eric von Hippel’s famous exploration of Open Source patterns in extreme sports and other hardware²¹, and Baldwin and Clark’s really-super-neat work on modularity in the hardware of information systems⁷ (i.e. computers) whose development they describe as another form of collective intelligence¹⁶. Finally, we shouldn’t forget that much of the hardware world is already made from a commons of standardized components; this is well documented in Yates and Murphy’s Engineering Rules⁹.

2.1 Design Patterns Proceed Direct Integration of Designs

So, Commons Based Peer Production is a wonderful framework to understand how our technical world can be built out of a kind of collective intelligence, but it mostly explains how direct commons operate once they have been established. Since this little essay is supposing that OSH would be helped along by establishing more of these types of commons’, we want to understand also how they emerge.

To differentiate, we call a direct commons any identifiable collection of designs - these are concrete, dimensioned, rigorously specified drawings, source codes, schematics, circuit layouts, documents, etc: these are all contributions that can be immediately put to use. Before these emerge, we can identify commons of practice. These are just as real, but they are made up of design patterns; ideas about how to generally go about making a thing. This might include sets of kinematic arrangements like corexy, belt tensioning strategies, embedded programming patterns like static memory allocation, etc. Design patterns could probably be conveyed successfully in conversation, or on a whiteboard, but designs require more tooling: dimensioned drawings, file formats, etc. Between direct commons and common practices, the latter are probably more important overall, but they’re squishier: harder to delineate and measure. They also tend to emerge as pre-requisites.

I like to take Object Oriented Programming (OOP) as an example. The idea has been around for nearly half of a century, kicked off with Parnas’ ideas about information hiding²²²³, and popularized by the likes of Christopher Alexander²⁴. So, it was in the soup - and then formalized into handfuls of programming languages. Without this idea and the associated ethos (encapsulation, modularization, organization of function), we wouldn’t have anything like PIP or NPM, whose users deploy this design pattern without thinking twice about it.

Design Patterns emerge first, and last longer than literal Designs themselves. Common understanding of a set of Design Patterns is normally a pre-requisite for the emergence of a Direct Commons.

This is all to say that need a collective understanding of how our shared systems work before we can succesfully use them; NPM requires OOP and APIs which require a broader idea of information hiding. Open Hardware might need some new (or more practice with) design patterns as well: networking, concurrency, asynchronousness, reflective programming, feedback systems, interface design, etc.

For more thinking on shared design patterns, consult Nuvolari² and Allen²⁰ who showed that openly disseminated plans for steam engines (at left in the figure above) formed a backbone of the English industrial revolution. Eric von Hippel²¹ explores how design patterns bounce around in communities of users, and looks in particular at how innovation happens ‘on the ground’ in niche extreme sports (at right).

2.2 Commons Emerge Around Systems Integration Methods

Once a common practice is established, there is viable grounds on which to build a direct commons: a place where literal, immediately-useful designs can be shared. But there is one more component needed, which is some organizing integration method or deployment tool.

This notion really struck me when Prusa renamed their design-sharing site to just “Printables” - it’s a commons made up of designs that can be deployed immediately by any member of the community who knows how to use a 3D Printer.

NPM
Things you can use in your JavaScript program.

PIP
Things you can use in your Python program.

Arduino (libraries)
Things you might be able to use in your Arduino project.

Printables
Things you can Print!

These all have some caveats, mostly because users’ actual systems are all a little different: i.e. some Printables Prints will only work on multi-filament machines, many Arduino libraries are not compatible with all Arduinos²⁵, and NPM and PIP packages will grant variable milage depending on their users’ setups: typescript-or-not, esm-version, dependencies… The world is heterogeneous, and commons-es require some homogeneity, but luckily not absolute homogeneity.

And that’s the point here: a stable method of integration provides the homogeneity required for a technical commons to emerge. NPM and PIP were built on the back of JS and Python’s ubiquity, Printables on the ubiquity of the FDM printer. Those are simple examples, but Open Hardware in general is an umbrella of many other disciplines, each of which has variable integration systems (most of which are either ad-hoc or proprietary). But, it also points the way: if we can make good systems integration tools for various sub-disciplines in hardware, we stand a better chance at building in the commons.

Thinking about this might also help us identify missing commons. For example, everyone is basically agreed on how to draw a schematic for a circuit, and how to render and share design files (with Gerbers). BOMs, also, are relatively easy to standardize. GitHub is (so far) poorly suited to host these types of content because it doesn’t have built-in tools to render them and because some of the formats are difficult to track using git’s versioning system. However, kitspace has emerged as a commons for circuits and it provides the utility of automatically ordering boards and BOMs of anything contributed there.

3. Technical Properties of the Commons

So, we see that together with a set of ubiquitous design patterns, systems integration tools enable commons’ of directly shareable designs to emerge. For modularity, we need modular systems architectures. A quote from Benkler is worth our time here:

The remaining obstacle to effective peer production is the problem of integration of the modules into a finished product. […] In order for a project to be susceptible to sustainable peer production, the integration function must be either low-cost or itself sufficiently modular to be peer-produced in an interative process.
From Coase’s Penguin, or, Linux and ‘The Nature of the Firm.’¹⁰

Researchers who study these tools note that many of them exhibit similar properties, each of which gets a sub-section here.

3.1 Modularity

This property is pretty obvious and tautological; commons’ are made of modular bits. Hidden behind this is the more complex requirement for a module to generalize - even if we make a stepper motor driver, say, if it doesn’t generalize across machine and robotics and home automation applications, we don’t have a very good module. Hardware modules have the added challenge of scaling across applications: the motor we want for an excavator is a very different one than we want for a surgical robot, so even in domains where functional blocks can be abstracted using similar descriptors, their hardware instantiations will still need to be wide-ranging. This one of the reasons why I’m also trying to build self-tuning motion systems, to automatically align generalizeable, abstract models with the real world’s heterogeneity.

Modularity always poses a challenge to interface and API designers: how to expose enough of a component’s functionality so that it can be altered for multiple uses, while not over-burdening users with configurations. It is normally impossible for module designers to foresee all possible uses of their contribution, so API design remains something of an art; it is also a domain that will get many software engineers hot under the collar… strong opinions abound, and there are not many rigorous methods available with which to evaluate them.

3.2 Granularity

One of the core tensions of modular systems design regards the size of a module. Larger modules can be more performant and faster to write for a specific system, but they tend not to generalize as well into components for other applications (complexity tends to go hand-in-hand with specificity). Larger modules also require more time up-front to understand and edit, and can be harder to modify down the road; technical debt is commonly found in the form of ad-hoc, monolithic chunks of code that are difficult to decipher.

Successful commons’ tend to be highly granular, meaning they are made up of many small contributions rather than a few big ones. This has the technical advantages of comprehensibility and extensibility but it also has the economic function of breaking complex work into smaller chunks of labour. CBPP theorists point this out as an important feature because most contributors to Open Source do so “on the side” - meaning that they don’t have the weeks and months of time required to author large, complex contributions. Instead, they have nights and weekends and tend to make small additions at a time.

3.3 Minimal Burden of Integration

This is perhaps the most important property of a succesful commons: it should be very easy to contribute a new module to the commons, and it should be very easy to deploy a module into an application. This extends to the perhaps unintuitive notion that, in order to use someone’s module, we should not have to understand it, which is where the notion of collective intelligence¹⁶ emerges. If we can use the outputs of one anothers’ intellect without comprehending it, the systems we build from those components can embody an intelligence greater than our own and we get to keep some left-over brainpower with which to make our own new and specialized contribution.

We’ve talked about how easy module integration is with software, where programming languages are de-facto systems integration tools and modular design patterns are already diffused in the discipline of computer science. That baseline has been aided as languages have added first-class support for module importing to further reduce the burden of integration, like ES6’s import statement (that was originally a node.js kludge). In languages like C++, where no language-level import statement has existed (until recently), no ubiquitous package manager (or commons of libraries) exists.

The arithmetic on why this metric is important is pretty simple; developers weigh the time it takes to contribute their module to the value they get from contributing it. Where incentives are small, the time required needs to be small. If developers don’t contribute, the commons doesn’t grow. Similarely, if it’s difficult to include someone else’s work, users are more likely to write things from scratch - meaning they will build less of the new stuff, and our collective ability dwindles.

Along these lines, I recently pushed an update to Modular Things that makes it easier to build high-level computing interfaces for hardware modules, in an effort to lower the barrier for device-to-device integration (by allowing folks to “do it in software”).

People are also not likely to use an ecosystem of components when plugging into that ecosystem is difficult, or when large changes to their existing project are required in order for them to interface with it. This is why the start-up cost of these commons’ is often quite high. We need to build out enough of the systems such that developers will be incentivized to use it as a first-class citizen in their projects. Or, systems need to be simple and flexible enough that they are easily added to existing projects.

4. To the Commons !

So, this ends Part One. I hope you’ve seen how the commons can help us build complex and meaningful technologies using open contributions, and how they emerge and grow - and what we need to build to enable them.

Perhaps you are thinking “gee, much of OSH is already looking like this” - and you would be correct; I plan to explore that topic next in more detail next. I think our current state is modular-ish, but that we haven’t accomplished true ‘plug-and-play’ interoperability that lets a commons flourish.

I want to return to this central idea: OSH is trying to build end-user facing products with a lot of re-engineering. Instead, we should look towards modular systems, and build a commons of interoperable parts. This would let us build a more diverse set of more complex artefacts that are easier to use and understand.

But I started by considering the plight of Open Hardware firms, who are still relatively plucky (vs. their Open Software counterparts, and vs. the industrial incumbents who dominate our economy). A lot of the people I talk to want to launch these kinds of firms. They want to build stuff, to do R&D, maintain a shop, make meaningful contributions and share them in the open. This means they effectively want to donate their expertise as a public good: free and useful information. They just want to do it sustainably, and they probably want a little bit of clout, as a treat - and as they are warranted.

I think that governments, businesses and academic institutions aught to be listening to them, and that we should all be trying to figure out how to enable their dream; these are talented individuals whose other door opens into giant tech salaries and NDAs with businesses that dont pay their taxes. It would be a shame if we let this generations’ engineering talents continue to build up those walled gardens when we have such a promising opportunity to divert them into a shared pool of collective intelligence and wealth, and so many meaningful problems to solve.

I think that building the commons is where it begins, but it is for sure not the whole picture… next up is a discussion on the current state of modularity, then a look at device-to-device integration systems, and finally I’ll explore some forwards-looking economics for the commons.

Notes / References

Consumers are also incentivized to to purchase from original developers because they know they are getting a product that has been manufactured to the original intent, whereas third-party vendors occasionally use i.e. off-brand chips that cause compatibility issues, or otherwise cut corners. ↩
Nuvolari, A. “Collective Invention during the British Industrial Revolution: The Case of the Cornish Pumping Engine.” Cambridge Journal of Economics, vol. 28, no. 3, May 2004, pp. 347–63. DOI.org (Crossref), https://doi.org/10.1093/cje/28.3.347. ↩ ↩² ↩³
Lampson, Butler W. “Software components: Only the giants survive.” Computer Systems: Theory, Technology, and Applications. New York, NY: Springer New York, 2004. 137-145. ↩
Today, researchers use Design Structure Matrices to study these kinds of couplings. DSMs are also one of the only quantitative manners I have found to measure modularity, and they’re a useful thinking tool. For more, there’s Alan MacCormack’s Exploring the Structure of Complex Software Designs: An Empirical Study of Open Source and Proprietary Code. wherein the structure of Mozilla Firefox is analyzed in the context of Open Source (Firefox’s authors were preparing to OS the code, and wanted to do a modular refactor beforehand; they understood that the code would only be successful as an open project were it made modular) and similar work: The Impact of Component Modularity on Design Evolution. ↩
The first notion of information hiding that I’ve found is Parnas’ (1972) On the criteria to be used in decomposing systems into modules, also-interesting is his (1985) The Modular Structure of Complex Systems. Since we’re on the topic, I think the first paper is a fascinating read in the context of hardware vs. software modularity because it reminds us that compilers remove modularity when they optimize a program… this seems to be a good reminder that modularity, which seems to easy in computing systems, actually tends to come with a huge performance cost. We get away with building modular systems in software because we have automatic translators (compilers) between bespoke systems (code as compiled) and the systems representations we use to design them (modular codebases). Bespoke things are faster. Everyone knows this, it’s why bespoke or proprietary is synonymous with hot, fast and sexy. In hardware, we have to reckon directly with the thing itself, and live with the real-world costs of our modular systems descriptions. Maybe. ↩
This is hot in the business science world, or it was a decade ago. See i.e. Martin Kenney’s (2016) The Rise of the Platform Economy. ↩
Baldwin, Carliss Young, and Kim B. Clark. Design rules: The power of modularity. Vol. 1. MIT press, 2000. ↩ ↩²
For example Intel, who championed the currently dominant instruction set (x86) was founded in 1968, two years after the System/360s release. We are already seeing a shift to ARM-based instruction sets, which itself should give way to RISC-V, an open-source ISA. ↩
Yates, JoAnne, and Craig N. Murphy. Engineering rules: Global standard setting since 1880. JHU Press, 2019. ↩ ↩²
Benkler, Yochai. “Coase’s Penguin, or, Linux and ‘The Nature of the Firm.’” The Yale Law Journal, vol. 112, no. 3, Dec. 2002, p. 369. DOI.org (Crossref), https://doi.org/10.2307/1562247. ↩ ↩² ↩³
Lerner, Josh, and Jean Tirole. “Some simple economics of Open Source.” The journal of industrial economics 50.2 (2002): 197-234. ↩
Eghbal, Nadia. Working in public: the making and maintenance of Open Source software. San Francisco: Stripe Press, 2020. ↩ ↩²
Raymond, Eric. “The Cathedral and the Bazaar.” Knowledge, Technology and Policy, vol. 12, no. 3, 1999, pp. 23–49. ↩
Stallman, Richard. “Why ‘Open Source’ Misses the Point of Free Software.” Communications of the ACM, vol. 52, no. 6, June 2009, pp. 31–33. DOI.org ↩
Stallman, Richard. The GNU Manifesto. 1985. ↩
Benkler, Yochai, Aaron Shaw, and Benjamin Mako Hill. “Peer production: A form of collective intelligence.” Handbook of collective intelligence 175 (2015). ↩ ↩² ↩³
Ostrom, Elinor. Beyond Markets and States: Polycentric Governance of Complex Economic Systems. Nobel Prize Lectures, 8 Dec. 2009. ↩
Ostrom, Elinor, and Charlotte Hess. Understanding Knowledge as a Commons: From Theory to Practice. MIT Press, 2007. ↩
Lessig, Lawrence. The Future of Ideas: The Fate of the Commons in a Connected World. “Chapter 2: Building Blocks” and “Chapter 14: Alt.Commons” 1st ed., Random House, 2001. ↩
Allen, Robert C. “Collective Invention.” Journal of Economic Behavior & Organization, vol. 4, no. 1, 1983, pp. 1–24. ↩ ↩²
Hippel, Eric von. Democratizing Innovation. MIT Press, 2005. ↩ ↩²
Parnas, David Lorge, Paul C. Clements, and David M. Weiss. “Enhancing reusability with information hiding.” Proceedings of the ITT Workshop on Reusability in Programming. 1983. ↩
Parnas, David Lorge. “On the criteria to be used in decomposing systems into modules.” Communications of the ACM 15.12 (1972): 1053-1058. ↩
Alexander, Christopher. A pattern language: towns, buildings, construction. Oxford university press, 1977. ↩
On the topic of messy-arduino-gotchas, I could go on for a while. While Arduino attempts to enable code portability across microcontrollers, it is a largely leaky abstraction: libraries compile on some devices and not others, libraries can collide with one another (when they both need to use the same UART or SPI peripheral, for example), and can also easily overflow any givne microcontrollers’ compute or memory resources. I think it boils down to the fundamental non-portability of embedded code. With microcontrollers, more than any other computing system, we need to align our programs with their physical situation: not just the particular MCU running underneath the Arduino layer, but also the circuit that the MCU is attached to. Arduino basically relegates this alignment-hunting to the CPP compiler, which is not made for such a task (compilers being purely feed-forward tools). Template programming might be able to do it, but that sounds like hell also. I remain convinced that doing Arduino’s job well requires a ground-up new-arduino, and probably something akin to a DSL, and probably something based on a more universal hardware description language for circuits. My buddy Erik is cooking something up along these lines, bless his soul. ↩

Clangers and Bangers

Fri, 21 Jul 2023 00:00:00 -0400

Werner Herzog does Music Machines

We had a fun week trying to make this off-the-research-piste project at Haystack this July. This was part of Haystack Labs 2023, and I had the pleasure of working with Christopher Taylor, Quentin and Ivana Dama.

In the Jewelery Studio at Haystack

About Composable Systems

I’ve always wanted to build some kind of prototyping kit for music machines. I’m not a musician myself but it seems that the medium is highly composable - meaning it’s hard to make permutations of a musical system that are fundamentally broken.

That seems like a crazy statement if you’re just interested in music making. I’ve spent a long while trying to build machine controller systems; i.e. for 3D printers, laser cutters, cutting machines and their ilk. We want their assembly, repair and modification to be playful, fun … inviting for newcomers. This is work towards the end that more amongst us might end up engaged in machines’ making and maintenance, and that as a result more could engage with the material world in a less alienating manner, or be more likely to sieze the economic means to do automation, precision fabrication, etc (and all that would entail).

In any case, when we build machine systems, there are lots of ways to break them. Things can go wrong pretty quickly, basically: machine systems are not very composable - that’s the whole challenge. On the other hand, music is composable - we can throw new things in the mix, improvise, bang on pots and pans etc, and nothing goes dangerously wrong.

Scheming on protocols, and schemas of keymaps.

What We Built

So we brought a kit of circuits for whacking (with solenoids) and thrumming (with transducers), and a new bus protocol we had been working on. None of which were especially finished.

Our ad-hoc pots-and-pans banger fixture.

The above, as deployed.

The results are evident in the video at the head of this page (and below); not wonderful, but kind of funny. I wanted badly to make something that would bring Christopher some joy as he played it, but I’m not convinced we managed it.

Christopher Taylor, virtuosic pianist (and talented programmer), banging and clanging.

I think Christopher did have a blast programming though - he helped debug our protocol, and invented the keymapping system we used (to translate midi from our keyboard into various solenoid-strokes).

What We Learned

This was a fun little misadventure on my part - and I even learned a few valuable things;

The world of Musical Systems Assembly is rich, and full of ad-hoc solutions that work.

I’m often complaining that the CAD/CAM/Machining world suffers from a lack of field mechanic-ness - i.e. the property (often found on i.e. farms, on the ISS, or amongst other infrastructure maintainers) that tools and systems are built, modified and maintained by those who use them.

In particular, I think that mechatronic systems present a barrier to this kind of work because their complexity is often large and those who use them (largely physical-world dwellers) tend not to intermingle with those who develop them (largely mathematics-loving, abstract-space dwellers). I suspect that CNC remains clunkly largely because of this.

There’s a much larger discussion in there, but the gist here is that musicians are good at making composable systems and their tools, which are authored largely by their own community-members, are awesome. I’m thinking puredata, MaxMSP, open sound control.

I’m thinking also about MIDI (~ three bytes per event), and Control Voltage (~ one volt per octave) - really slick, simple, and easily grokk’d protocols. It’s like, UTF-8 level extensibility.

One of the reasons these are so wonderful is probably that you don’t need to pull anyone’s code to start using them: they’re easy enough to understand that you can build their guts from-scratch in whatever system you want in a few hours. That might be difficult for embedded systems in general, but it might be critical.

Construction Kits need to deploy in a “Batteries Included” state.

We didn’t show up to this party with everything finished, we should have. That means earlier testing, and more scaffolding:

circuit should not be fragile
we shouldn’t have to wait for any 3D prints to finish before we can do stuff
systems should not require explainers

We already know most of this.

Construction Kits need out-of-the-box Examples

This is basically the same point as above; we should have four or five fun things to do with the kit before we leave the lab. They should work right away, and they should make what-all is going on behind the systems clear to everyone involved.

Fin

So, good to learn in practice what we suspected to be true, and to have some motivation to do a little better next time, scaffolding wise.

Huge thanks to Haystack for hosting us!

Jake Robert Read

Visting Shenzhen

From 2026 01 04

From 2026 01 20 (on the flight to Boston)

Canadian Architecture Student visits the Big City and Gapes at Buildings

Trucks of Shenzhen

The Markets: HQB and Yihua (and LCSC and JLC)

Working in the Markets

E-Commerce and MaaS

Engineers (and Artists) from a World of Manufacturing

Seeed HQ

An Aside on Open Hardware

Western Firms in Shenzhen

Factories, Automation, Tools

Seeed’s Flex Factory

A Flex PCBA Line

Injection Mould Making

The Motor Factory

About those Street Sweepers

The Blair Winch Project

What / Where

Homing, and Learning Where the Winches Are

An Aside: Solving w/ Unknown Offsets

Controlling the Orb

1: Where is the Orb ?

2: Move the Orb

MIDI -> Orb

What We Learned, What We Missed

More Power == Better

Control with Smarter Limits

Learning with Torques to boot

Learning All The Time

Whereis Python Good for Control

Linear Algebra is Neat and Sometimes Non-Obvious

In Summary

Field Oriented Control with (almost any?) Stepper

Stepper FOC vs. Brushless Direct Current (BLDC) Motors

The Hardware

The FOC Algorithm

The Magic: Current Control in a Stable Reference Frame

Current Sensing

PWM Aligned ADC Sampling

Interleaving Other ADC Readings

Calibrating the Encoder

LUT Trigonometry for the Transforms

Filtering Encoder Readings

Kalman4Idiots

“Alpha Beta Gamma” Observers

Current Controller PI Gains

A Position PID Loop, and Loop Timings

Measuring Motor Parameters, and some Modelling

The Canonical Motor Model

Spin-Up Data and Filtering

Fitting

What’s up with the Mismatch ?

Measuring Torque + Trajectory Following

What’s Next

A Thesis Proposal!

Modular Code for Modular Hardware w/ Metaprogramming

Automated RPCs and Templated Proxy Classes

Automated RPCs in the Wild

Replacing Serial String Parsers with RPCs and Proxies

What’s in a Proxy

The ‘Metaprogramming’

Naming and Case Conventions

Writing a Boilerplate main.py

Future Efforts: Protocols, Standards

Some Discussion

MUDLink: a Modular UART Duplex Link

MUDLink on GitHub / Arduino

Packet Framing

Bit and Byte Errors

Flow Control and Feedback

Enter MUDLink, a Useful UART Link

Packet Framing with COBS

Error Catching with CRC16-CCITT

Flow Control and Delivery Guarantees

Statistics Tracking

API and Protocol

System Limits, Future Work, and What I Learned While I Did It

Writing a Boilerplate `main.py`