This is a bit of a rant, but at ${WORK} I am currently implementing a network protocol from a specification provided by another company. They designed a proprietary protocol for integrating to their device and implemented it. Our product is supposed to communicate with theirs to receive a real-time feed of data which we will then process and do stuff with, which requires us to implement the their protocol.

This is something I do a lot of, so perhaps I’m a bit of a connoisseur of network protocols and their documentation at this point, and this one has me ripping my hair out. Mainly I want to talk about one small change to their protocol that would have made it so much more ergonomic and less brittle to work with.

When sending data over a network connection, you need to provide some way for the receiver to figure out when one message ends and the next one starts. In UDP based protocols, this is often just reduced to “Each message is its own UDP packet”, but for TCP, the receiving software can’t see packet boundaries unless they are using raw sockets or libpcap. The receiver just sees a stream of bytes and doesn’t know how the sender packetized it, so the protocol must define an in-band method to provide that information. The two most commonly seen methods are a certain character sequence that marks the end or beginning of a packet, like two newlines in SIP, or a length field that prefixes every payload, as with RTP interleaved inside a RTSP stream (just to bring up two random examples).

Once the receiver is certain that they have received an entire packet of data, they need to extract the useful data out of it. If the data is simple enough it could just be sent as a bunch of fixed length fields in a preset order. More complex data could be formatted as something like JSON or XML, or ASN.1, or something like ProtocolBuffers which also provides a solution to the previous issue of message boundaries as well as decoding the message.

In this case, the protocol designers decided to roll their own, each packet starts with a “Packet Type” byte, and then the payload format depends on what type the packet is, and is a combination of fixed length fields (like a 5 byte timestamp, or a 4 byte flag field), as well as variable length integers and Pascal style strings. There are also arrays of further structures where the first byte tells how many entries are in the array, and then each array element itself starts with a ‘sub-block type’ which determines the (variable length) length of the sub-block.

Technically this works fine: My code can read the PacketType, then scan the message byte by byte and make sense of all the components. However one piece of missing information would have made this protocol so much easier and better: They should have included a Packet Length after the Packet Type. Yes, this isn’t strictly necessary as we can figure out the length of each message as we parse it by knowing the fields that make it up and the rules for reading lengths of individual fields as we go. But this one oversight has several problematic consequences.

• There’s no way to skip over a message and get the next one without implementing all the parsing code for it. In our case, only about half of the Packet Types and sub-block types have information relevant to us, but I have to implement parsing code for all of them, or one message we didn’t know how to figure out the length of would throw off our synchronization ad we’d have no idea of which future byte represents the Packet Type for the next Message.

• Any parsing error is catastrophic, if we make any error in figuring out the length of a packet, it throws off synchronization, and we can’t find the next Message in the stream. This has already been a problem for me as I’ve found at least one case so far where their implementation and their specification didn’t match– the Specification claimed a 16-bit GUID field in one of their messages that didn’t actually exist on the wire in all cases. Not good. If they had a length field, it would be trivial for us to detect a problem like this and resynchonize and parse future messages properly.

• There is no backwards or forwards compatibility between protocol versions. If they add any new Packet Types in the future, our working implementation will silently break the first time we receive one of these new Packet Types, as we will have no ability to skip it and move on as we’d want to do. We are tied to only working with the exact protocol version we implemented against. This is even worse in this case as their protocol also doesn’t provide a version field, so we won’t even know anything is wrong until our parsing goes off the rails.

In summary, if you are designing a protocol, please be kind to your consumers and give them explicit length fields rather than make them calculate them.

I first decided to learn Emacs in 2000. At the time I’d been a regular Linux user for about five years, and had gone through a Computer Science degree, working mainly through a terminal to a Unix system. I knew the very basics of Emacs and vi, but always used a simple text editor like pico on the terminal, and an X11 GUI IDE or text editor when editing code locally. I actually don’t recall what IDE I was using for coding on Linux at the time, which goes to show how little it mattered.

At the time I was mainly working on an Open Source project in Java, which ran on Windows and Linux and was related to MIDI and music. I mostly developed on Windows since that’s where most of my other MIDI software ran. I had the idea of migrating this aspect of my computer usage to Linux, hence the Open Source project. On Windows I mainly just used edit.com in a DOS Window.

A second motivation for writing this Java MIDI program, was to learn Java, I’d just graduated college and was finding some disconnect in the skills I saw in job postings and the skills I had at the time. Java was definitely in demand around the year 2000. Learning Emacs was another aspect of this. I’d hoped to develop software for Linux or Unix, and Emacs and vi were what the professional “Real Linux Developers (TM)” used– or at least that’s what I read online.

I figured to be more productive and not embarrass myself, I’d need to be a competent user of one of those two editors. I’d never liked vi, so Emacs it was. I remember starting out going through some tutorials, and trying to memorize some basic keystrokes, and just changing over for writing code, and for twenty years, I’ve never looked back.

My first real developer job was indeed working in a Linux environment, and I knew I’d made the right choice; everyone around me coded in either vi or Emacs, not a lesser editor to be found.

In my first few years using Emacs, I ran the X11 GUI version, and customized it a fair bit, even going as far as hacking together some elisp code to do some more advanced jumping between declarations and definitions in my C++ code that took into account the way we structured our code and branches. I used Emacs to write my code, to compile it and fix the errors, and to debug in a gdb buffer, never leaving the editor. My .emacs file grew longer and longer. I still only used a tiny fraction of Emacs’s functionality, but I had everything I need.

At some point I changed from running Emacs as an X11 application to running it as a terminal application under GNU Screen. This was useful as I started working some days from home. From home I could just ssh into my work PC, reattach the screen session, and continue exactly where I had left off– not just as far as Emacs was concerned, but any other applications or shells I had running in other windows.

During this time, my usage of emacs actually simplified. More and more I found myself editing in emacs, but then using another terminal under Screen to compile my code or run gdb, using grep on a command line instead of multi-file finds in Emacs. Over time I moved from an “Emacs as the IDE” idiom to a “The OS is the IDE, Emacs is the text editor” idiom, which is still how I work and suits me well. When I changed jobs a few years later, I didn’t even bring my .emacs with me, by now there were just a handful of non-default settings I really cared about.

Twenty years later I’m still only really comfortable in an editor with Emacs keybindings, but I feel guilty pretending to be an Emacs user. I use Emacs as a basic text editor, and on occasion, google how to do something more complex. I remember reading some advice on interviewing and hiring which was to make sure to figure out if someone really has “Ten years of experience in X”, or has “One year of experience in X, ten times.”. I’m definitely in the “One year experience X times” when it comes to using Emacs, where by now X=20.

Of course I knew how much more complex and powerful Emacs was compared to what I was using it for, but I was only really driven home a couple of weeks ago, when an ‘Emacs user survey’ was posted on a forum I read, which I decided to take. Some of the questions were pretty eye opening, “What package manager to you use with Emacs”, Uh,I do have one third party .el which I downloaded with wget and ‘require’d in my .emacs, does that count?

Its not that I regret being a twenty year novice of my text editor, The ‘OS is the IDE’ model I referred to earlier suits me, and while I’ve tried for a period of time bringing more of my functionality back into Emacs, I haven’t found all that much efficiency or synergy over just using emacs to edit text, and a separate terminal window for building and debugging. I will at times run my terminal in an ansi-term buffer, but that’s about as integrated as I get these days.

A few months ago I tried moving over to Visual Studio Code for a few weeks, it had some nice features, at least on paper, integrating with the docker environment which I had moved to for building code. It didn’t last though, I used it long enough so I could feel I’d given it a fair shake, but more and more I found myself just opening Emacs for ‘a quick edit’ on a file.

Interestingly between the time I conceived of this blog post and posted it, I read something similar on a blog I read written by a vim user. In his case, he at least feels like an ‘intermediate’ user of vim after all these years. I don’t think I’m quite that far along in the Emacs journey.

I wrote in a previous post about my experiences using GhostBSD on my primary desktop. For several months everything was going great; I especially liked the update station app, which pops up in my Xfce panel whenever there are new updates to install and gives me GUI showing what packages need to be upgraded or reinstalled. Then I can just click ‘install updates’ and everything automatically downloads and installs, prompting me for a reboot when it’s finished. You really can’t put off the reboot much, some things really do stop working until you do. I especially like how GhostBSD updates the core OS and packages in the same manner, there’s no separate tooling for upgrading the base OS separately from the packages. Update station is all you need. Another great feature is a little checkbox to “Create Boot Environment Backup” which really saved my butt recently.

It was September 28th and I was about to head to bed. Update Station had been trying to get my attention for a couple of days, but I’d been looking the other way because I was in the middle of some stuff and didn’t want to reboot. Finally, before going to bed, I told it to do its thing and turned off the monitor. The next morning when I turned my monitor on, I saw the upgraded were done and GhostBSD wanted to reboot. “Sure, go ahead” I told it. I’d applied updates at least a dozen times for GhostBSD in the few months since I’d been using it, and never had an issue; but that streak was about to change. Upon boot I was greeted with:

OpenRC 0.41.2 is starting up FreeBSD 12.2-STABLE (amd64)

Press I to enter interactive boot mode

* Caching service dependencies ...
* WARNING: abi has started, but is inactive
* Starting the System Clock Adjuster (UTC) ...

“Cool, new OS version, I didn’t even realize FreeBSD 12.2 was out yet” was my first reaction, and “Hmm, whats taking so long” was my second. I waited and waited and finally came to the conclusion that my system was hung. I tried a few different things, including spamming that “I” key mentioned during boot up, but couldn’t get anything different to happen. Finally, I booted into the boot manager and told it to boot from the backup boot environment labeled as being from September 28th, and was relieved to be greeted by my familiar desktop. I used backup for a few days to get my work done, and I opened a thread on the GhostBSD forums and got a response a few days later from a GhostBSD developer telling me I had “The VirtualBox Kernel Problem”. That was good news to me, since if the developers had a name for the bug I was experiencing, that meant they knew what it was and a fix should be forthcoming. When more updates were available in the update station, I downloaded them greedily,rebooted back to the normal non-backup boot environment, and promptly hung in the same spot. Back to the backup I was. More upgrades became available over the next month and I installed those as well, with no improvement. Eventually I pinged the thread only to be told that the problem was resolved and I just needed to upgrade… but I already had the latest upgrades– or so I thought.

Eventually I had some time off work and decided to finally get to the bottom of this. My plan was that since I was booted off of a backup boot environment, I would first need to mount the ‘initial’ boot environment somewhere and troll through and find the spot it was hanging in. I quickly found that the OS had already mounted the ‘initial’ environment under /mnt for me, which was convenient. I installed ‘beadm’ anyway so I could manipulate my boot environments and read up on what they actually are, and then my mistake hit me. I’d been thinking of my backup boot environment kind of like a backup kernel that you leave when you compile a new one in case your new one doesn’t work. Actually, however, its a bootable clone of a ZFS snapshot of the entire filesystem. So when I was booted in my backup and installing updates, I was installing them into this clone, not the ‘initial’ environment. So every time I tried to boot ‘initial’ I was still booting the same snapshot that had the issue all along, here I was installing updates into my backup-2020-09-28 boot environment and expecting them to affect my ‘initial’ environment. Duh. Further, my backup-2020-09-28 environment now had all the latest and greatest updates installed including the upgrade to FreeBSD 12.2, and was working fine. My issue was already resolved, I just hadn’t realized it yet.

My first thought was that I’d need to figure out some way to get ‘initial’ to boot so I could install the upgrades there, but after reasoning a bit about boot environments and ZFS snapshots, I decided there wasn’t anything special about ‘initial’ and renamed it to ‘broken’, then renamed my backup-2020-09-28 environment to ‘initial’ and made it active. After that, on reboot, everything worked fine again without having to choose a backup environment in the boot loader. ZFS (and the GhostBSD devs) save the say and we all live happily ever after.

I’ve written a lot on here about running FreeBSD on my laptop, but after moving to full-time work from home in March, I wanted a stand-alone desktop system to work from for ergonomics and to free up my laptop to be my portable system since my docking / undocking experience wasn’t great. In the beginning of May I made the trek into the office for a day to exchange some hardware, and one of the things I borrowed was a desktop for home office use. The box was a Dell XPS i7-4790 @3.60Ghz with 16G of RAM, so it was also about twice as powerful as my T450 (not that the T450 isn’t plenty zippy for my usage under FreeBSD)

At work I had been using it as a Windows system to test some software for an old project, but it had been sitting turned off in the corner for a few months before I stopped going into the office in March. First thing upon getting it set up that night in my home office was to install an OS that wouldn’t make me miserable.

My first idea was, of course, to put FreeBSD on it, but I wanted it up and running the same day, and my experience with getting FreeBSD running smoothly on my T450 was that it took a bit of time to get it installed and configured to my liking and everything running smoothly. I briefly installed Void Linux on it, as it was a distribution I had been reading about and was interested in trying it out, but I ran into issues immediately with my SmartCard reader, which I couldn’t quickly resolve, and moved on. I hope to go back to Void Linux one day and either resolve the issue, or gather enough information to file a bug report, but I was looking to have everything ready for work in the morning so I moved on.

I knew everything I needed to use worked under FreeBSD, but didn’t want to take the time to configure it, so I decided to try GhostBSD. GhostBSD’s website claims it as “A simple, elegant desktop BSD Operating System”. GhostBSD seems to consider itself as its own OS “based on FreeBSD”, but I tend to think of it as FreeBSD preconfigured for a good desktop experience. Even ‘uname’ seems to agree with me, and reports the OS as FreeBSD.

I chose the community driven Xfce Desktop environment instead of the standard Mate edition, since I tend to use Xfce/OpenBox everywhere these days. I burned the ISO to a USB drive, booted from it, and installation was smooth. In short order I was booted into the X GUI, and everything seemed to be configured and working out of the box including my nvidia graphics card and ZFS. I spent some time installing packages I knew I would need and copying over configurations and customizations from my laptop, and before I went to bed, had a quick set up that would serve as my ‘work from home’ computer for the forseeable future.

My GhostBSD desktop isn’t quite as customized as my laptop, I left a lot more of the default Xfce stuff the installer had set up for me in place, but almost everything works fine. I’ve had a couple of strange issues with it which I haven’t dug into enough to resolve, but they’re pretty minor:

• My Xfce theme oddly doesn’t seem to apply to my Panel. It works everywhere else, but the panel usually remains the default colors. A couple of times after a reboot it actually came up correct, but 90% of the times it doesn’t
• I’ve found an nvidia-settings command that lets me stop screen tearing, but it only works on one or the other monitor, if I run it for both, it doesn’t work
• USB Hotplug seems tempermental, If I remove a USB device, the USB subsystem sometimes gets angry and stops detecting devices. Even ‘usbconfig’ hangs when run.
• I did have one ‘hangs on boot up after applying updates’ problem which was quickly fixed by the developers in the next set of updates and which I’ll post a separate post about in the future. Thanks to ZFS and Boot Environments, I was able to keep using my system while waiting for a fix.

In summary, I recommend GhostBSD for anyone who wants a BSD Desktop system and wants it now, with a minimum of twiddling with software configuration and settings.

After twenty plus years using Linux, I finally took a deep dive into the world of BSD last year. Before that I had dipped my feet into FreeBSD– I had an install of FreeBSD in a VM for a while, but never used it much beyond logging in and looking around (Yup, this is like Linux, but different). Last year, when I repeated the experience, I actually took the time to learn how to get FreeBSD set up to be usable as my work environment and work through the issues I had, and started using it daily, first in a VM, but soon enough directly on the hardware.

My experience with FreeBSD wasn’t quite ‘love at first site’, there was a learning curve, and a ‘pulling my hair out trying to make this thing that I needed for $JOB to work’ stage, but there was an immediate draw that was both refreshing and surprising, being that I had never used a BSD before– nostalgia. Even if FreeBSD was a little different than what I was used to, it reminded me of what I had loved about Linux when I first started using it, and didn’t realize I’d lost. There were two components to this:

First, there is a certain fun and mystery in having this new playground to experiment and learn in. I remember twenty-five years ago, just sitting in front of a shell prompt, after installing Linux, trying to figure out what everything was and what it did, finding all sorts of interesting commands, services, functionality. At the time it wouldn’t have been all that surprising if I’d found the secret to eternal life, shoved in some cryptic corner of the file system; it all seemed that big and complicated. There was an entire world here, with its own history and mythology; and I was on a quest to explore it. The more I learned, the more that world shrank, and Linux became just an Operating System. but just the little twists that FreeBSD threw at me was enough to bring back those memories. For the first time in years I knew the relief and accomplishment upon finally getting X11 running. I remember spending weeks (on and off) banging my head against XFree86 configuration files in 1996 getting a proper resolution and color depth (No thank you 640x480). I’ve been using modern Debian now for so long, I just expect it to all work out of the box and boot me to a perfect desktop right out of the installer. And this is a good thing of course, I’m not arguing that the old way was better, but there is something to be said for doing it on your own.

Second, FreeBSD reminds me in a lot of ways of the Linux I used to use. Linux has changed a lot on 25 years, but it did so so slowly, that I almost didn’t realize that it had kind of morphed into a new thing, and I’d gone along for the ride. I tend to think of Linux as basically a Unix-clone, and it certainly started that way, but over the years it’s diverged from the original Unix feel, and bolted on new idioms and uses. And again, this isn’t a bad thing, this is progress, but I almost feel like Linux is to Unix what Mac OS X Is to Unix, or what Android is to Linux. Yes under the hood its the same, but the OS Environment you use and live in is something a bit different. FreeBSD is more like the Old Linux I remember from my college days. Simpler and purer in some ways, but more cryptic and impenetrable in others perhaps. In FreeBSD I don’t have to feel guilty every time I use ‘ifconfig’ instead of ‘ip’, or remember whether the configuration file I want to edit can be edited directly, or if systemd has claimed authority over it and I have to go through some higher level subsystem to get the changes I want to trickle down to where I want them.

I commonly see threads on r/FreeBSD and other places where a Linux user basically asks “Why should I switch to FreeBSD”, and while there are plenty of answers people like to give, for example see here and here. Also here.. What I would add to all the technical and ideological arguments is that, for the right kind of Linux user, FreeBSD is more fun, and more like the Linux we originally fell in love with.

And everything I wrote about FreeBSD above, seems to be even more true for NetBSD, which I’ve only just begun to explore. Unlike FreeBSD I haven’t yet been successful in setting it up to be usable for my use cases so I can run it and actually get work done, so I’ve been limited to just playing around in it. FreeBSD so far seems like a good middle-ground between the enhanced hardware and application support of Linux, and the simplicity, pureness,and awesomeness of BSD.