| tags:duplex c glider labs categories:programming

duplex in C, take two
a less screwed up version

I seem to be doing a bunch of rewrites of my older projects nowadays. But sometimes that’s necessary. It is for the case of duplex (and libduplex).

Background

When I was working at DigitalOcean over the past summer, my main project was a communication framework called duplex. Duplex’s aim was to create a simple messaging framework that could potentially be used for RPC. The first version was prototyped in Go, and I created libdpx, which was the first version in C.

Unfortunately, libdpx had some major flaws - I didn’t use a proper build system, I used libtask for coroutines (switching that out with lthread, then switching back to libtask after lthread didn’t work with shared libraries), and in general I didn’t manage memory well. You could lose data along the way with the race conditions.

Jeff realised that a lot of the things we wanted to implement with duplex would be hard, because essentially we were creating a new protocol. So recently, he took to creating a second version in Go, but this time sitting on top of the SSH protocol.

The SSH protocol provides a bunch of the logic so that we don’t have to worry about it - authentication, security, and channel management. This was our way of wrapping it in a user friendly form.

libduplex

libduplex is the second version of duplex in C. However, this time, I’m being extra careful to, essentially, not fuck up the threading. This time there are no lightweight coroutines or libtask; instead, I directly call pthreads in order to manage calls. The API is still threadsafe, and I still run a thread dedicated to each ssh session (because libssh does not support multiple threads working on sessions, which is understandable given the complexity of the task - but it does have a non-blocking mode, and I rely on that). Currently you can bind to tcp and unix sockets, and if any other reliable protocol becomes available, I will gladly implement that.

The intent behind this is to make the Go version and C version be fully compatible. I will not cgo these bindings this time, because that seems like a poor idea if they’re both going to be the same - the pure Go version would work better in Go. However, there is an interest in making a Python version for libduplex, which I will get around to after libduplex is fully compatible with the already existing Go counterpart.

Managing Time

The last duplex suffered immensely from a lack of manpower - Jeff has his own projects and clients to focus on, and I have university which I have to concentrate on. However, I don’t intend to abandon this at all - if possible, I’d like this second version to become the main version of duplex.

We want to take duplex further, too. Not just for standard communication and RPC between applications, but further into plugin systems, APIs over the internet, and more.

So I definitely want to keep working on it despite university.

(You can make my life slightly easier by making pull requests.)

Where from here?

Going from here remains to be seen. My immediate priorities are getting libduplex off the ground so that it can be at least used in a beta state. Maybe even declare it 1.0 if it gets to be good enough without any major flaws.

But I’m heading along for the ride and I definitely want to see what happens.

Github

libduplex is licensed under the MIT license. See HACKING for building instructions and more.