I seem to be doing a bunch of rewrites of my older projects nowadays. But sometimes that’s necessary. It is for the case of duplex (and libduplex).
When I was working at DigitalOcean over the past summer, my main project was
a communication framework called duplex.
Duplex’s aim was to create a simple messaging framework that could potentially
be used for RPC. The first version was prototyped in Go, and I created
libdpx, which was the first version in C.
libdpx had some major flaws - I didn’t use a proper build
system, I used
libtask for coroutines (switching that out with
then switching back to
lthread didn’t work with shared
libraries), and in general I didn’t manage memory well. You could lose data
along the way with the race conditions.
Jeff realised that a lot of the things we wanted to implement with duplex would be hard, because essentially we were creating a new protocol. So recently, he took to creating a second version in Go, but this time sitting on top of the SSH protocol.
The SSH protocol provides a bunch of the logic so that we don’t have to worry about it - authentication, security, and channel management. This was our way of wrapping it in a user friendly form.
libduplex is the second version of duplex in C. However, this time, I’m
being extra careful to, essentially, not fuck up the threading. This time
there are no lightweight coroutines or
libtask; instead, I directly call
pthreads in order to manage calls. The API is still threadsafe, and I still
run a thread dedicated to each ssh session (because libssh does not support
multiple threads working on sessions, which is understandable given the
complexity of the task - but it does have a non-blocking mode, and I rely
on that). Currently you can bind to tcp and unix sockets, and if any other
reliable protocol becomes available, I will gladly implement that.
The intent behind this is to make the Go version and C version be fully
compatible. I will not
cgo these bindings this time, because that seems
like a poor idea if they’re both going to be the same - the pure Go version
would work better in Go. However, there is an interest in making a Python
version for libduplex, which I will get around to after
fully compatible with the already existing Go counterpart.
The last duplex suffered immensely from a lack of manpower - Jeff has his own projects and clients to focus on, and I have university which I have to concentrate on. However, I don’t intend to abandon this at all - if possible, I’d like this second version to become the main version of duplex.
We want to take duplex further, too. Not just for standard communication and RPC between applications, but further into plugin systems, APIs over the internet, and more.
So I definitely want to keep working on it despite university.
(You can make my life slightly easier by making pull requests.)
Where from here?
Going from here remains to be seen. My immediate priorities are getting libduplex off the ground so that it can be at least used in a beta state. Maybe even declare it 1.0 if it gets to be good enough without any major flaws.
But I’m heading along for the ride and I definitely want to see what happens.
libduplex is licensed under the MIT
HACKING for building instructions and more.