imdb-api releases and more
It’s been a few weeks since I last updated this blog. But also like, I haven’t been super busy.
imdb-api
I’ve shipped a few new versions of imdb-api. Mostly I’m dealing with some garbage of the downstream I’m calling changing their interface and requiring payment.
I’ve also added some very necessary features, like searching and timeouts. Not sure how I didn’t include these in the first round but you know, live and learn.
mypy-rt
I’ve started a project that takes python function type annotations and enforces them at runtime, by dynamically injecting bytecode into the compiled function.
This is largely useful if you maintain a library, so that you can enforce types at your API boundary without writing extra code beyond your type annotations.
tsserver
I added a tiny commit that fixes tsserver on OpenBSD
Angband on OpenBSD
This weekend, I was lucky enough to go to the Roguelike Celebration in SF! I’ve honestly only really played nethack, and only in passing; I preferred working on the codebase to playing it!
I got to see a talk from some of the Angband developers, which definitely piqued my interest in the game. At the conference, I popped open my new OpenBSD laptop to install it and play.
The version in ports was a bit old (3.3.2), so I set out to update it to 4.0.5. Building the game had changed a bit, and introduced some issues with detecting ncurses on OpenBSD. After some poking, I found out it was trying to use ncurses5-config to grab the cflags and ldflags of ncurses - which isn’t present on OpenBSD! I quickly added an additional check using AC_CHECK_LIB, in the case we failed to find find ncurses5-config.
Once that was done, I found another problem: angband -mgcu would hang on startup. I found out that mvwaddnwstr was guarded by _XOPEN_SOURCE_EXTENDED. By not defining that macro, I’d get implicit decl warnings from the compiler, and it’d only be runnable because that symbol would be found in the linked libncurses.
So why does this work when I define _XOPEN_SOURCE_EXTENDED? mvwaddnwstr (and many other functions) are implemented both as function-like macros and as functions. Normally the function implementations aren’t used - you have to define NCURSES_NOMACROS. It looks like (I’ve seen one other report of this as well) that the mvwaddnwstr function has a bug that can sometimes cause it to loop infinitely. The macro version works just fine, so once I unguard it by defining the _XOPEN_SOURCE_EXTENDED macro, everything works fine.
Submitted upstream and to ports.
https://github.com/angband/angband/pull/463
https://marc.info/?l=openbsd-ports&m=147434362321356&w=2
imdb-api and wsh
This weekend, I wanted to spend some serious time working on projects. While I wanted to focus on privdrop support for pkgsrc, I got sidetracked by a few smaller OSS projects that I created.
wsh
wsh is a tool that I wrote in college that manages performing commands on a large number of machines relatively quickly. It ended up getting used at LnkedIn, which is pretty cool.
Big features I worked on:
- Passing ssh options on the command line
- Figure out how many threads to run automatically
- Killing idle threads
I also did some PR management
imdb-api
node-imdb-api is one of my smallest, oldest projects, but is surprisingly used. I don’t like admitting that I had a javascript phase, but uh, it happens. :) Luckily, for my sanity I wrote it in typescript.
I mostly did a rewrite this weekend, which keeps most of the same API, but uses an entirely different backend, and adds a lot more type safety. I won’t link all the commits here, but I will link the big ones.
youshouldnt.date
This weekend I also launched my bathroom reading site, youshouldnt.date. It simply aggregates relationship stories from reddit, so I can peruse completed stories rather than having to wade through a bunch of relationship questions. It’s very rough, and needs styling / more features. I’ll do a proper announcement once I’m happy with it. Code is all here.
{py}gradle
My company just released a Gradle plugin so that you can build Python projects with Gradle! imo this is mostly useful if you have many existing projects at your company that also build with Gradle, and you want to standardize on a single build system for all projects.
As a hackday project, I wanted to add an optional linter check using mypy. After writing all the code to make this work, which was pretty easy despite not know Java nor Groovy, I wrote an integration test that would actually run mypy when your build.gradle specified that you were using python 3.5 and when you opted into the check.
To my surprise, my tests failed immediately – but not because of my changes. Turns out the in the initial release, pygradle didn’t actually support the documented method for setting the python version didn’t actually work. In fact, the whole details block where you could specify your python version, python path or venv didn’t work.
I talked with one of the maintainers, Ethan Hall, and he and I worked to gether to come up with a solution. I was super out of my depth working on this, but I did learn quite a bit about Groovy and Gradle on the way.
https://github.com/linkedin/pygradle/pull/36
pkgsrc + uwsgi
Just like my last post, I get to start this one off by saying that it’s been a long time since I’ve done any work on FOSS :) I’ve been busy traveling and seeing a bunch of old friends - both good problems to have.
That said, I feel obligated to post a little bit about what I’ve been doing lately.
uwsgi
I’ve been using uwsgi quite a bit lately, and I’ve found a few minor issues.
https://github.com/unbit/uwsgi/pull/1229/commits/5b20f8cead01e06fbdf5a950c500f9de17efc441
I added support for statsd and socket metrics to be emitted over a unix socket.
https://github.com/unbit/uwsgi/pull/1336/commits/f6d96c88d2b0022548163feb914d8dbd1b6473b4https://github.com/unbit/uwsgi/pull/1331/commits/90d9749ae629ad8eeb935088edabb23e8237e173
After working with the codebase a little bit, I got a bit curious and ran clang’s static analyzer on it. I fixed two of the more major issues. Luckily, the maintainers started doing the same afterwards, and fixed more of the problems that had cropped up in the results.
pkgsrc / pkgin / libnbcompat
I started using pkgsrc and pkgin on my work laptop + desktop as a replacement for homebrew and installing ancient RHEL packages. I found some features I’d like to add, but before I was able to do that, I found two issues…
https://gnats.netbsd.org/cgi-bin/query-pr-single.pl?number=51424
libnbcompat did not properly guard the definition of their strdup replacement, so that if you had a project that tested for strdup presence, the libnbcompat declaration would shadow the system one. Worse, if strdup is a macro (like on Fedora amd64), then compilation blows up.
https://github.com/NetBSDfr/pkgin/pull/72/commits/c111a49f248d9f972c34bdc1e89f44c57c59a651
In pkgin, I found a small misstep that made compilation fail if strsep was defined as a macro. In this case, since there was no check for the function definition, HAVE_STRSEP would be undefined, causing libnbcompat’s version to get picked up. Like in the last case, if strsep was a macro, compilation would blow up.
Calibre Wizard
It’s been awhile since I’ve actually worked on any FOSS, which is a damn shame. Things at work have been in a bit of an upheaval for me as I’m about to switch teams, and on top of that, I recently moved to a new apartment in SF.
I finally subscribed to the excellent Model View Culture quarterly, and wanted to put the first issue of the year on my Kindle, so I started setting up Calibre on my linux desktop.
The Kindle integration in Calibre is a bit interesting - rather than treating the Kindle as a USB mass storage device, they opt for emailing content to the email address associated with your Kindle. To make this setup easy, they have a wizard to help you set this up the first time you start the program.
Well, when I was setting up Calibre, I made the mistake of entering the wrong port for my mail server. While waiting for the test request to time out, I noticed that the entire wizard UI was blocked on sending the email.
I started digging, and found that Calibre has a job framework for starting up new threads that don’t block the UI. Unfortunately, this is heavily tied to the main Calibre process, which isn’t running at the time of the wizard invocation. Welp.
I bitched about it to my new, infinitely-patient roommate, and while talking to her, remembered Qt signals/slots. The Qt framework, which Calibre makes heavy use of, has a convenient mechanism for inter-object communication: signals and slots.
Signals are more or less what they sound like (although aren’t the same as POSIX signals). They define messages that you can send to a receiving slot. Slots receive and act on signals. As long as a slot accepts the correct type of signal, you can hook them up any which way. They’re super convenient. My particular usecase was having the email testing thread update the GUI thread when the test had completed.
All-in-all, I came up with this super simple patch that I’ve submitted for review: https://github.com/worr/calibre/commit/24bf2685b527170fae1de04ca677a42ff224baf1
However, the maintainer scrapped this, and re-worked that dialog entirely obviating my patch.
Upcoming Unicode blog?
I’ve been feeling pretty burnt out on tech projects atm, which is why my project blog has been, well, quiet. I’ve been taking it easy and trying to enjoy the free time I have around trying to get my shit together to move and staying on top of my job.
That said, I have recently started putting together code samples and such for a new blog post about one of my new favorite topics - Unicode! I’m not quite ready to show off the code samples/testcases just yet, but I am at least getting that together.
To kind of sum up my goal with this blog, I’m going to putting together a blogpost about dealing with Unicode in C, python2/3, rustlang and golang. I’m going to be writing the bulk of the code in C, using that to write about a lot of the gotchas when dealing with Unicode, and then use the other languages as examples of how things have changed, as well as an indication as to where the Unicode-handling landscape is now. I’d like to touch on problems like dealing with string length, character classes, character composition, and normalization? I’m worried it might get a bit long, so I might split it into two parts.
If you have suggestions, questions or requests for anything specific to talk about, let me know via whichever way is easiest for you to get in touch with me.
bsdinstall(8) troubles
rust xattr
I made a blog post a few days ago on my main blog about extended attributes. This reminded me that my rust patch is still in limbo, so I pinged the author, and he was nice enough to take a look and provide fantastic feedback.
https://github.com/Stebalien/xattr/pull/3/
bsdinstall
After working on fixing up my patches, I was in the mood to try and figure out why loading echo.ko was causing kernel panics. I started setting up a new FreeBSD instance in bhyve(8) to figure this one out.
Installing FreeBSD proved to be a problem.
This is the first time that I’m using the new bsdinstall over sysinstall, and I had a few problems actually make it to a successfully installed FreeBSD system.
The major issue that I encountered is that some components would either drop me back to the beginning of the installer, or drop me back to the beginning of that component. For example, partedit errors would drop me back to partedit, and zfsboot errors would restart the installation.
What made going through the installation over and over again extra frustrating, was that netconfig_ipv4 would error if you tried to configure an interface with a running dhclient. I submitted a report and simple patch to kill any dhclients if you’re configuring your installation interface to use dhcp.
Another bug that I found while proceeding through the install was that modifying a ZFS partition and just setting the mountpoint would cause partedit to segfault. Bug report and patch.
While I was looking through all this, I happened to find a potential issue that would hang the installer. When writing the bootcode to disk, partedit reads the file in a loop without checking for read(2) errors. If read(2) fails once for whatever reason, this will hang this installer, as that loop will never terminate. There was also an unchecked malloc(3) that took the result of an unchecked lseek(2) as well. I replaced all of that with an mmap(2)..
CFEngine on bad variable expansions - part 3
This title is misleading, since I did more than this. But whatever, it’s part of a series, I guess.
First, the things that have nothing to do with bad variable exapnsions, in increasing relevancy.
twitter-trollers
I had a shower thought, and decided to create a quicky Twitter bot that would watch a stream for polls, and vote on random results. I do this manually for fun anyway, so I thought it’d be trivial to automate.
Part way through the project, I discovered that Twitter still hasn’t exposed a poll API. I’m not wholly surprised, since Twitter has only become more developer-hostile ever since the whole API key revocation scandal years ago.
Code is here.
cookiecutter-pypackage
I discovered a cute little project that can generate language-agnostic project templates. I hate doing scaffolding, so I latched onto this immediately. I forked the author’s pypackage template, and stripped out half of the shit I didn’t care about, and amended it for my particular python requirements (mypy, only 3.5+).
I’m thinking of putting together a basic C template for some of my simple C projects as well.
My python template is here.
CFEngine PR update
I made fixups to my silent-returnszero patch.
CFEngine testall –verbose
I finally discovered the testall script in CFEngine’s tests/acceptance folder. I’ll need to amend HACKING.md to include some more detailed information about testall, since it’s easy to miss. It’s also a pain to develop tests without it.
PR is here.
And finally…
CFEngine now fails promises when variables don’t fully expand
It looks like I’ve almost finished this patch. To be honest, I’m not happy with the implementation just yet. I decided to tackle part of this problem by amending the Rval struct to include a field indicating whether or not that Rval has been fully expanded. That doesn’t seem like the best place to put that information, but as an experiment it was the easiest.
After doing that, I was able to run almost the entire test suite without error. This is huge progress, especially since this safeguard will prevent all kinds of CFEngine policy errors.
Commit is here.
Lastly…perl 6
Perl 6 finally came out. Somehow my name got on the list of contributors even though the code I wrote was almost certainly excised a few VMs ago. Neat though.
CFEngine on bad variable expansions - part 2
Today, I started taking a stab at actually implementing the logic to start bailing on bad variable expansions (this is not the finished commit message).
I started right off the bat, by taking all of the extant calls to ExpandScalar and starting to guard their return values, since literally no invocation of ExpandScalar checks the success of variable expansion. However, starting off by guarding these expansions didn’t actually work - largely because even on unexpanded variables, ExpandScalar was returning success.
It shouldn’t surprise anyone that a function whose return value was never checked returned bogus results in some cases.
After fixing that bug (and running all the tests), the guards suddenly work, and I got this delightful message from my test-case:
error: Could not expand variable '$(bar)'.
My super simple test-case:
bundle agent main {
files:
"/foo"
copy_from => cp("$(bar)");
}
body copy_from cp(source) {
source => "$(source)";
}
This was already a massive success. I continued guarding, running tests, and I got the commit linked above, with no test failures. I haven’t finished just yet, but it’s a great start.
One problem I will have soon though, will be guarding variable expansions in rvalues in variable assignments. Currently, ExpandPrivateRval doesn’t seem to have any mechanism for reporting error or referencing the calling promise, so it makes it difficult to report error on bad vars there.
Also this weekend, I made a quick PR for the work I’ve done on replacing select(2) with poll(2) in CFEngine. I added a poll(2) implementation for Windows from LibreSSL, and have been replacing select(2) calls where I can.