Ekam Eclipse Plugin

Ekam now has an Eclipse plugin! As always, all code is at:

http://code.google.com/p/ekam/

A story

So, let me start with a story. A little earlier today, I was making a small edit to Ekam. I made a change, saved the file, then saw an error marker (a red squiggly underline) appear on the line of code I had just written. I hovered over it and a tooltip popped up describing a compiler error. I figured out what the problem was, edited another file to fix it, and saved that. The error marker in the first file immediately went away.

It wasn't until several moments later than I realized that Eclipse's CDT (C++ developer tools) does not actually mark errors like this in real time. Normally you have to run a build first, and let Eclipse collect errors from the build output. It's normally only in Java that you get such fast feedback as you type.

What had actually happened here is that when I saved the file, Ekam immediately built it, then it sent the error info to my Ekam Eclipse plugin, which in turn marked the error in my editor. This was the first time I had actually edited Ekam code with the new plugin active. It literally took less than a minute for the plugin to start saving me time, and I didn't even notice until after the fact because the process seemed to natural.

As soon as I realized what had just happened, I started squealing with delight.

Features

Here's what the plugin does so far:

• Connects to a running Ekam process in continuous-build mode.
  
• Individual build actions are displayed in a tree view, corresponding to the source tree.
  
• Output from each action is shown and errors in GCC format can be clicked to browse to the location of the error in the source code.
  
• Errors are also actively marked using Eclipse "markers", meaning you get the squiggly red underline in your text editor.
  
• Passing tests have a big green icon while failures (whether tests or compiles) have a big red icon (and also cause parent directories to be marked, so you can find problems in a large tree).
  

UI is Weird

I have very little experience with UI code. One thing I'm finding interesting is that when I run my code, I immediately notice obvious UX problems that weren't at all obvious when I was imagining things in my head. Little tweaks can make a huge difference. I really have to try things to find out what works.

Time to Apply

Neither Ekam as a whole nor the Eclipse plugin in particular are anywhere near complete. However, I think they are working well enough now that it's time to work on something else using Ekam. I could go on adding features that I think are useful forever, but actually trying to use it will tell me exactly what the biggest pain points are. I will then go back and tweak Ekam as needed to facilitate whatever I'm working on.

So, next week, I plan to work on Captain Proto, which I've put off for too long.

Ekam: Works on Linux again; queryable over network

Ekam works on Linux again, and supports the ability to query the build state over the network. As always, the code is at:

http://code.google.com/p/ekam/

Real Linux support

Ekam not only works on Linux again, but I've gone and written a whole EventManager implementation based on epoll, signalfd, and inotify. I like epoll, but I am not sure if it really deserves to be advertised as "simpler" than kqueue. While it is a narrower interface, I don't feel like it took significantly less work to use.

inotify in particular is a really painful interface. This is what you use to watch the filesystem for changes. The painful part is that there is no dedicated system call for reading events from the inotify event stream. Instead, you just use read, and it happens to produce bytes that match a particular struct. This would be reasonable, except that the struct is actually variable-length (the last member is a NUL-terminated string). In addition to some annoying pointer arithmetic, this means that it is impossible to read just one event at a time, because you must provide a buffer big enough to hold any possible event, and it is likely that multiple actual events will fit into the buffer.

The problem with reading multiple events at a time is that it makes cancellation really complicated. Say you read two events, and then you try to handle them in order. You handle the first event by calling a callback. But what if that callback actually cancels the second event? That event is still in the buffer, and when you get to it you have to have some way of figuring out that it was canceled. If there were a way to read one event at a time, then this becomes the kernel's problem and userspace apps have a much easier time. The kernel has to solve this problem either way, so it would be nice if userspace could leverage its solution.

To make matters even worse, the ID numbers which the inotify interface uses distinguish directories being watched can be implicitly freed when particular events occur. Namely, if a watched directory is moved or deleted, then the ID number assigned to that directory is immediately available for reuse after that event is delivered. But say that you get two events, and the *second* event is the "directory deleted" event. Now say, in the course of handling the first event, you start watching a new directory. That new watch is likely to be assigned the same ID number as the one that was just freed. But you don't actually know yet that that watch has been freed, because you haven't looked at the second event. So, it looks like inotify just assigned you a duplicated watch ID, and all kinds of confusion ensues.

To work around all this, I ended up writing some very complicated code. It seems to work, but I do not completely trust it, much less am I happy with it. In contrast, the kqueue file watching implementation for FreeBSD was a breeze to write.

Network status updates

Ekam can now run in a mode where it listens for connections on a particular port, and then serves streaming information on the build state to anyone who connects. The protocol is (unsurprisingly) based on Protocol Buffers. I intend to use this to implement an Eclipse plugin.

If you would like to write this plugin, or a plugin for some other editor/IDE, let me know!

Kubuntu desktop

FreeBSD fails me

Hose my computer while trying to update my OS once, shame on me.

Hose my computer while trying to update my OS twice, shame on my OS.

Last night I tried to update my "ports" (basically all the non-core software) on my FreeBSD system. There are three different programs to do this: portupgrade, portmanager, and portmaster. AFAICT they all do the same thing, so I chose the one that was installed: portupgrade.

Unfortunately, the process isn't completely automatic. There is this file called UPDATING in the ports tree which contains a list of things that have changed which require manual intervention. It appears that several things get added to the list every week. Ugh. But after whittling it down, I concluded that the only thing I really needed to do was follow some instructions to remove KDE, because the KDE port had been updated to a new release and some things changed in incompatible ways.

Fine, whatever. I did as instructed. But then when I tried to run portupgrade, it started complaining about ports being in an inconsistent state because the KDE libs had been removed. It directed me to run some pkgdb, which started asking me obscure questions for which there didn't appear to be any right answer. After struggling with that for a bit, I canceled it and went back to portupgrade, this time passing a flag which it had suggested I use if I wanted to skip verification of port states. It started installing, so I let it stew for awhile, and when I came back, my FreeBSD install was thoroughly broken.

I don't have time for this crap. It was fun while it lasted, FreeBSD, but it's time to find something a little lower-maintenance.

Let's try Kubuntu

I hear Ubuntu is popular, and it even has a specialized KDE-based version called Kubuntu. I decided to give it a try instead. Let's catalog the problems I run into and see how they compare to installing FreeBSD.

Can only prepare USB installer from Windows or Linux

My spare machine is a MacBook. When installing FreeBSD, I was able to download an installer image explicitly designed for a USB stick. Kubuntu can be installed from USB, but the images provided on the web site are strictly CD images. You can convert these into USB images, but it seems the only tools available to do this run on Windows and Linux, not OSX or FreeBSD. After struggling a bit I rebooted my MacBook into Windows.

syslinux hangs

Once I had the USB stick prepared, I tried to boot from it. Failure. All it did was print the version of syslinux it was using and then hang. "syslinux" is apparently a utility for producing bootable USB sticks based on Linux.

I really had no idea what to do about this, and a Google search didn't produce any hints. On a lark I tried downloading syslinux on my Windows machine and then running it to re-initialize the USB stick. I noticed that the version the Kubuntu installer had chosen to use was 3.86, whereas 4.03 is now available. The package contained an executable called "syslinux.exe". Not knowing what else to do, I ran it on the command line. It informed me of a bunch of options, none of which made any particular sense to me.

I tried "syslinux g:" to initialize the G drive. Nothing appeared to happen. But just in case, I then tried booting from it and, miraculously, it worked! Wow. Did not expect.

Installer fails

Unfortunately, immediately after choosing what to do from the bootloader menu, I was dumped to a shell with the error message: Can not mount /dev/loop1 on cow

Some Googling revealed that this happens if, when running the Ubuntu tool to prepare the USB stick, you choose the option to allocate some space to save temporary files on the stick. This is the only option the tool gives you, and it is enabled by default. Apparently it doesn't work. You have to disable this option. WTF?

OK fine. Re-did the USB stick and tried again. Hung at syslinux. Re-fixed syslinux and tried again. Finally, it works!

Whoa

I...

Whoa.

Holy polished OS, Batman. I mean, really. Wow.

Let's go through the problems I had on FreeBSD, and compare.

Installer

Once I got the USB stick correctly initialized, I have to say that the Kubuntu installer was the best OS installer I have ever used. The thing ran at native 2560x1600 resolution. It gave me the option to choose my keyboard layout (Dvorak) at the very first menu. The partition manager was simple and intuitive, and even gave me the option of using btrfs (though I held off for now since it seems it is not yet ready for prime-time). And the installer let me configure the system while it copied the files -- although there was very little configuration to do. The whole process took about ten minutes. It was better than the OSX installer. This was completely the opposite of FreeBSD.

Installing Packages

There's a beautiful GUI for this, and it's of course backed by apt. All packages are available in binary form and are updated regularly.

Graphics Driver

The included Nouveau driver immediately had me running at 2560x1600. Of course, it was a bit sluggish, so I set out to install the nVidia driver.

Kubuntu includes a GUI app for this. It fetches the driver for you, builds, and installs it, all with a nice progress bar.

Holy. Crap.

Graphical login

This was already working on first boot.

Audio

Here I have a minor problem. Audio works, but goes to both my headphones and speakers. Plugging in the headphones does not mute the speakers. This is not a big deal for me since I can separately turn off my speakers, but I was a little bit surprised to find that Linux did not seem to provide me any way to fix this. I actually found a GUI app for twiddling all kinds of things related to the HD Audio driver, but couldn't find any knob that would make it mute the speakers when the headphones were plugged in. AFAICT from the internets, this is a common problem, and the solution is usually to wait for the driver to be updated for your hardware.

Oh well. At least it does actually play to both the speakers and headphones by default, unlike FreeBSD which did not do anything with the headphones until I started poking at the boot configs.

Chrome

Well, obviously, I just installed the Google-provided package, which will now happily auto-update Chrome for me. No mucking around with source code here.

Flash

This worked out-of-the-box. Contrary to my experience on my work Linux machine, I have not seen any performance problems.

Fonts

A few fonts looked a little weird at first, until I installed the msttcorefonts package. Now everything looks good.

Suspend/Resume

The "sleep" option in the Kubuntu menu seems to work fine. The only odd part is that I have to press power to wake it -- keyboard keys are ignored. But I guess that's not a big deal.

iTunes replacement

Amarok is available, obviously. But after using it for awhile on FreeBSD, I have decided it is pretty much crap. The UI is weird and just does not do the things I want. I'll be on the lookout for something new, but this isn't really an OS issue.

ZFS

Linux does not have ZFS due to stupid licensing squabbles. ZFS is open source. Linux is open source. But the technicalities of the licenses do not allow them to be used together. Lame.

But I guess Linux now has BTRFS, which is similar. The Kubuntu installer actually gave me the option to use it, unlike the FreeBSD installer where I had to set up ZFS manually on the console. However, from what I've read, BTRFS is not quite ready yet, because it lacks an fsck-like tool. This apparently means that a BTRFS partition can be left unrecoverable after a power failure. It looks like this will be resolved Real Soon Now, but I decided to play it safe until then.

Eclipse

Eclipse officially supports Linux, of course. But the packages available from apt were for version 3.5, whereas 3.6 has been out for a few months now. So, I decided to install the official release instead of the apt package.

Boot-up Splash Screen

Kubuntu provides a very nice, minimalist splash screen by default, and it even runs at native resolution (which the FreeBSD one certainly didn't).

Wine

The default apt repositories did not include Wine 1.3, but there's apparently an alternate repository listed on the Wine site that does. Installed from there. Piece of cake.

Starcraft 2 installed and patched with no problems, following basically the same procedure as I did on FreeBSD, except without the part where I had to upgrade the kernel, or the part where the patcher kept crashing. The latter is probably due to the newer Wine version.

Conclusion

Well, I'm pretty much blown away. What took me two or more weekends to set up on FreeBSD took only two weeknights on Kubuntu. Of course, it's no real surprise that Kubuntu would be more usable than FreeBSD, but I honestly didn't expect this much polish. I could actually see non-technical users using this. Canonical has done an amazing job.

Ekam continuously builds

I just pushed some changes to Ekam adding a mode in which it monitors your source code and immediately rebuilds when a file changes. As usual, code is at:

http://code.google.com/p/ekam/

Just run Ekam with -c to put it in continuous mode. When it finishes building, it will wait for changes and then rebuild. It also notices changes which happen mid-build and accounts for them, so you don't need to worry about confusing it as you edit.

I've only implemented this for FreeBSD and OSX so far, but things are still broken on Linux anyway (see previous post).

kqueue/EVFILT_VNODE vs. inotify

Speaking of Linux and monitoring directory trees, it turns out that Linux has a completely different interface for doing this vs. FreeBSD/OSX. The latter systems implement kqueue, which, via the EVFILT_VNODE filter, can monitor changes to a file pointed at by an open file descriptor. This includes directories -- adding or removing a file from a directory counts as modifying the directory.

Linux, on the other hand, has inotify, a completely different interface that, in the end, does the same thing.

There is a key difference, though: kqueue requires you to have an open file descriptor for every file and directory you wish to monitor. inotify takes a path instead. Normally I prefer interfaces based on file descriptors because they are more object-oriented. However, the number of file descriptors that may be opened at one time is typically limited to a few thousand -- easily less than the number of files in a large source tree. On OSX (which seems to have lower limits than FreeBSD) I was not even able to monitor the protobuf source tree without hitting the limit.

Furthermore, watching a directory using inotify also means watching all files in the directory. While I have to assume that this is NOT transitive (so you can't watch an entire tree with one call), it still (presumably) greatly reduces the overhead of watching an entire tree, since the OS doesn't have to flag every file individually.

I'm still waiting for confirmation from the freebsd-questions mailing list on whether EVFILT_VNODE is really so much less scalable, but lots of Googling didn't produce any answers. I just see lots of people asking for inotify on FreeBSD and being told to use EVFILT_VNODE instead, as if it were an equal replacement.

This may mean that FreeBSD and OSX will have to use some sort of gimpy polling strategy instead, but that will have its own scalability issues as the directory tree gets large. Sigh.

I may give up and switch my desktop to Linux. Fully-supported Chrome and Eclipse would be nice, as well as a more robust package update mechanism (I'm getting sick of compiling things). Figuring out how to do everything on FreeBSD was fun, but getting a bit old now.

Ekam finds includes, runs tests

Sorry for missing the last couple weeks. I have been working on Ekam, but didn't have time to get a whole lot done and didn't feel like I had anything worth talking about. But now, an update! As always, the code can be found at:

http://code.google.com/p/ekam/

Ekam now compiles and runs tests:

Here you can see it running some tests from protocol buffers. It automatically detects object files which register tests with Google Test and links them against gtest_main (if necessary). Then, it runs them, highlighting passing tests in green, and printing error logs of failing tests which show just the failures:

By the way, compile failures look pretty nice too:

Is that... word wrap? And highlighting the word "error" so you can see it easily? *gasp* Unthinkable!

Intercepting open()

Ekam now intercepts open() and other filesystem calls made by the compiler. It works by using LD_PRELOAD (DYLD_INSERT_LIBRARIES on Mac) to inject a custom shared library into the compiler process. This library defines its own implementation of open() which makes an RPC to the Ekam process in order to map file names to their actual physical locations. Essentially, it creates a virtual filesystem, but is lighter-weight than FUSE. This serves two purposes:

1. Ekam can detect what the dependencies of an action are, e.g. what headers are included by a C++ source file, based on what files the compiler tries to open. With this information it can construct the build graph and determine when actions need to be re-run. (See previous blog entries about how Ekam manages dependencies.)
   
2. Ekam can locate dependencies just-in-time, rather than telling the compiler where to look ahead of time. For example, Ekam does not need to set up the C++ compiler's include path to cover all the directories of everything the code might include. Instead, the include path contains only a single "magic" directory. When the compiler tries to open a file in that directory, Ekam searches for that file among all public headers it knows about across the source tree.
   

The down side of this approach is that every OS has quirks that must be worked around. FreeBSD seems to be the least quirky: the only thing I found weird about this system is that I had to define both open() and _open() to catch everything (and similarly for all other system calls I wanted to intercept). OSX and Linux, meanwhile, have some ridiculous hacks going on in which certain system calls are remapped to different names at compile time, so the injected library has to be sure to intercept the remapped name instead of the original. I still am running into some trouble on Linux where some calls to open() seem to bypass my code, while others hit it just fine. I need to spend more time working on it, but I do not have a Linux machine at home. (Maybe someone else would like to volunteer?)

Try it out

If you are running FreeBSD or OSX, you can try using Ekam to build protobufs or your own code (currently broken on Linux as described above). I've updated the quick start guide with instructions.

Ekam: Core model improvements

Finally got some Ekam work done again. As always, code can be found at:

http://code.google.com/p/ekam/

This weekend and last were spent slightly re-designing the basic model used to track dependencies between actions. I think it is simpler and more versatile now.

Ekam build model

I haven't really discussed Ekam's core logic before, only the features built on top of it. Let's do that now. Here are the basic objects that Ekam works with:

• Files: Obviously, there are some set of input (source) files, some intermediate files, and some output files. Actually, at the moment, there is no real support for "outputs" -- everything goes to the "intermediate files" directory (tmp) and you have to dig out the one you're interested in.
  
• Tags: Each file has a set of tags. Each tag is just a text string (or rather, a hash of a text string, for efficiency). Tags can mean anything, but usually each tag indicates something that the file provides which something else might depend on.
  
• Actions: An action takes some set of input files and produces some set of output files. The inputs may be source files, or they may be the outputs of other actions. An action is specific to a particular set of files -- e.g. each C++ source file has a separate "compile" action. An action may search for inputs by tag, and may add new tags to any file (inputs and outputs). Note that applying tags to inputs is useful for implementing actions which simply scan existing files to see what they provide.
  
• Rules: A rule describes how to construct a particular action given a particular input file with a particular tag. In fact, currently the class representing a rule in Ekam is called ActionFactory. Each rule defines some set of "trigger" tags in which it is interested, and whenever Ekam encounters one of those tags, the rule is asked to generate an action based on the file that defined the tag.
  

Example

There is one rule which defines how to compile C++ source files. This rule triggers on the tag "filetype:.cpp", so Ekam calls it whenever it sees a file name with the .cpp extension. The rule compiles the file to produce an object file, and adds a tag to the object file for every C++ symbol defined within it.

Meanwhile, another rule defines how to link object files into a binary. This rule triggers on the tag "c++symbol:main" to pick up object files which define a main() function. When it finds one, it checks that object file to see what external symbols it references, and then asks Ekam to find other object files with the corresponding tags. It does this recursively until it can't find any more objects, then attempts to link (even if some symbols weren't found).

If not all objects needed by the binary have been compiled yet, then this link will fail. That's fine, because Ekam remembers what tags the action asked for. If, later on, one of the missing tags shows up, Ekam will retry the link action that failed before, to see if the new tag makes a difference. Assuming the source code is complete, the link should eventually succeed. If not, once Ekam has nothing left to do, it will report to the user whatever errors remain.

Note that Ekam will retry an action any time any of the tags it asked for change. So, for example, say the binary calls malloc(). The link action may have searched for "c++symbol:malloc" and found nothing. But, the link may have succeeded despite this, because malloc() is defined by the C runtime. Later on, Ekam might find some other definition of malloc() elsewhere. When it does, it will re-run the link action from before to make sure it gets a chance to link in this new malloc() implementation instead.

Say Ekam then finds yet another malloc() implementation somewhere else. Ekam will then try to decide which malloc() is preferred by the link action. By default, the file which is closest to the action's trigger file will be used. So when linking foo/bar/main.o, Ekam will prefer foo/mem/myMalloc.cpp over bar/anotherMalloc.cpp -- the file name with the longest common prefix is preferred. Ties are broken by alphabetical ordering. In the future, it will be possible for a package to explicitly specify preferences when the default is not good enough.

The work I did over the last two weekends made Ekam able to handle and choose between multiple instances of the same tag.

Up next

• I still have the code laying around to intercept open() calls from arbitrary processes. I intend to use this to intercept the compiler's attempts to search for included headers, and translate those into Ekam tag lookups. Once Ekam responds with a file name, the intercepted open() will open that file instead. Thus the compile action will not need to know ahead of time what the dependencies are, in order to construct an include path.
  
• I would like to implement the mechanism by which preferences are specified sometime soon. I think the mechanism should also provide a way to define visibility of files and/or tags defined within a package, in order to prevent others from depending on your internal implementation details.
  
• I need to make Ekam into a daemon that runs continuously in the background, detecting when source files change and immediately rebuilding. This will allow Ekam to perform incremental builds, which currently it does not do. Longer term, Ekam should persist its state somehow, but I think simply running as a daemon should be good enough for most users. Rebuilding from scratch once a day or so is not so bad, right?
  
• Rules, rules, rules! Implement fully-featured C++ rules (supporting libraries and such), Java rules, Protobufs, etc.
  
• Documentation. Including user documentation, implementation documentation, and code comments.
  

After the above, I think Ekam will be useful enough to start actually using.

No coding this weekend

I spent this weekend on things far more important than neat projects. I expect to get back to work next weekend.

Last Monday, though, I did manage to write up some proof-of-concept code to intercept open() system calls via LD_PRELOAD, as described in last week's post. It works pretty well. I'm excited to actually apply this technique next weekend.

Ekam: No longer restricted to building self, and can be taught new rules

Two big updates to Ekam this weekend! As always, the source code is at:

http://code.google.com/p/ekam/

Updates

Arbitrary Code

Ekam is no longer restricted to only building code in the "ekam" namespace. It will now build pretty much any C++ code you throw at it, so long as the code depends only on the standard C/C++ library. In fact, when I pointed Ekam at the Protocol Buffers source code, it successfully churned out protoc! (It didn't do as well compiling the tests, though.)

I ended up accomplishing this not by indexing libraries, as I had planned, but instead by changing the behavior when a dependency is not found. Now, if the link action cannot find any object files defining a particular symbol, it just goes ahead and tries to link anyway to see what happens. Only after failing does it decide to wait for the symbols to become available -- and it retries again every time a symbol for which it was waiting shows up. So, once all non-libc symbols are available, the link succeeds.

Now, this has the down side that Ekam will go through a lot of failed linker invocations. But, I look at this as an optimization problem, not a correctness problem. We can use heuristics to make Ekam avoid too many redundant link attempts. For example, we can remember what happened last time. Or, an easier approach would be to just make sure failed operations are not tried again until everything else is done.

Related to this change, Ekam will now re-run even a successful operation if dependencies that were not originally available become available. I talked about why you'd want to do this last week.

Defining new rules

You can now implement new build rules by simply writing a shell script and putting it in the source tree, as I proposed last week. You can now see what the actual code for compiling C++ looks like (linking is not yet implemented as a script). Of course, you don't have to write these as shell scripts. Any executable whose name ends in .ekam-rule will be picked up by Ekam -- even binaries that were themselves compiled by Ekam.

Refactoring

I seem to spend a lot of time refactoring. Working on C++ outside of Google is almost like learning a whole new language. I get to use whatever style I want, including whatever C++ features I want. I've been refining a very specific style, and I keep realizing ways to improve it that require going back and rewriting a bunch of stuff. Some key aspects of the style I'm using:

• Exceptions are allowed, but expected never to occur in normal usage.
  
• Ownership of pointers, and passing of that ownership, is explicit. I have a template class called OwnedPtr for this. In fact, I never use new -- instead, I call OwnedPtr::allocate() (passing it the desired constructor parameters). It's not possible to release ownership of the pointer, except by moving it to another OwnedPtr. Thus, the only way for memory to become unreachable without being deleted is by creating an ownership cycle. Yet, I don't use reference counting, since it is notoriously slow in the presence of multiple cores.
  
• I'm using single-threaded event-driven I/O, since Ekam itself is just a dispatcher and does not need to utilize multiple cores. Events need to be cancelable. Originally, I had every asynchronous method optionally return a canceler object which could be called to cancel the operation. This got surprisingly hairy to implement, since the event effectively had to cancel the canceler when complete. Also, events owned their callbacks, which made a lot of things awkward and tended to lead to cyclic ownership (doh). What turned out to be much simpler was to simply have every asynchronous method return an object representing the ongoing operation. To cancel the operation, delete the object. With this approach, deleting a high-level object naturally causes everything it is doing to be canceled via cascading destructors, with no need to really keep track of cancellation.
  

Ideas

Currently, the compile action does not yet resolve header dependencies -- if you need to use special include directories you must specify CXXFLAGS manually. This is, of course, bad. But how can Ekam detect header dependencies?

One approach would be to attempt to compile and, if not successful, try to parse the error messages to determine missing includes. This is, of course, going to be pretty brittle -- it would need to understand every compiler's output, possibly for every locale.

Another approach would be to write some sort of an "include scanner" which looks for #include directives. It would not necessarily have to evaluate branches (#ifdefs and such), since it could just conservatively look for all the headers that are mentioned and take whatever it can find. However, this would still be pretty complicated to write and slow to run, and it wouldn't be able to handle things like macro-derived header names (yes, you can do that).

So here's my idea: Run the compiler, but use LD_PRELOAD to inject into it a custom implementation of open(). This implementation will basically send an RPC to Ekam (using the already-implement plugin interface) asking where to find the file, then replace the path with what Ekam sends back. The injected open() could pay attention only to paths in a certain directory which would be added to the compiler's include path. Problem solved! And better yet, this solution naturally extends to other kinds of actions and programming languages.

Ekam: Works on Linux and OSX, and looks pretty

Above is what Ekam looks like when it is compiling some code. The fuchsia lines are currently running, while blue lines are completed. Errors, if there were any, would be red, and passing tests would be green (although Ekam does not yet run tests).

This weekend I:

• Implemented the fancy output above.
  
• Got Ekam working on Linux and OSX.
  
• Did a ton of refactoring to make components more reusable. This means that visible progress ought to come much quicker when I next work on it.
  

Some critical features are still missing:

• Ekam starts fresh every time it runs; it does not detect what tasks can be skipped.
  
• Ekam can only really be used to build itself, as it assumes that any symbol that doesn't contain the word "ekam" (i.e. is not in the "ekam" namespace) is satisfied by libc. To fix this I will need to make Ekam actually index libc to determine what symbols can be ignored.
  
• The only way to add new rule types (e.g. to support a new language) is to edit the Ekam source code.
  
• Ekam does not run tests.
  

As always, source code is at: http://ekam.googlecode.com/

Defining new rules

I've been thinking about this, and I think the interface Ekam will provide for defining new rules will be at the process level. So, you must write a program that implements your rule. You can write your rule implementation in any language, but the easiest approach will probably be to do it as a shell script.

For example, a simplistic version of the rule to compile a C++ source file might look like:

#! /bin/sh

set -e

if $# == 0; then
  # Ekam is querying the script.
  # Tell it that we like .cpp files.
  echo triggerOnFilePattern '*.cpp'
  exit 0
fi

INPUT=$1

BASENAME=`basename $INPUT .cpp`

# Ask Ekam where to put the output file.
echo newOutput ${BASENAME}.o
read OUTPUT

# Compile, making sure all logs go to stderr.
c++ $CXXFLAGS -c $INPUT -o $OUTPUT >&2

# Tell Ekam about the symbols provided by the output file.
nm $OUTPUT | grep '[^ ]*  *[ABCDGRSTV] ' |
  sed -e 's/^[^ ]*  *. \(.*\)$/provide c++symbol:\1/g'

# Tell Ekam we succeeded.
echo success

You would then give this script a name like c++compile.ekamrule. When Ekam sees .ekamrule files during its normal exploration of the source tree, it would automatically query them and then start using them.

Of course, shell scripts are not the most efficient beasts. You might later decide that this script really needs to be replaced by something written in a lower-level language. Of course, Ekam doesn't care what language the program is written in -- you could easily use Python instead, or even C++. (Of course, to write the rule for compiling C++ in C++ would necessitate some bootstrapping.)

Resolving conflicts

Several people have asked what happens in cases where, say, two different libraries define the same symbol (e.g. various malloc() implementations). I've been thinking about this, and I think I'm settling on a solution involving preferences. Essentially, somewhere you would declare "This package prefers tcmalloc over libc.". The declaration would probably go in a file called ekamhints and would apply to the directory in which in resides and all its subdirectories. With this hint in place, if tcmalloc is available (either installed, or encountered in the source tree), Ekam will notice that both it and libc provide the symbol "malloc" which, of course, lots of other things depend on. It will then consult the preference declaration, see that tcmalloc is preferred, and use that.

When no preferences are declared, Ekam could infer preferences based on the distance between the dependency and the dependent. For example, a symbol declared in the same package is likely to be preferred over one declared elsewhere, and Ekam can use that preference automatically. Also, symbols defined in the source code tree are likely preferred over those defined in installed libraries. Additionally, preferences could be configured by the user, e.g. to force code to use tcmalloc even if the developer provided no hint to do so.

Note that this preferences system allows for an interesting alternative to the feature tests traditionally done by configure scripts: Write multiple implementations of your functionality using different features, placing each in a different file. Declare preferences stating which implementation to prefer. Then, let Ekam try to build them all. The ones that fail to compile will be discarded, and the most-prefered implementation from those remaining will be used. Obviously, this approach won't work everywhere, but it does cover a fairly wide variety of use cases.

Build ordering

There is still a tricky issue here, though: Ekam doesn't actually know what code will provide what symbols until it compiles said code. So, for example, say that you plop tcmalloc into your code tree in the hopes that Ekam will automagically link it in to your code. You run Ekam, and it happens to compile your code first, before ever looking into the tcmalloc directory. When it links your binaries, it doesn't know about tcmalloc's "malloc" implementation, so it just uses libc's. Only later on does it discover tcmalloc.

This is a hard problem. The easy solution would be to provide a way to tell Ekam "Please compile the tcmalloc package first." or "The symbol 'malloc' will be provided by the tcmalloc directory; do not link any binaries depending on it until that directory is ready.". But, these feel too much like writing an old-fashion build file to me. I'd like things to be more automatic.

Another option, then, is to simply let Ekam do the wrong thing initially -- let it link those binaries without tcmalloc -- and then have it re-do those steps later when it realizes it has better options. This may sound like a lot of redundant work, but is it? It's only the link steps that would have to be repeated, and only the ones which happened to occur before tcmalloc became available. But perhaps one of the binaries is a code generator: will we have to regenerate and re-compile all of its output? Regenerate, yes, but not recompile -- assuming the output is identical to before, Ekam can figure out not to repeat any actions that depend on it. Finally, and most importantly, note that Ekam can remember what happened the last time it built the code. Once it has seen tcmalloc once, it can remember that in the future it should avoid linking anything that uses malloc until tcmalloc has been built.

Given these heuristics, I think this approach could work out quite nicely, and requires minimal user intervention.

FreeBSD LAN Party, and some Ekam work

My weekend was mostly consumed by a LAN party. And the most interesting part: I used FreeBSD exclusively for the whole party.

Games that ran natively:

• Quake (darkplaces port)
  

Games that completely worked under Wine:

• Duke Nukem 3D (xDuke port)
  
• Alien Swarm
  
• StarCraft 2
  
• Unreal Tournament 2004*
  
• Unreal Tournament 3*
  

Games that ran but had significant problems under Wine:

• Left 4 Dead 2: Disconnected after a random time period claiming that the server could not authenticate me with Steam. Bug 23286, still undiagnosed.
  
• Team Fortress 2*: Fonts in configuration windows did not render. However, in-game fonts rendered fine, so the game was playable after copying over my configs from a Windows installation. Bug 19522, also undiagnosed.
  

Games that did not run at all under Wine:

• none
  

I'm seriously considering running the game stations in my future house on FreeBSD (or Linux) so that I can customize them better (and avoid paying for Windows).

* Games that I installed and tested, but were not played at the party.

Ekam work

I did spend some time working on Ekam, but it was mostly refactoring. Nothing interesting to report. (I should have it working on Linux and OSX soon, though.)

Kake: A build system with no build files

UPDATE: Renamed to "Ekam" because "Kake" apparently looks like a misspelling of a vulger German word. Link below updated.

I finally got some real coding done this weekend.

http://code.google.com/p/ekam/

Kake is a build system which automatically figures out what to build and how to build it purely based on the source code. No separate "makefile" is needed.

Kake works by exploration. For example, when it encounters a file ending in ".cpp", it tries to compile the file. If there are missing includes, Kake continues to explore until it finds headers matching them. When Kake builds an object file and discovers that it contains a "main" symbol, it tries to link it as an executable, searching for other object files to satisfy all symbol references therein.

You might ask, "But wouldn't that be really slow for a big codebase?". Not necessarily. Results of previous exploration can be cached. These caches may be submitted to version control along with the code. Only the parts of the code which you modify must be explored again. Meanwhile, you don't need to waste any time messing around with makefiles. So, overall, time ought to be saved.

Current status

Currently, Kake is barely self-hosting: it knows how to compile C++ files, and it knows how to look for a "main" function and link a binary from it. There is a hack in the code right now to make it ignore any symbols that aren't in the "kake2" namespace since Kake does not yet know anything about libraries (not even the C/C++ runtime libraries).

That said, when Kake first built itself, it noticed that one of the source files was completely unused, and so did not bother to include it. Kake is already smarter than me.

Kake currently requires FreeBSD, because I used kqueue for events and libmd to calculate SHA-256 hashes. I thought libmd was standard but apparently not. That's easy enough to fix when I get a chance, after which Kake should run on OSX (which has kqueue), but will need some work to run on Linux or Windows (no kqueue).

There is no documentation, but you probably don't want to actually try using the existing code anyway. I'll post more when it is more usable.

Future plans

First and foremost, I need to finish the C++ support, including support for libraries. I also need to make Kake not rebuild everything every time it is run -- it should remember what it did last time. Kake should also automatically run any tests that it finds and report the results nicely.

Eventually I'd like Kake to run continuously in the background, watching as you make changes to your code, and automatically rebuilding stuff as needed. When you actually run the "kake" command, it will usually be able to give you an immediate report of all known problems, since it has already done the work. If you just saved a change to a widely-used header, you might have to wait.

Then I'd like to integrate Kake into Eclipse, so that C++ development can feel more like Java (which Eclipse builds continuously).

I'd like to support other languages (especially Java) in addition to C++. I hope to write a plugin system which makes it easy to extend Kake with rules for building other languages.

Kake should eventually support generating makefiles based on its exploration, so that you may ship those makefiles with your release packages for people who don't already have Kake.

Kake will, of course, support code generators, including complex cases where the code generator is itself built from sources in the same tree. Protocol Buffers are an excellent test case.

To scale to large codebases, I'd like to develop a system where many Kake users can share some central database which keeps track of build entities in submitted code, so that Kake need not actually explore the whole code base just to resolve dependencies for the part you are working on.

More FreeBSD tinkering, and some coding

So, I spent most of this weekend -- up until the beginning of Sunday -- tinkering with FreeBSD some more. I've added the boring details to my previous entry, in order to keep all related stuff in one place.

I did finally get a bit of time to work on a new project (the same project I was planning to work on last weekend, before realizing that I hate coding on Mac). However, it's still in the early stages, and so I'm not going to go into details yet. There should be interesting stuff to show next weekend.

FreeBSD Desktop

Introduction

This weekend I build myself a FreeBSD desktop. Notably, it runs Chrome (or rather, Chromium) flawlessly and supports Flash video (HD, full screen, full audio, no stuttering). Furthermore, it turns out KDE 4 is a beautiful desktop environment which I really enjoy using. I totally didn't expect this to turn out as well as it did.

Motivation

I originally had a coding project in mind for this weekend. However, when I sat down on Saturday to begin work, I was immediately confronted by an annoying fact: I hate coding on Mac. This is mostly for little reasons: the home and end keys uselessly scroll to the top or bottom of the document rather than move to the beginning or end of the line; ctrl+left/right does what home and end are supposed to do, rather than skipping forward or back by one word; pgup/pgdn move the view but not the cursor; GUI hotkeys all use "command" instead of "control" but terminal apps still revolve around "control"; etc. Some of these things can be partially mitigated by configuration, but not completely -- the command/control inconsistency is a stubborn one that isn't fixed by merely swapping the keys.

Of course, if I were a devoted Mac user I could get used to all this, but I do the vast majority of my coding at work, on Linux. Also, given Apple's anti-competitive behavior with the iPhone platform, I've begun to feel dirty using even their desktop products.

Meanwhile, playing with hardware last weekend was fun and I was itching to do it again. So, the natural thing to do was to go out and buy the pieces for a new machine.

Hardware

Requirements:

• Must run FreeBSD.
  
• Optimized for compiling code -- a task that is easily parallelized.
  
• Must drive two 30-inch dual-link DVI monitors.
  
• Avoid excessive power consumption.
  

Non-requirements:

• Not a gaming machine.
  
• Does not need top-of-the-line (read: exponentially expensive) hardware.
  
• Does not need much hard drive space.
  

I ended up settling on:

• Intel Core i7-930 processor: Four cores, hyperthreaded. Costs a quarter of the six-core i7 extreme.
  
• Intel DX58SO "smackover" motherboard: If it's Intel's processor and Intel's chipset anyway, might as well let them make the board too. More likely to work right. There does not appear to be any nForce chipset for the i7 currently.
  
• GeForce GTS 250: nVidia actually offers FreeBSD drivers based on the same codebase as their Windows drivers, including the whole OpenGL implementation. The 250 appeared to be the cheapest modern card that met the dual-DVI requirement.
  
• 3x2GB RAM: The motherboard has three RAM channels so you want to put three sticks in it.
  
• 128GB SSD: Yay for fast seeks and cool, quiet operation. Mass storage belongs on the network anyway.
  

Operating System

I chose FreeBSD for two reasons:

1. When writing Evlan in 2004-ish, I learned that the FreeBSD kernel offered many features that I wanted which Linux sorely lacked. Things like kqueue and AIO are key to implementing a framework based on event-loop concurrency, as Evlan was. I believe Linux has since gained equivalent features, but I was left with an impression that FreeBSD was more thoughtfully designed whereas Linux was more ad hoc.
   
2. I explicitly wanted a challenge. If I wanted something that "just worked", I'd be installing Windows. But I want a project where I learn something, and feel some sense of satisfaction in the end. FreeBSD is more difficult to set up, but in the process you learn how the system works, and once configured it can run essentially all the same software as Linux.
   

I installed FreeBSD 8.0 (UPDATE: 8.1 -- see end of post).

Details

Make no mistake: FreeBSD is not ready for the desktop. It works great once you get it set up, but it requires a lot of work to get there. As I said above, I explicitly wanted to tinker, because it is fun and educational. But anyone who wants a system that "just works" should look elsewhere.

Below are things I did, what worked, what didn't, and how I solved those problems. Hopefully someone will find these useful.

Installer

FreeBSD can be installed from a USB memory stick. They even provide a disk image for this exact purpose. Yay! I was able to write the disk image from within Mac OSX using the "dd" command as described in the FreeBSD release notes. The System Profiler utility reported that my USB drive was at /dev/disk1. However, before OSX would let me write, I had to unmount the flash drive -- NOT by "ejecting" it, which made it disappear entirely, but by using the "umount" command.

The FreeBSD installer is not very user-friendly. Being difficult to understand is bad, but being difficult to understand *and* not having any way to go back if you made a mistake is much worse. It turns out the standard installation mode has this property -- you usually cannot go back if you chose the wrong thing. However, the "custom install" mode -- which is labeled "for experts" -- actually presents you with a list of all the steps, from which you can choose which ones to perform. You should still go through them in order, but this allows you to repeat steps or go backwards when you make a mistake.

The installer initially could not detect my memory stick, even though it was installing from said stick. To fix this I had to choose "options" from the top-level menu, and then choose "rescan devices" (or something like that) from the options menu. This appeared to do nothing, but when I then went back and told the installer to install from USB again, it worked. (UPDATE: Fixed in FreeBSD 8.1.)

Installing packages

The memory stick installer only provides basic packages. Any interesting programs have to be installed separately, through the "packages" system (pre-compiled binaries downloaded from freebsd.org) or the "ports" system (scripts which automatically download original source code and compile/install it for you). I prefer packages whenever they are available, since they're faster. The "sysinstall" command can be run after installation in order to select additional packages to download, which is good for going through and getting the basics together. "pkg_add -r" can be used to install specific packages by name, on-demand (e.g. "pkg_add -r protobuf").

With this I installed X.org and kde4, among other things.

Graphics driver

The nVidia driver is not included with FreeBSD because it is not open source, and requires agreeing to a license. Whatever. There is a "port" for it, but it is out of date. I downloaded the most recent version from the nVidia web site and followed their installation instructions. This turned out to be surprisingly easy -- they actually provide a script which updates all the necessary config files for you! What a crazy idea.

For a long time, I could not convince Xorg to display at a reasonable resolution; it kept giving me something like 1280x800 (though I swear it looked more like 320x240; I must be spoiled). Turns out I needed this line in the "Screen" section of my xorg.conf:

Option "AllowDualLinkModes" "True"

WTF? Why on Earth would an option like this not be on by default? (This was actually before I installed the nVidia driver; I was still using the open source "nv" driver. Not sure if it makes a difference.)

Graphical login

To enable graphical login I had to edit /etc/ttys and change the "ttyv8" line to:

ttyv8   "/usr/local/kde4/bin/kdm -nodaemon"     xterm   on secure

This gave me a nice KDE-style login prompt on startup, but with a major problem: the keyboard layout was Qwerty, while I use Dvorak. Changing my keyboard layout in the KDE settings UI did not fix it, since those are user-specific and don't apply to the login prompt. Changing xorg.conf *also* did not fix it: under recent versions of FreeBSD, InputDevices in xorg.conf are ignored in favor of letting the USB daemons "automatically" figure it out. Of course, said daemons don't know what keyboard layout I use. The FreeBSD manual contained instructions on how to tell it (via /usr/local/etc/hal/fdi/policy/x11-input.fdi), but they didn't work -- the file seemed to have no effect. I banged my head against this for a long time. Nothing I did could even make Xorg print an error; it just ignored everything.

Eventually I figured out that kdm runs /usr/local/kde4/share/config/kdm/Xsetup before displaying the login dialog. So I inserted "setxkbmap dvorak" into it, which fixed the problem. It's a hack but gets the job done.

Audio

Enabling audio was as simple as loading the right sound driver. In my case:

kldload snd_hda

However, this created four separate /dev/dsps. It turned out /dev/dsp0 referred to the line-out jacks on the back of my machine, while /dev/dsp1 referred to the front headphone jack (the other two were digital outputs). Most apps only play to /dev/dsp0; I'd prefer for this to go to all jacks.

It turns that the snd_hda driver supports rather complex configuration. Annoyingly, it requires you to know details that are different for every chipset. Although the driver knows these details, there is no simple tool to query it; you must reboot and pass -v as a kernel boot flag, then examine the results with dmesg. For me, the relevant output was:

hdac0: Processing audio FG cad=2 nid=1...
hdac0: GPIO: 0xc0000002 NumGPIO=2 NumGPO=0 NumGPI=0 GPIWake=1 GPIUnsol=1
hdac0:  nid 17 0x01451140 as  4 seq  0     SPDIF-out  Jack jack  5 loc  1 color   Black misc 1
hdac0:  nid 18 0x411111f0 as 15 seq  0       Speaker  None jack  1 loc  1 color   Black misc 1
hdac0:  nid 20 0x01014410 as  1 seq  0      Line-out  Jack jack  1 loc  1 color   Green misc 4
hdac0:  nid 21 0x02214420 as  2 seq  0    Headphones  Jack jack  1 loc  2 color   Green misc 4
hdac0:  nid 22 0x01016011 as  1 seq  1      Line-out  Jack jack  1 loc  1 color  Orange misc 0
hdac0:  nid 23 0x411111f0 as 15 seq  0       Speaker  None jack  1 loc  1 color   Black misc 1
hdac0:  nid 24 0x02a19860 as  6 seq  0           Mic  Jack jack  1 loc  2 color    Pink misc 8
hdac0:  nid 25 0x01011012 as  1 seq  2      Line-out  Jack jack  1 loc  1 color   Black misc 0
hdac0:  nid 26 0x01813450 as  5 seq  0       Line-in  Jack jack  1 loc  1 color    Blue misc 4
hdac0:  nid 27 0x01a1985f as  5 seq 15           Mic  Jack jack  1 loc  1 color    Pink misc 8
hdac0:  nid 28 0x411111f0 as 15 seq  0       Speaker  None jack  1 loc  1 color   Black misc 1

This shows all the inputs and outputs the hardware has. The number after as indicates to which /dev/dsp the line will be connected (off by one). The seq indicates which speaker pair this is -- e.g. front, back, etc. Following the instructions, I added a line to /boot/device.hints:

hint.hdac.0.cad2.nid21.config="as=1 seq=15"

This says that to move nid 21 (the headphone jack) to as 1 (the first output device, which includes the back line-out jacks). Setting seq to 15 has a special meaning for headphones: when connected, all other speakers on the same as should be muted.

So, not only does my headphone jack work, but as an added bonus, plugging in headphones automatically mutes the other speakers. I completely didn't expect FreeBSD to support that. I wish it would have been set up that way by default, though. But I guess it's neat that I could actually set up each jack (three in the back, one in the front) to be an independent stereo output, with different programs playing to each one. I could use that for my house, to power the speakers that I'll have around the place.

Chrome

Following the instructions on the FreeBSD Chromium wiki page, I built Chromium. There were a few difficulties:

• Pulling the Git repo takes forever -- it's over 1GB! Next time, I'll try subversion.
  
• The "Sync source" step fails with an error, but this error appears to occur after everything has been synced. It looks like it tries to start some of the build process, but fails because you haven't applied the FreeBSD patch yet.
  
• The wiki fails to mention bison among the dependencies. pkg_add -r bison
  
• The wiki lists ALSA as a dependency, but this package doesn't exist in FreeBSD 8.0. I had to grab the ALSA packages from a FreeBSD 8.1 release candidate instead. I got a warning about dependency version mismatches when I tried to pkg_add them, but it seems to have worked fine.
  

With all that done, Chromium built and ran flawlessly. It's just like using Chrome on any other system. I was expecting some stuff to be semi-functional, but so far I've found nothing. It's fast, too.

Flash

I totally didn't expect to be able to use Flash at all on FreeBSD. On my Linux machine at work, Flash animations of any sort seem to blow away the CPU, and video in particular stutters horribly with no audio.

It turns out that FreeBSD has a port which sets up the Flash plugin to run in the Linux emulation layer. Amazingly, the plugin can run on the Linux ABI even when the rest of the browser is running natively.

The port installer ran into trouble when the Linux Flash installer downloaded from Adobe did not match the expected size and hash. It looks like they had released a new version since the port was written -- the file was not versioned. I had to modify the port to tell it the new hash so it would accept the file (I didn't want to find the old version because it may have had security problems). This did not seem to create any problems.

The port installer had several dependencies on Linux stuff which was pulled from Fedora Core 10. It seemed to have a lot of trouble finding a working Fedora mirror for this stuff. I had to download several packages manually after finding them on Google. (Fortunately all the hashes matched.)

Once installed properly, Chrome picked up the plugin automatically. Amazingly, video played flawlessly, with sound, at HD resolution, full screen.

Fonts

On initially installing Chrome, I noticed that some fonts (especially the debug console) looked bad or even unreadable (due to under-sampling). Also, most unicode characters were replaced with boxes -- these turn out to be common even in English text. The solution was to install two ports:

• webfonts: This pulls a bunch of Windows truetype fonts off some site that seems to be hosting them and has nothing to do with FreeBSD. Technically since these are part of Windows, you must have a Windows license to use them. I, of course, have zillions of Windows licenses. (UPDATE: A commenter notes that actually Microsoft allows gratis redistribution of these fonts on the condition that they be distributed only in the form of the original exe files. So you have to run a script that extracts the fonts from the exes. I was confused by the pkg-descr which mentioned that a Windows license was needed only for the Tahoma font.)
  
• code2000: A "shareware" unicode font. Installing this got rid of the boxes. Apparently I'm supposed to pay someone $5 if I continue to use it, though.
  

Suspend/Resume

After digging around for awhile, I figured out that the proper way to make FreeBSD go to sleep is with the zzz command. So far I've tested this once, and it failed to wake up -- it partially awoke (responded to pings), but then spontaneously rebooted. Will have to play with this more.

iTunes replacement

I was sad to learn that Songbird -- a popular open source iTunes clone -- had discontinued Linux support (wat?), and certainly did not support FreeBSD. Doh.

However, Amarok looks like another decent alternative. I'm currently in the process of importing my iTunes library into it.

Update: So far Amarok failed at importing my iTunes library. The importer appeared to be O(n²) somehow -- it started out fine, but scanning each file became progressively slower, and after running overnight I came back to find Amarok using 100% of one CPU and making no progress. Grr. I'll have to investigate this next weekend. (UPDATE: A newer version of Amarok fixed this problem.)

Amarok's name and icon keep making me think of Three Wolf Moon.

Conclusion

Back in 1999 I used to be a huge Linux desktop fan, but in 2000 I gave it up in favor of Windows 2000 (which, unlike previous versions, did not crash daily). I had become tired of all the work it took to configure and maintain a Linux machine. Ten years later, configuring FreeBSD is about as much work as Linux was at the time. But, I think these days I'm more interested in this hands-on approach, and the ability to customize every aspect of my OS is appealing.

Meanwhile, I'm delighted to find that KDE seems to have kept up with the UI innovations made in Windows and Mac OSX over the last ten years. KDE can even do Mac-style Expose -- perhaps the thing I miss most whenever I work on a non-Mac system.

In general I've been pleasantly surprised by what FreeBSD/KDE has to offer as a desktop system. I had originally intended to use this machine for coding projects only, but now it looks like I may as well make it my primary desktop. Now that everything's configured, I don't see anything significant that I'll miss from OSX, and using a fully open source system feels so much nicer.

Dénouement

So, as it turns out, a mere two days after I installed FreeBSD 8.0, they released version 8.1. Argh. So this weekend I had to update, leading to a new series of issues...

Hard drive has a new address

Apparently the new kernel version decided that that it really preferred to put my hard drive at /dev/ad10 instead of /dev/ad6 like 8.0 did. As a result, when I first tried to boot the new kernel, it couldn't find the boot volume. Amazingly, though, instead of just crashing, it brought me to a prompt where it told me what hard drives and options it knew about, and asked me what to do. I was able to figure out what happened and tell it where to boot from. It then dumped me into a single-user shell from which I was able to edit /etc/fstab to fix the boot config, then reboot happily.

portupgrade

The update instructions say that you need to re-compile all of your software after updating the system, as the new base libraries may not be binary-compatible with the old. Ugh. It suggested running portupgrade -af for this purpose. I decided instead to run portupgrade -afP to tell it to use pre-compiled packages where available. I also updated my ports tree before starting so that I'd get the latest versions of everything.

Alas, one of these two changes (or maybe both?) seems to have been a deadly mistake. portupgrade took forever, and quite a few packages failed to install because of new dependencies that weren't present. KDE was completely broken after this. I tried to fix it by running portupgrade -RP kde4 to make it re-try installing kde4 and all its new dependencies. This again took ridiculously long, and repeatedly complained that it couldn't install required dependency Perl 5.10 because it conflicted with Perl 5.8. After manually forcing an uninstall of Perl 5.8 and install of Perl 5.10, I got to the point where I could start up KDE, but other things seemed broken. Chrome, in particular, was now SIGBUSing regularly, and continued to do so even after re-compiling it from scratch.

Screw it, start over

I downloaded the FreeBSD 8.1 memstick image, backed up a couple config files that I didn't want to rewrite, and set off to re-install from scratch. Sigh.

ZFS

Since I had to start over, I decided to do something I had forgotten on my first time through: use ZFS. This is a new, radically different filesystem which is, IMO, the way filesystems should be -- read the Wikipedia entry for details. Linux does not support ZFS because the ZFS code is under a GPL-incompatible license. FreeBSD, however, has no problem using ZFS. Hah!

There is still no support for ZFS in the FreeBSD installer. Lots of guides on the internet suggest a variety of ways to get FreeBSD on ZFS. I ended up using this one. The basic idea is to install a minimal FreeBSD system on a small partition, then create the ZFS volumes and migrate all data to them. The small original partition must stick around as the boot kernel cannot yet be read from a ZFS volume. But, once the kernel is up, it can map in the ZFS drives, so everything else can live there.

Eclipse

After setting up the rest of the system as before, I wanted to install Eclipse to do some coding. There were a couple bumps in the road.

First, you really need to install the eclipse-devel package. Plain old eclipse is way out-of-date.

Second, the package depends on jdk-1.6, but I actually installed OpenJDK instead. Luckily jdk-1.6 cannot be installed automatically since you have to accept a Sun license agreement -- this means that installing Eclipse will not accidentally install a whole second JDK. I just had to tell the installer to ignore the dependency problem, and then set my JAVA_HOME to OpenJDK and everything worked fine. (UPDATE: This lead to some crashes, unfortunately. So I gave in and installed JDK 1.6, which worked better. The annoying part about this is that you must build JDK from sources (licensing restrictions), and since it is itself written in Java, the port actually installs yet another JDK called "diablo" first. So now I have three whole Java implementations installed. Urgh.)

Third, when I went to Eclipse's built-in plugin installer to install the plugins I need, it didn't have any download sites registered. I had to copy them over from the Eclipse install on my Mac.

Fourth, Eclipse refused to talk to any of these sites, giving a cryptic error message about an invalid argument (to what, it didn't say). Googling revealed that the problem was that Java tries to use IPv6 by default, and if that doesn't work, throws an exception. Brilliant. The fix is to pass -vmargs -Djava.net.preferIPv4Stack=true to Eclipse to force IPv4.

Finally, when I installed the C++ development plugin (CDT) through the updater, it appeared to install fine, but seemed to have no effect -- none of the C++ development features showed up in Eclipse. After lots of head-scratching, I figured out that the CDT has a couple of native-code parts which have to be compiled explicitly for FreeBSD. I didn't want to download the full CDT source code to compile, but I figured out that I could pull the necessary bits out of the eclipse-cdt FreeBSD package despite it being way out-of-date -- apparently these parts don't change much. Here's what I did:

mkdir temp
cd temp
fetch ftp://ftp.freebsd.org/pub/FreeBSD/ports/amd64/packages-8.1-release/Latest/eclipse-cdt.tbz
tar xf eclipse-cdt.tbz
cp -rv eclipse/plugins/*freebsd* ~/.eclipse/org.eclipse.platform_3.5.0_1216342082/plugins

Boot-up Splash Screen

It turns out that FreeBSD supports setting a boot-up splash screen, so you don't have to look at all that ugly text flying by. I followed these instructions to set it up -- he even provides a pretty nice image to use.

On the bright side

• The installer now detects the USB drive without a rescan.
  
• The newer version of Amarok successfully imported my iTunes collection.
  
• ALSA libs (needed by Chrome) are now part of the released distro.
  
• KDE looks a little prettier than before.
  
• OpenJDK 7 wasn't available in 8.0.
  

At this point, everything appears to be working. Hopefully now I can write some code!

Starcraft 2!

This is amazing. I can run Starcraft 2 on FreeBSD!

Kernel update

I had to upgrade to 8-STABLE. 8.1-RELEASE, even though it is only a couple weeks old, is too old: apparently a patch to fix Starcraft was just submitted recently. Note that the symptom of not doing this is that the game crashes when you try to log in.

Install Wine

I followed these instructions to install Wine. This is a little tricky: Only the 32-bit build of Wine currently works on FreeBSD, but of course I'm running the 64-bit kernel. Luckily, FreeBSD has a compatibility layer for running 32-bit binaries. Unfortunately, the ports system is not very well set up for this. I basically had to build/unpack a 32-bit FreeBSD base image into /compat/i386, chroot into it, and install Wine there. I also had to install the 32-bit version of the nvidia drivers, for the GL libraries -- note that it's important to use the exact same driver version.

winetricks

On the advice of this page I used winetricks to install several packages:

winetricks droid fontfix fontsmooth-rgb gdiplus gecko vcrun2008 vcrun2005 allfonts d3dx9 win7

I'm not sure all of these were actually necessary, though. Dan Kegel, winetricks author (and former Google employee whom I've met), replied to that post saying he didn't think all of those packages were really needed. I also had to install IE6 using winetricks as the Gecko-based HTML widget seemed to have troubles with the SC2 installer.

Get the installer

My machine does not have an optical disk drive, so I copied the files off the DVD from my Mac. This was a little tricky because by default my mac mounted the Mac partition of the DVD, hiding the Windows installer. I had to manually mount the correct partition:

umount /dev/disk1s2
mkdir mountpoint
mount_udf -o rdonly /dev/disk1s1 mountpoint

I could then copy all the files from mountpoint. I later couldn't get the disk to eject (even after mounting it normally) and had to reboot. Hrm.

TURN OFF COMPOSITING

If you are using a compositing window manager, disable it! For KDE, go to System Settings -> Desktop -> Desktop Effects and click the "Suspend Compositing" button. Otherwise 3D games are going to render slower than usual. Turn it back on when you're done playing. :)

Install

The installer ran fine. The patching process did not. It kept popping up something like "Microsoft C Runtime error". However, I noticed that even when this error popped up, the download progress would continue to go up for awhile, until finally getting stuck. If I then killed Starcraft II.exe and started it again, it would continue where it left off, crash again, but get further. After several iterations of this I had managed to download and install all the patches. Note: This problem may have been fixed by the kernel update, which I did not actually install until later. (Update: Nope, still happens.)

Sound

The internets claimed that if I built Wine with OpenAL support, sound would work. I did this, but the game was silent when I first ran. I think I fixed this by going into winecfg, under the "Audio" tab, and setting "Hardware Acceleration" to "Emulation". On the advice of another site, I also went under the "Libraries" tab and added an override for "mmdevapi", which I set to "disabled". I'm not sure if the latter fix was actually needed.

That's it!

I think that's everything I did. The game runs pretty smoothly at 2560x1600 with all graphics settings on "High" (although it defaulted everything to "Ultra", which was too much for my GeForce GTS 250).

So far I've played one round against the computer. Everything went fine until I accidentally activated KDE's "Desktop Grid" view which causes it to zoom out and display all windows from all desktops such that they don't overlap (like OSX's "Expose"). Amazingly, Starcraft kept on rendering through this, in its little window, only finally crashing when I tried to restore it. I will avoid activating that in the future. :)

New server for evlan.org

My Old Server

So, the server for evlan.org (and fateofio.org, theposse.org, etc.) has been hosted on a cheap-ass FreeBSD dedicated server in Montreal for over five years now.  I need my own machine because the web server is actually written in Evlan, my programming language.  Other than that, there's really no reason the server needs dedicated hosting -- it certainly doesn't get any significant amount of traffic.

Sometime in the last couple months, the machine stopped accepting SSH logins.  The web server was chugging along fine; I just couldn't log in.  I ignored the problem for awhile because I rarely need to log in to my server...  but it is a good idea to make a backup now and then.

Last week, though, my attention was drawn to the server when I noticed that some spammer had registered a few hundred new accounts for the sole purpose of creating spammy profiles for all of them.  WTF?  Why would a spammer take the time to write a script designed specifically to log into *my* server and create dummy profiles?  There are only two servers on the internet running this software.  You'd think it wouldn't be worth their time.  Especially given that it probably took them far longer to write the script than it took me to simply block all profile pages for user IDs over 500 -- so that profiles of original users are still visible, but all the spam users and any new users are gone.

Idiotic Support

So finally I decide that I should probably get the SSH fixed.  The conversation with tech support went something like this...

Me: My server is serving HTTP just fine but won't accept SSH -- it starts the handshake but hangs and then times out before getting to the password prompt. I tried from several different machines on different networks. Rebooting did not help.

Tech support: Did you try rebooting?

Me: ... yeah, that didn't help.

Tech support: What's your root password?

Me: ::grumble:: It's (password1).

Tech support: And the non-root user/password you log in with?

Me: ::sigh:: kentonv/(password2).

Tech support: The login you provided is not working. Is your data backed up? Maybe we should just wipe the machine.

This was obviously going nowhere.

Luckily, my Evlan server happens to feature the ability for me to log in and interactively execute Evlan code.  It doesn't provide any way to execute shell commands, but I was able to read and download all important files from my machine.

In the process, I took a look at /var/log/auth.log, where I saw this:

Jul  9 13:20:13 server013 login: 1 LOGIN FAILURE ON ttyv0
Jul  9 13:20:13 server013 login: 1 LOGIN FAILURE ON ttyv0, kentonv/(password2)

Obviously, auth.log does NOT normally contain plain-text passwords -- only usernames.  The tech guy had actually typed "kentonv/(password2)" as the *username*.

Abandon Ship

So, having rescued my data, I decided to abandon the silly Quebecois server.  Since the thing gets very little traffic anyway, I decided to just move it to my DSL.

One problem:  I didn't have a suitable server machine.  In fact, my main computer is a laptop that sleeps most of the time.  I actually quite like the fact that my electricity bill is $15/mo., not the $50/mo. it was back when I had a big power-hungry alway-on desktop.

So, I headed down to Fry's and picked up:

• Intel Atom CPU/motherboard (D510M0) $79.99
• 2GB PC6400 RAM $42.99
• 30GB SSD $84.99
• Mini-ITX case w/65W PSU $59.90

Total: $267.87

The best part about this little guy is the power supply.  It's 65W.  As in, the machine is not capable of using more than 65 watts.  That's less than a tenth of a modern gaming rig's power supply.  And the machine probably actually uses much less than 65W, given that it doesn't do video, has no peripherals attached, doesn't even have a CD drive, and uses flash storage.

I installed FreeBSD from a USB memory stick prepared using unetbootin, then set up:

• DJB's daemontools for service management.
• DJB's tinydns for my domains' DNS server.
• stunnel, which takes my HTTPS traffic, decrypts it, and forwards it on to Evlan.  I originally tried using the OpenSSL library directly, but its interface was absolutely horrid.
• Evlan, of course.

Despite the low power, the machine performs significantly better than the old one (a 5-year-old Celeron).  Overall I'm pretty impressed by how well it works.

UPDATE: Actual Real Ticket History

We begin right after the support people finally figured out how to log in...

Support:

hello,

even on root access we get permission denied why if im root?

i think ssh was disabled and need to be re-enable

thanks

Me:

What do you mean by this? What did you try to do that was denied?

SSH is not disabled -- it is still accepting connections, it just doesn't complete the handshake.

Support:

when i try some commands on root it says permission denied

now i have enable root access but its still doesn't give me a ssh box

thanks

Support (again):

and also when it reboots it gets stuck at

setting date via ntp?

do you have any idea about this?

thanks

Me:

What commands did you try?

Are you really root, or are are you still kentonv?

Support:

i was root all the time

any commande regarding ssh

thanks

Me:

Please be specific. What exact command did you type that said "permission denied"?

Support:

i don't remember i tried so many did you try to connect with PUTTY?

I can't believe I pay these people! (I'm still trying to log in just so that I can wipe the hard drive myself; I obviously don't trust them to do it.)