Last week, I spent some time looking at the feasibility of using SSHFS on OS X to access Google’s centralized source tree for the purpose of issuing local builds. My goals were two-fold: first, to avoid having to “clone” the large source code of the apps I wanted to build; and, second, to avoid having to port the source file system (a FUSE module) to the Mac.

What I found highlights that SSHFS is not the right choice for locally building a remote source tree. That said, the overall study process was interesting, fun, and I am now tempted to make SSHFS viable for this use case. Read on for the details.

Obligatory disclaimer: There is not much specific to Google here and any opinions are my own. In fact, I had used SSHFS in the past for similar purposes: to expose a NetBSD source tree from the host machine to a VM running on it, using the virtual network interface. Knowing these details at the time would have been very helpful in understanding the behavior of the setup.

The source file system

The Google source tree is exposed to developer machines by a FUSE file system that exposes the (very) large centralized repository at any desired revision; think of paths of the form .../src/<revision>/. Let’s call this FUSE file system the source file system, about which you can learn more in the Build in the cloud: Accessing source code blog post and in the following presentation:

The key characteristics, for the understanding of this article, of the source file system are:

If this sounds interesting, there is a similar filesystem in spirit available in GitHub under the hanwen/p4fuse repository.

Experimental setup

In order to quantify if the source file system mounted over SSHFS is a workable environment for large builds, I tested various configurations and measured the build times of different iOS applications.

In particular, I used the following configurations:

In the text below, the local machine is always the OS X system; the remote machine (hosting the source file system) is always a Linux machine.

The experiments were done on a Mac Pro (Late 2013) with a 6-core Intel Xeon X5 CPU @ 3.5 GHz and 32GB of RAM using a remote Linux workstation of similar hardware. Both machines were physically connected to the same network offering average ping times of 0.5ms and a throughput around 50MB/s for SSH-based file copies. The Linux machine was running OpenSSH 7.1.

Results

The table below illustrates the time it took to build some iOS apps under the tested configurations. The numbers shown in the tables are the best numbers in multiple runs, some of which involved tweaked versions of both SSHFS and pCacheFS to obtain higher performance from them.

Colums showing Not tested are for measurements that were not taken because previous measurements on other configurations showed that they would be worthless: i.e. that no performance gain would be obtained. The SSHFS cold/hot references account for both SSHFS and the remote source file system being hot or cold in unison because they are tightly related in the common case.

Target Local SSHFS (cold) SSHFS (hot) SSHFS+pCacheFS (cold) SSHFS+pCacheFS (hot)
Small app 5m 23m 6m 18m 13m
Larger app 5m 54m 25m Not tested Not tested

Yes, I only tested two builds because these were enough to yield conclusions. The Demo app showed some promising results when the caches were hot, but as soon as I tried a real-world app (the Larger app), it became clear that this setup was unfeasible.

As you can see in the table above, the results are disheartening. For the small build, the source file system over SSHFS is a feasible choice assuming the caches are hot. For any larger build, SSHFS introduces a significant performance penalty on the build times. This negative impact does not justify the simplicity of using the SSHFS-based setup and developers should just resort to local checkouts of the parts of the tree they need.

Knowing this, let’s move on to the more interesting part of this article, which is all the learnings I got from the file systems implementations and all the tweaks I made in an attempt to obtain higher performance.

SSHFS observations

pCacheFS observations

Conclusions

Neither the SSHFS client file system, the sshd server, nor the pCacheFS file system have been optimized for the workload of a large build using a source tree mounted over SSHFS. There are performance bottlenecks in various areas and there are missing features in all layers.

Here are some ideas on how the current situation could be improved:

I consider my experimentation on this area done for now, but all of the items above sound interesting. In fact, after spending so many hours studying this, I cannot help but think of implementing a solution and watching it work. Maybe I’ll give them a try… if I have the time.

Oh, and how easy this all was! Developing file system code using something like FUSE coughrumpcough is a joy. The last time I dealt with low level file system code was back in 2005 when I wrote tmpfs for NetBSD, and I did not have virtual machines nor user-space file systems available! That was painful.

Annex: Contributions

This work resulted in the following open source contributions so far:

Subscribe via RSS · Go to posts index

   Delivered by FeedBurner