FRidh's blog

Discovery consists of seeing what everybody has seen and thinking what nobody has thought -- Albert Szent-Gyorgyi

Parameterized and distributed calculations using the Nix package manager

As part of my work I regularly perform simulations. We measure the density of a fluid flowing through a pipe by exciting the pipe and measuring the resulting vibrations. We would like to know for example how sensitive we are to density, but also pressure and temperature. Different models are used but the analysis is essentially always the same: compute a frequency spectrum as function of several parameters, such as pipe dimensions and material properties, fluid properties and state variables such as pressure and temperature.

These calculations typically take quite some time, so I would like to spread the load across multiple machines. There are plenty of tools available for this. For example, in clusters one could use PBS/TORQUE. The models I am using are implemented in Python, and for Python there are also many tools that can be used for this, such as Dask, IPython Parallel and Ray.

Then, there is the question where to keep your data. One issue I always run into is where to store your data. You might have a folder somewhere, but then you need a name, an identifier. And then you change a parameter, and of course a new identifier is needed.

Regardless of the solution you choose, you need to set up workers on the various hosts you want to use, and you need to solve the naming issue. I'd like to present here using the Nix package manager for this purpose.

Nix is a package manager. It's primary use case is building and installing software. Every artifact (or store path) it creates is a function of all the inputs used to create it. This means it creates a unique path in case any input changes, be it one of your calculation parameters, or maybe your script or Python interpreter. It also supports remote building, and hence it becomes possible to distribute calculations. While it lacks in features of monitoring your cluster, it is a simple method for distributing work, especially when you already have around some machines with Nix.

I've published a repository with a helper function for doing parameterized calculations. The flake.nix also shows a very simple example.

compute = builders.compute {
  fixedParameters = {
    a = 1;
    b = 2;
  variableParameters = {
    c = [1 2 3];
  executable = "${pkgs.python3.interpreter} ${./}";
  name = "result.json";

Constant parameters are defined as well as variable parameters. A Cartesian product is computed over the variable parameters. The executable defined is a script that should take a JSON file as input containing the parameters of a single calculation. A name is defined for the output file or path of a single calculation. The result of building the derivation returned by compute is a file listing all output files. Merging results is kept out of scope here.

Remote builders can be configured in NixOS as shown in the following snippet:

nix.buildMachines = [
    hostName = "";
    system = "x86_64-linux";
    maxJobs = 16;
    sshKey = "/root/.ssh/id_rsa";
    sshUser = "freddy";
    hostName = "";
    system = "x86_64-linux";
    maxJobs = 8;
    sshKey = "/root/.ssh/id_rsa";
    sshUser = "freddy";

which results in a

$ cat /etc/nix/machines 
freddy@ x86_64-linux /root/.ssh/id_rsa 16 1   -
freddy@ x86_64-linux /root/.ssh/id_rsa 8 1   -

Note it is important to use a key without passphrase.

Finished PhD

Back in September, I defended my PhD thesis. The PDF can be found in the Chalmers library. Furthermore, I made available a routine to build the PDF from source, including the majority of the simulations. This is done using the Nix package manager.

Urban Sound Planning Symposium

On the 15th and 16th of September the Urban Sound Planning Symposium took place at Muller-BBM in Planegg, Germany. This occassion also marked the end of the SONORUS project. At the symposium we mostly looked back at what we've achieved these last three years. There was a short poster session and after the event was over we got our "Urban Sound Planner" certificates. Finally I have something to hang up next to my shoe-lacing certificate.

Research visit to NASA Langley Research Center

In March I was at NASA Langley Research Center in Hampton, Virginia, for a four week research visit. The goal of the visit was to exchange knowledge on auralization of aircraft and atmospheric turbulence modeling. During this period we improved a model I had been working on and demonstrated that this model can be incorporated in the NASA Auralization Framework (NAF).

I was staying with a nice family in the town of Poquoson. While I've grown up nearby the coast I had never lived this close to the waterside before. I like the strong wind and smell of salt. It was an interesting experience living in this part of the United States and quite unlike the places I've lived in so far.

SONORUS update

It's been quite a while I've written on this blog about SONORUS, or anything else in fact. A lot has happened since then.

In October and November I was at Chalmers again for 6 weeks, following some courses. Furthermore, we had the school on Auralization and Visualization as well as a workshop around the Gothenburg test site.

After the research visit I attended NixCon in Berlin. NixCon was a conference about the Nix package manager and the ecosystem around it. Ever since I found out about Nix it fascinated me. Last summer I began using Nix and in autumn I switched to NixOS, an operating system entirely managed with Nix.

In January we had another SONORUS workshop, this time in Rome. It was my first time in Rome and I was glad to finally see the Colosseum as well as the Apostolic Palace.

In March I went to the U.S. for a four week research visit to NASA Langley Research Center. I wrote more about that in another post.

And just a couple of days ago we had another workshop, this time in Antwerp, Belgium.

The SONORUS project is almost over. In September there will be a final meeting in Munich. More about that later...

Synthesis of aircraft emission and propagation

This is a repost of a post of mine at the SONORUS blog

In an earlier post I gave an outline of my project, and mentioned that I am currently working on an aircraft emission model for auralization. Since then I developed a fully automated method to extract features from aircraft recordings.

The method is roughly as follows:

  • Backpropagate from source to receiver in time-domain, undoing the Doppler shift, atmospheric attenuation and spreading. The ground effect is ignored for now. We now have a signal that roughly corresponds to what is emitted from the airplane.
  • Determine fundamental frequency. An aircraft spectrum consists mostly of noise and tones, which are mostly harmonics. Knowing the fundamental frequency, allows you to determine the power of not only the fundamental, but also of each harmonic.
  • And that is the final step. Determine power of the tones, and consider the rest of the spectrum as noise.

The plan is now to determine these features from a large amount of events and develop a basic emission model. But before then, its important to already ask the following question. Are these features sufficient to create a realistic emission signal? Or, even better, are these features, taking into account the developed propagation model, providing a realistic signal at the receiver? Does, what you hear, really sound like an aircraft? That's in the end what we're after, right.

Synthesis of emission

To test whether the obtained features are sufficient to reproduce a realistic auralization, I considered a couple of events, and extracted the features for each. I synthesized an emission signal by summing all these time-varying components (about 200 tones and 30 1/3-octave bands), linearly interpolating the samples (features were obtained every second) and smoothing the interpolated components.

Click here to listen to a synthesis of the emission. It's an Airbus A320 taking of from Zurich Airport.

The first part of the fragment you can hear the tonal components quite clearly but in the latter part noise takes over. This is due to the directivity of the components; tones generated by the fan blades radiate mostly forward.

For comparison, here is the backpropagated signal. You'll notice it sounds very different. This is because the (unwanted) ground effect is still in it. And because of it, it sounds much more already like an actual aircraft fly-over!

At the receiver

So, let's listen now at a receiver at a height of 4 meters above the ground. When the aircraft is closest by, the distance is about 180 meters. Here are two fragments: fragment A and fragment B. One is a full auralization, the other is a recording. Can you hear which is which?

(hint: look at the URL in the top part of your browser if you want to know for sure)

There are two quite noticeable differences between the recording and the auralization. This specific auralization does not include the effects of turbulence, and the level of the blade passing frequency is lower than it should be.

Swiss Air Force demonstrations in Duebendorf

The headquarters of the Swiss Air Force is located in Duebendorf, and the airstrip of the base is not that far from where I work. Often you can see paratroopers and helicopters or hear other aircraft take off.

Occasionally, one of the demonstration teams practices. So far I've only seen the PC-7 Team, which use Swiss-manufactered Pilatus PC-7 turbo prop trainers. These aircraft can be quite noisy. Actually, it's not that the sound is very loud, it's just that the sound can be annoying. The propellors have a strong directivity and therefore, as they do their maneuvers, the level goes up and down.

They don't practice that often, and their sessions generally don't last more than a couple of hours. Therefore, I can't really be bothered too much by it. If I hear them, and it bothers me, I just close my window or turn the music up a bit. But what astonishes me is that they actually do these activities above dense populated areas. Many people are living and working in the area. What if an accident occurs?

I realize there aren't many places in Switzerland where they could build airstrips and do this kind of training without flying over someone's home. But flying over a dense area like this or flying over one or two farms is quite a difference. I am curious now to their risk assessment and how they've eventually decided to keep doing these activities here.

To conclude, these activities can be noisy, and perhaps also dangerous. However, on a more positive note, they also provide good entertainment during your break!