The most recent Naiad release contains GraphLINQ, a new library of LINQ-like operators full of graph-specific optimizations. GraphLINQ is an example of how one can build up domain-specific libraries with carefully tailored implementations, which nonetheless integrate smoothly with Naiad and the rest of its libraries. This post (first of several) explains some of GraphLINQ’s methods by building up a PageRank example. We will talk more about asynchronous graph processing, and how GraphLINQ is built in future posts.
Since we first announced Naiad back in 2012, the system has grown in many ways, including performance, robustness, and support for other platforms. From a developer’s point of view, the biggest change came in early 2013, when we split the project into two parts, so that our differential dataflow implementation became a library on top of the more general distributed Naiad runtime. The main consequence of this split is that you can now build new frameworks and DSLs on top of Naiad, and take full advantage of its high-performance runtime. In future posts, we will talk more about some of the frameworks that we have released with Naiad. Today I’m going to show how easy it is to build a new framework, and the opportunities that this opens up.
One of the most commonly asked questions about Naiad is, “How on earth do you run it on a cluster?” When we first released Naiad, the only solution available was to grit your teeth and run the distributed programs manually, using scripts that were manually tailored to a particular cluster. In the mean time, Hadoop YARN has become a widely available framework for running data-processing applications on a cluster. This post describes how we ported the latest version of Naiad to run on top of Hadoop, and how this makes it easier to run Naiad programs on your data in Microsoft Azure.
Since we released Naiad as open-source software on GitHub last October, we have been busy adding features that make it easier to use Naiad for your applications, in settings ranging from your laptop to a distributed cluster. In the coming weeks, we’ll be posting about what’s new. This post gathers together the things that have changed.
We have recently been talking about Naiad’s low latency (see, for example, Derek’s presentation at SOSP). If you have ever tried to coax good performance out of a distributed system, you may be wondering exactly how we get coordination latencies of less than 1ms across 64 computers with 8 parallel workers per machine. In this series of posts we’re going to reveal to our curious – and possibly skeptical – users what we did in gory detail.