07-12-2018, 09:28 PM
SBC Clusters — Beyond Raspberry Pi
<div style="margin: 5px 5% 10px 5%;"><img src="http://www.sickgaming.net/blog/wp-content/uploads/2018/07/sbc-clusters-beyond-raspberry-pi.jpg" width="1920" height="893" title="" alt="" /></div><div><div><img src="http://www.sickgaming.net/blog/wp-content/uploads/2018/07/sbc-clusters-beyond-raspberry-pi.jpg" class="ff-og-image-inserted" /></div>
<p><span><span>Cluster computers constructed of Raspberry Pi SBCs have been </span><a href="https://www.networkworld.com/article/3156748/computers/10-amazing-raspberry-pi-clusters.html#slide1"><span>around for years</span></a><span>, ranging from supercomputer-like behemoths to simple hobbyist rigs. More recently, we’ve seen cluster designs that use other open-spec hacker boards, many of which offer higher computer power and faster networking at the same or lower price. Farther below, we’ll examine one recent open source design from Paul Smith at Climbers.net that combines 12 octa-core NanoPi-Fire3 SBCs for a 96-core cluster.</span></span></p>
<p><span><span>SBC-based clusters primarily fill the needs of computer researchers who find it too expensive to book time on a server-based HPC (high performance computing) cluster. Large-scale HPC clusters are in such high demand, that it’s hard to find available cluster time in the first place. </span></span></p>
<p><span><span>Research centers and universities around the world have developed RPi-based cluster computing for research into parallel computing, deep learning, </span><span>medical research, weather simulations, cryptocurrency mining, </span><span>software-defined networks, distributed storage, </span><span>and more. Clusters have been deployed to provide a high degree of redundancy or to simulate massive IoT networks, such as with Resin.io’s 144-pi </span><a href="https://resin.io/blog/the-evolution-of-the-beast-continues/"><span>Beast v3</span></a><span>. </span></span></p>
<p><span><span>Even the largest of these clusters comes nowhere close to the performance of server-based HPC clusters. Yet, in many research scenarios, top performance is not essential. It’s the combination of separate cores running in parallel that make the difference. The Raspberry Pi based systems typically use the </span><span>MPI (Messaging Passing Interface) library for exchanging messages between computers to deploy a parallel program across distributed memory.</span></span></p>
<p><span><span>BitScope, which is a leader in Pi cluster hardware such as its </span><a href="http://my.bitscope.com/store/?p=view&i=product+BB04A"><span>Bitscope Blade</span></a><span>, has developed a system with </span><span>Los Alamos National Laboratory </span><span>based on its larger </span><span>BitScope Cluster Module. The </span><a href="https://www.raspberrypi.org/blog/raspberry-pi-clusters-come-of-age/"><span>Los Alamos hosted </span><span>system</span></a><span> comprises five racks of 150 Raspberry Pi 3 SBCs. Multiply those 750 boards by the four Cortex-A53 cores on each Pi and you get a 3,000-core parallelized supercomputer. </span></span></p>
<p><span><span>The Los Alamos system is said to be far more affordable and power efficient than building a </span><span>dedicated testbed of the same size using conventional technology, which would cost a quarter billion dollars and use 25 megawatts of electricity. There are now plans to move to a 4,000-core Pi cluster.</span></span></p>
<p><span><span>Most clusters are much smaller 5-25 board rigs, and are typically deployed by educators, hobbyists, embedded engineers, and even artists and musicians. These range from open source DIY designs to commercial hardware rack systems designed to power and cool multiple densely packed compute boards. </span></span></p>
<h3><span><span>96-core NanoPi Fire3 cluster shows impressive benchmarks</span></span></h3>
<p><span><span>The </span><a href="https://climbers.net/sbc/nanopi-fire3-arm-supercomputer/"><span>96-core cluster computer</span></a><span> recently detailed on Climbers.net is the largest of </span><a href="https://climbers.net/sbc/"><span>several cluster designs</span></a><span> developed by Nick Smith. These include a </span><a href="https://climbers.net/sbc/40-core-arm-cluster-nanopc-t3/"><span>40-core system</span></a><span> based on the octa-core NanoPC-T3, and others that use the Pine A64+, the Orange Pi Plus 2E, and various Raspberry Pi models. (All these SBCs can be found in our recently concluded </span><a href="https://www.linux.com/blog/2018/6/raspberry-pi-3-b-tops-hacker-board-survey"><span>hacker board reader survey</span></a><span>.)</span></span></p>
<p><span><span>The new cluster, which was spotted by </span><a href="https://www.worksonarm.com/blog/woa-issue-61/"><span>Worksonarm</span></a><span> and further described on </span><a href="https://www.cnx-software.com/2018/07/10/nanopi-fire3-cluster-diy-portable-solution-develop-distributed-software/"><span>CNXSoft</span></a><span>, uses FriendlyElec’s open-spec </span><a href="http://linuxgizmos.com/two-tiny-nanopi-sbcs-debut-ubuntu-core-based-friendlycore/"><span>NanoPi Fire3</span></a><span>. </span></span></p>
<p><span><span>The open source cluster design includes Inkscape code for laser cutter construction. Smith made numerous changes to his earlier clusters intended to increase heat dissipation, improve durability, and reduce space, cost, and power consumption. These include offering two 7W case fans instead of one and moving to a GbE switch. </span></span><span><span>The Bill of Materials ran to just over £543 ($717), with the NanoPi Fire3 boards totaling £383, including shipping. The next biggest shopping item was £62 for microSD cards.</span></span></p>
<p><span><span>The $35 Fire3 SBC, which measures only </span><span>75x40mm, houses a powerful Samsung S5P6818. The SoC features 8x Cortex-A53 cores at up to 1.4GHz and a Mali-400 MP4 GPU, which runs a bit faster than the Raspberry Pi’s VideoCore IV.</span></span></p>
<p><span><span>Although the Fire3 has only twice the number of -A53 cores as the Raspberry Pi 3, and is clocked only slightly faster, Smith’s benchmarks showed a surprising 6.6x times faster CPU boost over a similar RPi 3 cluster. GPU performance was 7.5x faster. </span></span></p>
<p><span><span>It turned out that much of the performance improvement was due to the Fire3’s native, PCIe-based Gigabit Ethernet port, which enabled the clustered SBCs to communicate more quickly with one another to run parallel computing applications. By comparison, the Raspberry Pi 3 has a 10/100Mbps port. </span></span></p>
<p><span><span>Performance would no doubt improve if Smith had used the new </span><a href="http://linuxgizmos.com/raspberry-pi-3-b-wins-hacker-board-reader-survey/"><span>Raspberry Pi 3 Model B+</span></a><span>, which offers a Gigabit Ethernet port. However, since the B+ port is based on USB 2.0, its Ethernet throughput is only three times faster than the Model B’s 10/100 port instead of about 10 times faster for the Fire3. </span></span></p>
<p><span><span>Still, that’s a significant throughput boost, and combined with the faster 1.4GHz clock rate, the RPi 3 B+ should quickly replace the RPi 3 Model B in Pi-based cluster designs. BitScope recently posted an </span><a href="http://www.bitscope.com/blog/FM/?p=HB28A"><span>enthusiastic review</span></a><span> of the B+. In addition to the performance improvements, the review cites the improved heat dissipation from the PCB design and the </span><span>“flip chip on silicon” BGA package for the Broadcom SoC, which uses heat spreading metal. The upcoming Power-over-Ethernet capability should also open new possibilities for clusters, says the review.</span></span></p>
<p><span><span>Hacker board community sites are increasingly showcasing cluster designs — here’s a </span><a href="https://www.thingiverse.com/thing:2766426"><span>cluster case design</span></a><span> for the Orange Pi One on Thingiverse — and some vendors offer cluster hardware of their own. Hardkernel’s Odroid project, for example, came out with a 4-board, 32-core Odroid-MC1 cluster computer based on an Odroid-XU4S SBC, a modified version of the Odroid-XU4, which won third place in our hacker board survey. The board uses the same octa-core -A15 Samsung Exynos5422 SoC. More recently, it released an </span><a href="http://linuxgizmos.com/single-unit-version-of-odroid-mc1-cluster-computer-adds-flexibility/"><span>Odroid-MC1 Solo</span></a><span> version that lets you choose precisely how many boards you want to add.</span></span></p>
<p><span><span>The Odroid-MC1 products are primarily designed to run Docker Swarm. Many of the cluster systems are designed to run Docker or other cloud-based software. Last year Alex Ellis, for example, posted a </span><a href="https://www.linux.com/news/your-serverless-raspberry-pi-cluster-docker"><span>tutorial</span></a><span> on creating a Serverless Raspberry Pi cluster that runs Docker and the </span><a href="https://github.com/alexellis/faas"><span>OpenFaaS</span></a><span> framework. Indeed, as with edge computing devices running modified versions of cloud software, such as AWS Greengrass, cluster computers based on SBCs show another example of how the embedded and enterprise server worlds are interacting in interesting new ways using Linux.</span></span></p>
<p><em><span><span>Join us at </span><a href="https://events.linuxfoundation.org/events/elc-openiot-europe-2018/"><span>Open Source Summit + Embedded Linux Conference Europe</span></a><span> in Edinburgh, UK on October 22-24, 2018, for 100+ sessions on Linux, Cloud, Containers, AI, Community, and more.</span></span></em></p>
</div>
<div style="margin: 5px 5% 10px 5%;"><img src="http://www.sickgaming.net/blog/wp-content/uploads/2018/07/sbc-clusters-beyond-raspberry-pi.jpg" width="1920" height="893" title="" alt="" /></div><div><div><img src="http://www.sickgaming.net/blog/wp-content/uploads/2018/07/sbc-clusters-beyond-raspberry-pi.jpg" class="ff-og-image-inserted" /></div>
<p><span><span>Cluster computers constructed of Raspberry Pi SBCs have been </span><a href="https://www.networkworld.com/article/3156748/computers/10-amazing-raspberry-pi-clusters.html#slide1"><span>around for years</span></a><span>, ranging from supercomputer-like behemoths to simple hobbyist rigs. More recently, we’ve seen cluster designs that use other open-spec hacker boards, many of which offer higher computer power and faster networking at the same or lower price. Farther below, we’ll examine one recent open source design from Paul Smith at Climbers.net that combines 12 octa-core NanoPi-Fire3 SBCs for a 96-core cluster.</span></span></p>
<p><span><span>SBC-based clusters primarily fill the needs of computer researchers who find it too expensive to book time on a server-based HPC (high performance computing) cluster. Large-scale HPC clusters are in such high demand, that it’s hard to find available cluster time in the first place. </span></span></p>
<p><span><span>Research centers and universities around the world have developed RPi-based cluster computing for research into parallel computing, deep learning, </span><span>medical research, weather simulations, cryptocurrency mining, </span><span>software-defined networks, distributed storage, </span><span>and more. Clusters have been deployed to provide a high degree of redundancy or to simulate massive IoT networks, such as with Resin.io’s 144-pi </span><a href="https://resin.io/blog/the-evolution-of-the-beast-continues/"><span>Beast v3</span></a><span>. </span></span></p>
<p><span><span>Even the largest of these clusters comes nowhere close to the performance of server-based HPC clusters. Yet, in many research scenarios, top performance is not essential. It’s the combination of separate cores running in parallel that make the difference. The Raspberry Pi based systems typically use the </span><span>MPI (Messaging Passing Interface) library for exchanging messages between computers to deploy a parallel program across distributed memory.</span></span></p>
<p><span><span>BitScope, which is a leader in Pi cluster hardware such as its </span><a href="http://my.bitscope.com/store/?p=view&i=product+BB04A"><span>Bitscope Blade</span></a><span>, has developed a system with </span><span>Los Alamos National Laboratory </span><span>based on its larger </span><span>BitScope Cluster Module. The </span><a href="https://www.raspberrypi.org/blog/raspberry-pi-clusters-come-of-age/"><span>Los Alamos hosted </span><span>system</span></a><span> comprises five racks of 150 Raspberry Pi 3 SBCs. Multiply those 750 boards by the four Cortex-A53 cores on each Pi and you get a 3,000-core parallelized supercomputer. </span></span></p>
<p><span><span>The Los Alamos system is said to be far more affordable and power efficient than building a </span><span>dedicated testbed of the same size using conventional technology, which would cost a quarter billion dollars and use 25 megawatts of electricity. There are now plans to move to a 4,000-core Pi cluster.</span></span></p>
<p><span><span>Most clusters are much smaller 5-25 board rigs, and are typically deployed by educators, hobbyists, embedded engineers, and even artists and musicians. These range from open source DIY designs to commercial hardware rack systems designed to power and cool multiple densely packed compute boards. </span></span></p>
<h3><span><span>96-core NanoPi Fire3 cluster shows impressive benchmarks</span></span></h3>
<p><span><span>The </span><a href="https://climbers.net/sbc/nanopi-fire3-arm-supercomputer/"><span>96-core cluster computer</span></a><span> recently detailed on Climbers.net is the largest of </span><a href="https://climbers.net/sbc/"><span>several cluster designs</span></a><span> developed by Nick Smith. These include a </span><a href="https://climbers.net/sbc/40-core-arm-cluster-nanopc-t3/"><span>40-core system</span></a><span> based on the octa-core NanoPC-T3, and others that use the Pine A64+, the Orange Pi Plus 2E, and various Raspberry Pi models. (All these SBCs can be found in our recently concluded </span><a href="https://www.linux.com/blog/2018/6/raspberry-pi-3-b-tops-hacker-board-survey"><span>hacker board reader survey</span></a><span>.)</span></span></p>
<p><span><span>The new cluster, which was spotted by </span><a href="https://www.worksonarm.com/blog/woa-issue-61/"><span>Worksonarm</span></a><span> and further described on </span><a href="https://www.cnx-software.com/2018/07/10/nanopi-fire3-cluster-diy-portable-solution-develop-distributed-software/"><span>CNXSoft</span></a><span>, uses FriendlyElec’s open-spec </span><a href="http://linuxgizmos.com/two-tiny-nanopi-sbcs-debut-ubuntu-core-based-friendlycore/"><span>NanoPi Fire3</span></a><span>. </span></span></p>
<p><span><span>The open source cluster design includes Inkscape code for laser cutter construction. Smith made numerous changes to his earlier clusters intended to increase heat dissipation, improve durability, and reduce space, cost, and power consumption. These include offering two 7W case fans instead of one and moving to a GbE switch. </span></span><span><span>The Bill of Materials ran to just over £543 ($717), with the NanoPi Fire3 boards totaling £383, including shipping. The next biggest shopping item was £62 for microSD cards.</span></span></p>
<p><span><span>The $35 Fire3 SBC, which measures only </span><span>75x40mm, houses a powerful Samsung S5P6818. The SoC features 8x Cortex-A53 cores at up to 1.4GHz and a Mali-400 MP4 GPU, which runs a bit faster than the Raspberry Pi’s VideoCore IV.</span></span></p>
<p><span><span>Although the Fire3 has only twice the number of -A53 cores as the Raspberry Pi 3, and is clocked only slightly faster, Smith’s benchmarks showed a surprising 6.6x times faster CPU boost over a similar RPi 3 cluster. GPU performance was 7.5x faster. </span></span></p>
<p><span><span>It turned out that much of the performance improvement was due to the Fire3’s native, PCIe-based Gigabit Ethernet port, which enabled the clustered SBCs to communicate more quickly with one another to run parallel computing applications. By comparison, the Raspberry Pi 3 has a 10/100Mbps port. </span></span></p>
<p><span><span>Performance would no doubt improve if Smith had used the new </span><a href="http://linuxgizmos.com/raspberry-pi-3-b-wins-hacker-board-reader-survey/"><span>Raspberry Pi 3 Model B+</span></a><span>, which offers a Gigabit Ethernet port. However, since the B+ port is based on USB 2.0, its Ethernet throughput is only three times faster than the Model B’s 10/100 port instead of about 10 times faster for the Fire3. </span></span></p>
<p><span><span>Still, that’s a significant throughput boost, and combined with the faster 1.4GHz clock rate, the RPi 3 B+ should quickly replace the RPi 3 Model B in Pi-based cluster designs. BitScope recently posted an </span><a href="http://www.bitscope.com/blog/FM/?p=HB28A"><span>enthusiastic review</span></a><span> of the B+. In addition to the performance improvements, the review cites the improved heat dissipation from the PCB design and the </span><span>“flip chip on silicon” BGA package for the Broadcom SoC, which uses heat spreading metal. The upcoming Power-over-Ethernet capability should also open new possibilities for clusters, says the review.</span></span></p>
<p><span><span>Hacker board community sites are increasingly showcasing cluster designs — here’s a </span><a href="https://www.thingiverse.com/thing:2766426"><span>cluster case design</span></a><span> for the Orange Pi One on Thingiverse — and some vendors offer cluster hardware of their own. Hardkernel’s Odroid project, for example, came out with a 4-board, 32-core Odroid-MC1 cluster computer based on an Odroid-XU4S SBC, a modified version of the Odroid-XU4, which won third place in our hacker board survey. The board uses the same octa-core -A15 Samsung Exynos5422 SoC. More recently, it released an </span><a href="http://linuxgizmos.com/single-unit-version-of-odroid-mc1-cluster-computer-adds-flexibility/"><span>Odroid-MC1 Solo</span></a><span> version that lets you choose precisely how many boards you want to add.</span></span></p>
<p><span><span>The Odroid-MC1 products are primarily designed to run Docker Swarm. Many of the cluster systems are designed to run Docker or other cloud-based software. Last year Alex Ellis, for example, posted a </span><a href="https://www.linux.com/news/your-serverless-raspberry-pi-cluster-docker"><span>tutorial</span></a><span> on creating a Serverless Raspberry Pi cluster that runs Docker and the </span><a href="https://github.com/alexellis/faas"><span>OpenFaaS</span></a><span> framework. Indeed, as with edge computing devices running modified versions of cloud software, such as AWS Greengrass, cluster computers based on SBCs show another example of how the embedded and enterprise server worlds are interacting in interesting new ways using Linux.</span></span></p>
<p><em><span><span>Join us at </span><a href="https://events.linuxfoundation.org/events/elc-openiot-europe-2018/"><span>Open Source Summit + Embedded Linux Conference Europe</span></a><span> in Edinburgh, UK on October 22-24, 2018, for 100+ sessions on Linux, Cloud, Containers, AI, Community, and more.</span></span></em></p>
</div>