mirai - Parallel Integration

Parallel Integration

mirai provides an alternative communications backend for R. This functionality was developed to fulfil a request by R Core at R Project Sprint 2023.

make_cluster() creates a cluster object of class ‘miraiCluster’, which is fully-compatible with parallel cluster types.

Specify ‘n’ to launch nodes on the local machine.
Specify ‘url’ for receiving connections from remote nodes.
Optionally, specify ‘remote’ to launch remote daemons using a remote configuration generated by remote_config() or ssh_config().

Created clusters may be used for any function in the parallel base package such as parallel::clusterApply() or parallel::parLapply(), or the load-balanced versions such as parallel::parLapplyLB().

library(mirai)

cl <- make_cluster(4)
cl
#> < miraiCluster | ID: `0` nodes: 4 active: TRUE >
parallel::parLapply(cl, iris, mean)
#> $Sepal.Length
#> [1] 5.843333
#> 
#> $Sepal.Width
#> [1] 3.057333
#> 
#> $Petal.Length
#> [1] 3.758
#> 
#> $Petal.Width
#> [1] 1.199333
#> 
#> $Species
#> [1] NA

status() may be called on a ’miraiCluster` to query the number of connected nodes at any time.

status(cl)
#> $connections
#> [1] 4
#> 
#> $daemons
#> [1] "abstract://84c8107ee842139796c7f87f"
stop_cluster(cl)

Making a cluster specifying ‘url’ without ‘remote’ causes the shell commands for manual deployment of nodes to be printed to the console.

cl <- make_cluster(n = 2, url = host_url())
#> Shell commands for deployment on nodes:
#> 
#> [1]
#> Rscript -e 'mirai::daemon("tcp://kumamoto:42603",rs=c(10407,-2096125748,1743292253,-1955520902,-2036622925,1260071768,-1320342151))'
#> 
#> [2]
#> Rscript -e 'mirai::daemon("tcp://kumamoto:42603",rs=c(10407,-1702861686,161023499,1803127216,-1397060724,-897111933,-129599054))'
stop_cluster(cl)

Foreach Integration

A ‘miraiCluster’ may also be registered by doParallel for use with the foreach package.

Running some parallel examples for the foreach() function:

library(foreach)
library(iterators)

cl <- make_cluster(4)
doParallel::registerDoParallel(cl)

# normalize the rows of a matrix
m <- matrix(rnorm(9), 3, 3)
foreach(i = 1:nrow(m), .combine = rbind) %dopar%
  (m[i, ] / mean(m[i, ]))
#>                [,1]       [,2]      [,3]
#> result.1  0.3487084 -0.4731823 3.1244739
#> result.2  1.3038052  1.5895562 0.1066386
#> result.3 -0.3771049  1.5104437 1.8666612
# simple parallel matrix multiply
a <- matrix(1:16, 4, 4)
b <- t(a)
foreach(b = iter(b, by='col'), .combine = cbind) %dopar%
  (a %*% b)
#>      [,1] [,2] [,3] [,4]
#> [1,]  276  304  332  360
#> [2,]  304  336  368  400
#> [3,]  332  368  404  440
#> [4,]  360  400  440  480