module ChefUtils::ParallelMap

This module contains ruby refinements that adds several methods to the Enumerable class which are useful for parallel processing.

Public Instance Methods

flat_each () { |value| ... }

Source

# File lib/chef-utils/parallel_map.rb, line 82
def flat_each(&block)
  map do |value|
    if value.is_a?(Enumerable)
      value.each(&block)
    else
      yield value
    end
  end
end

The flat_each method is tightly coupled to the usage of parallel_map within the ChefFS implementation. It is not itself a parallel method, but it is used to iterate through the 2nd level of nested structure, which is tied to the nested structures that ChefFS returns.

This is different from Enumerable#flat_map because that behaves like map.flatten(1) while this behaves more like flatten(1).each. We need this on an Enumerable, so we have no Enumerable#flatten method to call.

[ 1, 2 ], [ 3, 4: ].flat_each(&block) calls block four times with 1, 2, 3, 4
[ 1, 2 ], [ 3, 4: ].flat_map(&block) calls block twice with [1, 2] and [3,4]

parallel_each (pool: nil, &block)

Source

# File lib/chef-utils/parallel_map.rb, line 61
def parallel_each(pool: nil, &block)
  return self unless block_given?

  parallel_map(pool: pool, &block)

  self
end

This has the same behavior as parallel_map but returns the enumerator instead of the return values.

@return [Enumerable] the enumerable for method chaining

parallel_map (pool: nil) { |item| ... }

Source

# File lib/chef-utils/parallel_map.rb, line 42
def parallel_map(pool: nil)
  return self unless block_given?

  pool ||= ChefUtils::DefaultThreadPool.instance.pool

  futures = map do |item|
    Concurrent::Future.execute(executor: pool) do
      yield item
    end
  end

  futures.map(&:value!)
end

Enumerates through the collection in parallel using the thread pool provided or the default thread pool. By using the default thread pool this supports recursively calling the method without deadlocking while using a globally fixed number of workers. This method supports lazy collections. It returns synchronously, waiting until all the work is done. Failures are only reported after the collection has executed and only the first exception is raised.

(0..).lazy.parallel_map { |i| i*i }.first(5)

@return [Array] output results