Sign Up for Free

RunKit +

Try any Node.js package right in your browser

This is a playground to test code. It runs a full Node.js environment and already has all of npm’s 1,000,000+ packages pre-installed, including concu with all npm packages installed. Try it out:

var concu = require("concu")

This service is provided by RunKit and is not affiliated with npm, Inc or the package authors.

concu v0.0.4

Concurrent execution of a method on a series of data

CircleCI

concu

A simple method wrapper for concurrent processing of an array in javascript.

Introduction

The Concu library helps you processing series of data (typically an array) by chunking your data in smaller bits and concurrently processing these chunks. This is typically useful if you have the possibility of processing data in parallel, however you want to have control over:

  • how many concurrent executions
  • size of the batch being processed

Limiting concurrency and batch size may be needed because of downstream I/O dependencies.

When opted for concurrency of 1, the method presents a simple sequential execution on partial data chunks.

The original focus was on helping to process large CSV files for which each row had a lot of IO, and each row could be processed independently of any other row.

Internals

Concu is basically a simple wrapper around around pLimit and Promise.all()

Usage

Typescript example

import { concu } from "concu"

// create an array with 10000 elements
const array = Array(1000);
const dataToProcess: number[] = [...array.keys()];


// Dummy method to just calculate the sum
const processData = (numbers: number[]): number => {
    return numbers.reduce( (sum, val) => sum+val, 0);
}

const concurrency = 10; // number of parallel executions
const chunkSize = 100; // size of every chunk

const run = async (): Promise<number[]> => {
    return await concu(processData, concurrency, chunkSize, dataToProcess)
};

run().then( (result: number[]) => {
    const sum = result.reduce( (total, value) => total + value, 0);
    console.log(sum);
});

API

concu(fun, chunkSize, concurrency, data, ...args)

returns a Promise resolving to an array of individual results of each individual execution

fun: (data: any[], ...args: any[]) => any

function which accepts a data array and other possible arguments.

chunkSize: number

How many elements of the array to be processed

concurrency: number

how many executions in parallel

...args: any[]

optional arguments that will be passed to fun

Acknowledgments

This work has been created at Locai Solutions as part of the data import efforts.

Metadata

RunKit is a free, in-browser JavaScript dev environment for prototyping Node.js code, with every npm package installed. Sign up to share your code.
Sign Up for Free