Design Pattern for Buffering Pipeline Input in a PowerShell Cmdlet

Question

I sometimes encounter situations where it makes sense to support the input of the pipeline to the cmdlet, but the operations that I want to perform (for example, accessing the database) make sense for batch processing a reasonable number of objects.

A typical way to achieve this is something like this:

function BufferExample {
<#
.SYNOPSIS
Example of filling and using an intermediate buffer.
#>
[CmdletBinding()]
param(
    [Parameter(ValueFromPipeline)]
    $InputObject
)

BEGIN {
    $Buffer = New-Object System.Collections.ArrayList(10)
    function _PROCESS {
        # Do something with a batch of items here.
        Write-Output "Element 1 of the batch is $($Buffer[1])"
        # This could be a high latency operation such as a DB call where we 
        # retrieve a set of rows by primary key rather than each one individually.

        # Then empty the buffer.
        $Buffer.Clear()
    }
}

PROCESS {
    # Accumulate the buffer or process it if the buffer is full.
    if ($Buffer.Count -ne $Buffer.Capacity) {    
        [void]$Buffer.Add($InputObject)        
    } else {
        _PROCESS 
    }
}

END {
     # The buffer may be partially filled, so let do something with the remainder.
    _PROCESS
}
}

Is there a less “boilerplate" way to do this?

One method may be to write a function, which I call "_PROCESS" here, to accept the argument (s) of the array, but not to enter the pipeline, and then for the cmdlet susceptible to the user, the proxy function created to buffer input and transfers in the buffer, as described in the proxy command .

, , , /.

Answer 1

(), , , , Process ( ) .

, ; Process, End; , Sort-Object.

, - , , , , . .

---

2 (, ?) .

, Graphite. Format-Graphite Out-Graphite; , (-) . , , Format-Graphite , , . , . , .

---

, "" - , , Begin End, Process .

, , Begin, , - SQL .

---

, , , / / .

, ; .

Answer 2

, , .

, , LOGS .

, Dot Source , script, .

- .

, .

Answer 3

I made a reusable cmdlet from your example.

function Buffer-Pipeline {
[CmdletBinding()]
param(
    $Size = 10,
    [Parameter(ValueFromPipeline)]
    $InputObject
)
    BEGIN {
        $Buffer = New-Object System.Collections.ArrayList($Size)
    }

    PROCESS {
        [void]$Buffer.Add($_)

        if ($Buffer.Count -eq $Size) {    
            $b = $Buffer;
            $Buffer = New-Object System.Collections.ArrayList($Size)
            Write-Output -NoEnumerate $b
        }
    }

    END {
     # The buffer may be partially filled, so let do something with the remainder.
        if ($Buffer.Count -ne 0) {  
            Write-Output -NoEnumerate $Buffer
        }
    }
}

Using:

@(1;2;3;4;5) | Buffer-Pipeline -Size 3 | % { "$($_.Count) items: ($($_ -join ','))" }

Output:

3 items: (1,2,3)
2 items: (4,5)

Another example:

1,2,3,4,5 | Buffer-Pipeline -Size 10 | Measure-Object

Count    : 2

Processing of each batch:

 1,2,3 | Buffer-Pipeline -Size 2 | % { 
    $_ | % { "Starting batch of $($_.Count)" } { $_ * 100 } { "Done with batch of $($_.Count)" }
}

Starting batch of 2
100
200
Done with batch of 2
Starting batch of 1
300
Done with batch of 1