Using PowerShell v3 Invoke-RestMethod for PUT / POST X MB binary

Question

I worked a bit with PowerShell v3 (CTP2 from here ) and its new Invoke-RestMethod method:

Invoke-RestMethod -Uri $ dest -mode PUT -Credential $ cred -InFile $ file

However, I would like to use this to push very large binary objects, and therefore, I would like to push a range of bytes from a large binary.

For example, if I have a VHD of 20Gb, I would like to break it into pieces, say, 5Gb each (without splitting and saving individual fragments) and PUT / POST them in the BLOB storage, for example S3, Rackspace, Azure, etc. I also assume the chunk size is larger than the available memory.

I read that Get-Content does not work very well on large binary files, but this does not seem like an obscure requirement. Does anyone have any ratings that can be used for this, especially in combination with the new PowerShell Invoke-RestMethod?

Answer 1

I believe the Invoke-RestMethod parameter you are looking for is

-TransferEncoding Chunked 

but there is no control over the size of the block or buffer. Someone can fix me if I'm wrong, but I think the block size is 4 KB. Each piece is loaded into memory and then sent, so you do not fill out the file that you send.

Answer 2

To get sections (chunks) of a file, you can create a System.IO.BinaryReader , which has a convenient dandy Read( [Byte[]] buffer, [int] offset, [int] length) method Read( [Byte[]] buffer, [int] offset, [int] length) . Here's a feature that makes it easier:

 function Read-Bytes { [CmdletBinding()] param ( [Parameter(Mandatory = $true, Position = 0)] [string] $Path , [Parameter(Mandatory = $true, Position = 1)] [int] $Offset , [Parameter(Mandatory = $true, Position = 2)] [int] $Size ) if (!(Test-Path -Path $Path)) { throw ('Could not locate file: {0}' -f $Path); } # Initialize a byte array to hold the buffer $Buffer = [Byte[]]@(0)*$Size; # Get a reference to the file $FileStream = (Get-Item -Path $Path).OpenRead(); if ($Offset -lt $FileStream.Length) { $FileStream.Position = $Offset; Write-Debug -Message ('Set FileStream position to {0}' -f $Offset); } else { throw ('Failed to set $FileStream offset to {0}' -f $Offset); } $ReadResult = $FileStream.Read($Buffer, 0, $Size); $FileStream.Close(); # Write buffer to PowerShell pipeline Write-Output -InputObject $Buffer; } Read-Bytes -Path C:\Windows\System32\KBDIT142.DLL -Size 10 -Offset 90;