VSTS Ask Agent Questions at the Publish Artifact: Fall Stage

Question

We see inconsistent crashes in our vsts builds. They came out of nowhere.

• The agent runs on-prem. When using a hosted build agent, we do not see any problems.
• Successive builds require many retries. Usually the assembly fails, but sometimes it is not.
• Fixed multiple design definitions in different projects.

We are trying to find the root cause, any help diagnosing this problem will be greatly appreciated!

Here is the relevant version information:

[2017-04-25 07: 25: 58Z INFO AgentProcess] Agent created for Windows - win7-x64. [2017-04-25 07: 25: 58Z INFO AgentProcess] RuntimeInformation: Microsoft Windows 6.3.9600. [2017-04-25 07: 25: 58Z INFO AgentProcess] Version: 2.111.1

Here is an excerpt from the magazines:

017-04-25T07: 28: 14.1561247Z ## [section] Start: Publish Artifact: fall 2017-04-25T07: 28: 14.1561247Z =================== =================================================== ==================================================== ========== 2017-04-25T07: 28: 14.1561247Z Task: Publish the artifact assembly 2017-04-25T07: 28: 14.1561247Z Description: Publish the artifact assembly to the server or file share 2017-04- 25T07: 28: 14.1561247Z Version: 1.0.40 2017-04-25T07: 28: 14.1561247Z Author: Microsoft Corporation 2017-04-25T07: 28: 14.1561247Z Help: Additional information 2017-04-25T07: 28: 14.1561247Z = =================================================== ========================================= ========== ============================= 2017-04-25T07: 28: 15.0779877Z ## [section] Async Command Started : Download artifact 2017-04-25T07: 28: 15.0779877Z Download 8 files 2017-04-25T07: 28: 20.1111153Z General file: 8 ---- Processed file: 7 (87%) 2017-04-25T07: 28: 30.2611071Z General file: 8 ---- Processed file: 7 (87%) 2017-04-25T07: 28: 35.3611234Z Could not load 'E: \ Build \ agents \ scully_work \ 4 \ a \ Project.Name.Web .zip 'due to an error that occurred while sending the request. '. 2017-04-25T07: 28: 35.3611234Z System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.Http.WinHttpException: the connection to the server was aborted abnormally in System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (task task) in System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebugger task) .Http.WinHttpHandler.d__101.MoveNext () --- The end of the internal check of the exception stack --- in Microsoft.VisualStudio.Services.Common.VssHttpRetryMessageHandler.d__3.MoveNext () --- The end of the stack trace from the previous place where it was selected an exception is in System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (task task) in System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (Task task) in System.Net.Http.HttpClient.d__58 stack traces from the previous place where the exception was thrown --- in System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (task task) in System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (Task task) in System.Runtime.funter. 1.ConfiguredTaskAwaiter.GetResult() at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<SendAsync>d__45.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.ConfiguredTaskAwaitable MoveNext () --- End Trace stack stacks from the previous place where the exception was thrown --- in System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (task task) in System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (Task task) in System.Runter.viceserterwervices.eruntervices. 1.GetResult() at Microsoft.VisualStudio.Services.FileContainer.Client.FileContainerHttpClient.<UploadFileAsync>d__16.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.VisualStudio.Services.Agent.Worker.Build.FileContainerServer.<UploadAsync>d__14.MoveNext() 2017-04-25T07:28:35.3611234Z Detail upload trace for file that fail to upload: drop/Project.Name.Web.zip 2017-04-25T07:28:35.3611234Z Begin chunking upload file 'drop/Project.Name.Web.zip', chunk size '4194304 Bytes', total chunks '3'. 2017-04-25T07:28:35.3611234Z Attempt '1' for uploading chunk '1' of file 'drop/Project.Name.Web.zip'. 2017-04-25T07:28:35.3611234Z Generate new HttpRequest for uploading file 'drop/Project.Name.Web.zip', chunk '1' of '3'. 2017-04-25T07:28:35.3611234Z Start uploading file 'drop/Project.Name.Web.zip' to server, chunk '1'. 2017-04-25T07:28:35.3611234Z Chunk '1' attempt '1' of file 'drop/Project.Name.Web.zip' fail to send request to server. Error: System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.Http.WinHttpException: The connection with the server was terminated abnormally at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Net.Http.WinHttpHandler.<StartRequest>d__101.MoveNext() --- End of inner exception stack trace --- at Microsoft.VisualStudio.Services.Common.VssHttpRetryMessageHandler.<SendAsync>d__3.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Net.Http.HttpClient.<FinishSendAsync>d__58.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.ConfiguredTaskAwaitable 1.GetResult() at Microsoft.VisualStudio.Services.FileContainer.Client.FileContainerHttpClient.<UploadFileAsync>d__16.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.VisualStudio.Services.Agent.Worker.Build.FileContainerServer.<UploadAsync>d__14.MoveNext() 2017-04-25T07:28:35.3611234Z Detail upload trace for file that fail to upload: drop/Project.Name.Web.zip 2017-04-25T07:28:35.3611234Z Begin chunking upload file 'drop/Project.Name.Web.zip', chunk size '4194304 Bytes', total chunks '3'. 2017-04-25T07:28:35.3611234Z Attempt '1' for uploading chunk '1' of file 'drop/Project.Name.Web.zip'. 2017-04-25T07:28:35.3611234Z Generate new HttpRequest for uploading file 'drop/Project.Name.Web.zip', chunk '1' of '3'. 2017-04-25T07:28:35.3611234Z Start uploading file 'drop/Project.Name.Web.zip' to server, chunk '1'. 2017-04-25T07:28:35.3611234Z Chunk '1' attempt '1' of file 'drop/Project.Name.Web.zip' fail to send request to server. Error: System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.Http.WinHttpException: The connection with the server was terminated abnormally at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Net.Http.WinHttpHandler.<StartRequest>d__101.MoveNext() --- End of inner exception stack trace --- at Microsoft.VisualStudio.Services.Common.VssHttpRetryMessageHandler.<SendAsync>d__3.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Net.Http.HttpClient.<FinishSendAsync>d__58.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.ConfiguredTaskAwaitable 1.GetResult() at Microsoft.VisualStudio.Services.FileContainer.Client.FileContainerHttpClient.<UploadFileAsync>d__16.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.VisualStudio.Services.Agent.Worker.Build.FileContainerServer.<UploadAsync>d__14.MoveNext() 2017-04-25T07:28:35.3611234Z Detail upload trace for file that fail to upload: drop/Project.Name.Web.zip 2017-04-25T07:28:35.3611234Z Begin chunking upload file 'drop/Project.Name.Web.zip', chunk size '4194304 Bytes', total chunks '3'. 2017-04-25T07:28:35.3611234Z Attempt '1' for uploading chunk '1' of file 'drop/Project.Name.Web.zip'. 2017-04-25T07:28:35.3611234Z Generate new HttpRequest for uploading file 'drop/Project.Name.Web.zip', chunk '1' of '3'. 2017-04-25T07:28:35.3611234Z Start uploading file 'drop/Project.Name.Web.zip' to server, chunk '1'. 2017-04-25T07:28:35.3611234Z Chunk '1' attempt '1' of file 'drop/Project.Name.Web.zip' fail to send request to server. Error: System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.Http.WinHttpException: The connection with the server was terminated abnormally at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Net.Http.WinHttpHandler.<StartRequest>d__101.MoveNext() --- End of inner exception stack trace --- at Microsoft.VisualStudio.Services.Common.VssHttpRetryMessageHandler.<SendAsync>d__3.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Net.Http.HttpClient.<FinishSendAsync>d__58.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.ConfiguredTaskAwaitable 1.ConfiguredTaskAwaiter.GetResult () in Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.d__45.MoveNext where (last place) --- in System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (task task) in System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (Task task) in System.Runtime.CompilerServices.ConfigureTask.Document. VisualStudio.Services.FileContainer.Client.FileContainerHttpClient.d__17.MoveNext (). 2017-04-25T07: 28: 35.3611234Z Failure of 5.16 seconds before attempting "2" chunk '1'

Update: The agent has been updated to the latest version, and the problem still exists:

[2017-04-25 08: 33: 04Z INFO AgentProcess] Agent created for Windows - win7-x64. [2017-04-25 08: 33: 04Z INFO AgentProcess] RuntimeInformation: Microsoft Windows 6.3.9600. [2017-04-25 08: 33: 04Z INFO AgentProcess] Version: 2.115.0

Answer 1

This usually means that there are network problems somewhere between your agent and the service where you get interrupts.

Starting with agent 2.x, we are actively trying to work with network conditions. We break each file into pieces (4 MB), open an HTTP stream and pass this fragment of the file to the service. Each fragment of the file is set for a long time (10 minutes)

We have attempted to retry for retrying.

Backoff 5.16 seconds before attempt '2' chunk '1'

Then, if we still cannot load the piece, even with the help of attempts, we put this file in the repeat queue for another pass.

So, as you can see, we are trying very hard.

The more network interrupts and the larger the file, the greater the chances. In addition, if the download speed slows down a lot, it can lead to crashes during block loading (4 MB in 10 minutes - a long time), but in your case it looks like the stream / network is interrupted.

You must check the network. Something is interrupted between your machine and the service.

Another experiment that needs to be tried is a private agent on a private virtual virtual machine (e.g. azure). This gives you a faster cloud network at boot time, as well as private agent management (install the right software, incremental source and packaging, etc.).

Do you see the repetitions that I am describing? I also forward this to our developer agent. I will edit if anything else is added.