You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository was archived by the owner on Sep 4, 2024. It is now read-only.
Repository navigation
This repository was archived by the owner on Sep 4, 2024. It is now read-only.
Kudu suddenly taking a long time to deploy to temp #1693
Normally, from pushing to build complete takes about 5 minutes. In the last week or so, it is taking between 15 and 30 minutes.
Initially, I thought it was because I had overridden MSBUILD_PATH to use the C#6 msbuild, however I have removed this from one site and the build still takes the same amount of time. Reviewing the Kudu trace log shows it is taking nearly 20 minutes to copy the build output to temp, the solution isn't that big so it really shouldn't take very long.
Relevant snippet of Kudu Trace:
2015-09-14T11:40:23 Copying all files to temporary location below for package/publish:
2015-09-14T11:40:23 D:\local\Temp\8d2bcf7a334374d.
2015-09-14T11:57:47 KuduSync.NET from: 'D:\local\Temp\8d2bcf7a334374d' to: 'D:\home\site\wwwroot'
We deployed Kudu 47.40908.1797.0 last Tuesday (you can see version by going to root of Kudu). Try setting KUDU_EXTENSION_VERSION=47.40826.1772 in the Azure App Settings to go back to the previous build (verify that setting is effective by going to Kudu root again).
This way we can isolate whether the issue is related to the new Kudu, or possibly some unrelated environmental thing.
Hmmm, that's concerning, as nothing fundamental has changed. I guess the one thing is the switch to the new msbuild, which also implies the new Roslyn-based compiler, etc...
Can you try going back to the newest Kudu, but set MSBUILD_PATH=D:\Windows\Microsoft.NET\Framework\v4.0.30319\MSBuild.exe. That would let us further isolate.
Also, you can try manually building clean from Kudu Console, comparing D:\Windows\Microsoft.NET\Framework\v4.0.30319\MSBuild.exe and D:\Program Files (x86)\MSBuild\14.0\Bin\MSBuild.exe
It's concerning that the new msbuild/csc could be that much slower for your solution. Is there anything unusual about it? Can you describe what it's using? e.g. is it MVC, Web API, etc... and what version?
Also, can you share your site name, either directly or indirectly? This may help us investigate.
Rather oddly, I thought I'd try reverting back to using the default msbuild today to check out the logs on a long build, and it seems to be fine now.
Unfortunately, I didn't copy the full trace from when it was going slow before, although the fact that all the time was spent on copying to temp makes me think it was something post-build, so not necessarily msbuild related, perhaps that was a red herring.
I'll continue doing deployments today and see if it occurs again.
Forget the above, the problem is back again. Will try a few things and repost.
The repo isn't public as it is a commercial application and I'm not sure our account levels at our source hosts would allow me to grant access to an external user. However, if you've got a private repo you could grant me temporary access to, I would be happy to push to that.
Some further testing:
It looks like msbuild is running with really high memory usage, see attached screenshot of process explorer.
This is occurring across 3 different site instances, located in West Europe, East US, and East Asia, so it isn't localised.
Two of the deployments that have taken a long time are:
b36f0323a4f63bf92f88db29cfcb967ca2a924b8, took approx 50 minutes.
230b443e6326ffbc411b5104ec61474b794b3657, in progress, up to 40 minutes so far. Build is 30, copying to temp has been in progress for 10 minutes.
Example of a good deployment:
53a699ab048989abf3025c9fd9046e374e403501, took approx 3 minutes.
I've tried not overriding msbuild, which was okay first thing this morning but after a few deployments everything reverted to long builds again.
Overriding msbuild with D:\Program Files (x86)\MSBuild\14.0\Bin\MSBuild.exe, which I understand is now the default anyway, results in long builds.
Overriding with the old msbuild works fine again.
Comparing deployment logs between deployments 230b and 53a6, when overriding msbuild with msbuild 14, it outputs warnings, which didn't occur in the good deployment. There doesn't seem to be any single item that takes longer, every item just seems to take longer.
It is worth noting that my repository also contains another MVC4 site and an asmx web service. These are deployed in to their own Web Apps on the same Shared server. When the main site is misbehaving, they take about 20 minutes to build. When the main site is using the framework msbuild, these are fine too without having their msbuilds overridden, so I suspect their speed problems are just the result of resource starvation due to whatever problem the main site is having.
I see. So the theory here is that the new Roslyn compiler (VBCSCompiler.exe) is not slower, but it eats up way more memory, causing the machine to thrash and be super slow if it doesn't have enough.
Note that you can monitor the CPU and memory usage of your overall VM over the last 24 hours using https://portal.azure.com/:
Go to your Web App
Click on App Service Plan
Click on quotas
This could confirm the theory if the slow builds correlate with high usage.
The CPU usage on the app service plan increased from 50% at 9am, peaking at 98% at 10am, and dropping down slowly after that. This approximately correlates with the big 50 minute build at 10 and a number of less slow, but still slower than they should be, builds either side of this.
On an additional likely related note, I've just had one deployment fail with the following error:
D:\Windows\Microsoft.NET\Framework\v4.0.30319\Microsoft.CSharp.targets(316,9): error MSB6006: "Csc.exe" exited with code -1. [D:\home\site\repository\Product\Production\Model\Model.csproj]
Failed exitCode=-1, command="D:\Windows\Microsoft.NET\Framework\v4.0.30319\MSBuild.exe" "D:\home\site\repository\Product/Production/MVC/MVC.csproj" /nologo /verbosity:m /t:Build /t:pipelinePreDeployCopyAllFilesToOneFolder /p:_PackageTempDir="D:\local\Temp\8d2bdd3aecbffeb";AutoParameterizationWebConfigConnectionStrings=false /p:SolutionDir="D:\home\site\repository\Product" /p:Environment=staging;Configuration=staging
Command 'starter.cmd deploy.cmd' was aborted due to no output nor CPU activity for 61 seconds. You can increase the SCM_COMMAND_IDLE_TIMEOUT app setting (or WEBJOBS_IDLE_TIMEOUT if this is a WebJob) if needed.\r\nstarter.cmd deploy.cmd
One or more errors occurred.
Command 'starter.cmd deploy.cmd' was aborted due to no output nor CPU activity for 61 seconds. You can increase the SCM_COMMAND_IDLE_TIMEOUT app setting (or WEBJOBS_IDLE_TIMEOUT if this is a WebJob) if needed.\r\nstarter.cmd deploy.cmd
This happened because it was going so slow that Kudu failed to detect activity for 60 seconds and aborted the build.
Well, I think we understand the issue now, and your best bet is to use the pre-Roslyn compiler.
We'll try to bring up the memory usage to the Roslyn team, but it's unlikely to be something they can just fix. They have a completely different architecture from before, and presumably it needs more memory to do its thing :(
I've just compared the csproj for the main site where I am having troubles and the MVC Rest site which is fine, below are the main differences that I can see.
Project ToolsVersion is 12.0 in main site, 4.0 in MVC Rest site
The following item is present in the main site but not the MVC Rest site:
how exactly do I apply the fix for this? I tried adding /p:UseSharedCompilation=false as one of the paramters for msbuild in the deploy.cmd file but the deploy still takes about 20 minutes.
I've also added it as an app setting through the azure web app as well as setting the msbuild version back.. still having issues with the length of the deployment.. is there anything else I should be looking for?
is there anything else that I should be looking for? I'm not really sure where to start.. it is an Orchard project which is a pretty big solution.. do you think that could be why it is taking this long?
Did you verify that /p:UseSharedCompilation=false was being passed? If so, please open a separate issue with more details about your situation. e.g. how long does it take? Did it use to be faster and then something happen? Where is the time spent in your deployment log? etc...
Hi @davidebbo, A similar behaviour is happening on our customer environment.
Kudu Extension Version is 59.51212.2600.
We've been advised by Azure Support that Site Precompilation should fix the problem. Interestingly they provided instructions for .NET 3.0?. We are deploying ASP.NET Web Apis (.NET Framework 4.5.2)
We are going to try pre-compilation, however, the customer would like to get a definitive answer from MS regarding this. Based on what you've seen, is pre-compilation the best approach? Is p:UseSharedCompilation=false still a valid approach?
Thanks @davidebbo. According to MS support, the problem is that vbcscompiler.exe is taking more than 60 mins to complete, and it is also occurring randomly, it does not happen on every instance. We'll update the thread if we find out the root cause.
Activity
davidebbo commentedon Sep 14, 2015
Update: please read if you are running into this issue
I created a new entry in the Known Issues page to capture everything that someone needs to know about this issue: https://github.com/projectkudu/kudu/wiki/Known-issues
Previous content:
We deployed Kudu 47.40908.1797.0 last Tuesday (you can see version by going to root of Kudu). Try setting
KUDU_EXTENSION_VERSION=47.40826.1772in the Azure App Settings to go back to the previous build (verify that setting is effective by going to Kudu root again).This way we can isolate whether the issue is related to the new Kudu, or possibly some unrelated environmental thing.
AlanParr commentedon Sep 14, 2015
I've just done a deployment and it's back down to 5 minutes using Kudu 47.40826.1772 so looks like it is most likely related to that.
Are there release notes so I can see what I am missing out on by sticking back a version?
davidebbo commentedon Sep 14, 2015
Hmmm, that's concerning, as nothing fundamental has changed. I guess the one thing is the switch to the new msbuild, which also implies the new Roslyn-based compiler, etc...
Can you try going back to the newest Kudu, but set
MSBUILD_PATH=D:\Windows\Microsoft.NET\Framework\v4.0.30319\MSBuild.exe. That would let us further isolate.davidebbo commentedon Sep 14, 2015
Also, you can try manually building clean from Kudu Console, comparing
D:\Windows\Microsoft.NET\Framework\v4.0.30319\MSBuild.exeandD:\Program Files (x86)\MSBuild\14.0\Bin\MSBuild.exeAlanParr commentedon Sep 14, 2015
New Kudu with old msbuild is 5 minutes also.
davidebbo commentedon Sep 14, 2015
It's concerning that the new msbuild/csc could be that much slower for your solution. Is there anything unusual about it? Can you describe what it's using? e.g. is it MVC, Web API, etc... and what version?
Also, can you share your site name, either directly or indirectly? This may help us investigate.
AlanParr commentedon Sep 14, 2015
It's just a bog-standard MVC4 app targeting .Net 4.5.
I'm using VS2015 which would be using msbuild 14 and am experiencing no issues locally.
Site is cms-staging-weu-site.
I do have a custom deploy.cmd, although from what I can remember, the only thing I changed is setting the solution path.
davidebbo commentedon Sep 14, 2015
Is the repo public? If not, any chance you can give me access to the repo such that I could try deploying it on my own test site?
We have test repos with all kind of variations of MVC and framework, but we have not seen this kind of extreme slow perf.
AlanParr commentedon Sep 15, 2015
Rather oddly, I thought I'd try reverting back to using the default msbuild today to check out the logs on a long build, and it seems to be fine now.Unfortunately, I didn't copy the full trace from when it was going slow before, although the fact that all the time was spent on copying to temp makes me think it was something post-build, so not necessarily msbuild related, perhaps that was a red herring.
I'll continue doing deployments today and see if it occurs again.
Forget the above, the problem is back again. Will try a few things and repost.
The repo isn't public as it is a commercial application and I'm not sure our account levels at our source hosts would allow me to grant access to an external user. However, if you've got a private repo you could grant me temporary access to, I would be happy to push to that.
AlanParr commentedon Sep 15, 2015
Some further testing:
It looks like msbuild is running with really high memory usage, see attached screenshot of process explorer.
This is occurring across 3 different site instances, located in West Europe, East US, and East Asia, so it isn't localised.
Two of the deployments that have taken a long time are:
Example of a good deployment:
Comparing deployment logs between deployments 230b and 53a6, when overriding msbuild with msbuild 14, it outputs warnings, which didn't occur in the good deployment. There doesn't seem to be any single item that takes longer, every item just seems to take longer.
AlanParr commentedon Sep 15, 2015
It is worth noting that my repository also contains another MVC4 site and an asmx web service. These are deployed in to their own Web Apps on the same Shared server. When the main site is misbehaving, they take about 20 minutes to build. When the main site is using the framework msbuild, these are fine too without having their msbuilds overridden, so I suspect their speed problems are just the result of resource starvation due to whatever problem the main site is having.
davidebbo commentedon Sep 15, 2015
I see. So the theory here is that the new Roslyn compiler (VBCSCompiler.exe) is not slower, but it eats up way more memory, causing the machine to thrash and be super slow if it doesn't have enough.
Note that you can monitor the CPU and memory usage of your overall VM over the last 24 hours using https://portal.azure.com/:
This could confirm the theory if the slow builds correlate with high usage.
AlanParr commentedon Sep 15, 2015
The CPU usage on the app service plan increased from 50% at 9am, peaking at 98% at 10am, and dropping down slowly after that. This approximately correlates with the big 50 minute build at 10 and a number of less slow, but still slower than they should be, builds either side of this.
On an additional likely related note, I've just had one deployment fail with the following error:
D:\Windows\Microsoft.NET\Framework\v4.0.30319\Microsoft.CSharp.targets(316,9): error MSB6006: "Csc.exe" exited with code -1. [D:\home\site\repository\Product\Production\Model\Model.csproj]
Failed exitCode=-1, command="D:\Windows\Microsoft.NET\Framework\v4.0.30319\MSBuild.exe" "D:\home\site\repository\Product/Production/MVC/MVC.csproj" /nologo /verbosity:m /t:Build /t:pipelinePreDeployCopyAllFilesToOneFolder /p:_PackageTempDir="D:\local\Temp\8d2bdd3aecbffeb";AutoParameterizationWebConfigConnectionStrings=false /p:SolutionDir="D:\home\site\repository\Product" /p:Environment=staging;Configuration=staging
Command 'starter.cmd deploy.cmd' was aborted due to no output nor CPU activity for 61 seconds. You can increase the SCM_COMMAND_IDLE_TIMEOUT app setting (or WEBJOBS_IDLE_TIMEOUT if this is a WebJob) if needed.\r\nstarter.cmd deploy.cmd
One or more errors occurred.
Command 'starter.cmd deploy.cmd' was aborted due to no output nor CPU activity for 61 seconds. You can increase the SCM_COMMAND_IDLE_TIMEOUT app setting (or WEBJOBS_IDLE_TIMEOUT if this is a WebJob) if needed.\r\nstarter.cmd deploy.cmd
davidebbo commentedon Sep 15, 2015
This happened because it was going so slow that Kudu failed to detect activity for 60 seconds and aborted the build.
Well, I think we understand the issue now, and your best bet is to use the pre-Roslyn compiler.
We'll try to bring up the memory usage to the Roslyn team, but it's unlikely to be something they can just fix. They have a completely different architecture from before, and presumably it needs more memory to do its thing :(
AlanParr commentedon Sep 15, 2015
I've just compared the csproj for the main site where I am having troubles and the MVC Rest site which is fine, below are the main differences that I can see.
I am assuming that Visual Studio is not installed on the VM, so this could be our culprit? Any idea what a suitable alternative might be?
34 remaining items
davidebbo commentedon Sep 24, 2015
To be more specific, the fix we will make is to modify kuduscript to always pass
/p:UseSharedCompilation=falseto the msbuild command line.skullycheese commentedon Nov 2, 2015
how exactly do I apply the fix for this? I tried adding /p:UseSharedCompilation=false as one of the paramters for msbuild in the deploy.cmd file but the deploy still takes about 20 minutes.
skullycheese commentedon Nov 2, 2015
I've also added it as an app setting through the azure web app as well as setting the msbuild version back.. still having issues with the length of the deployment.. is there anything else I should be looking for?
davidebbo commentedon Nov 2, 2015
Just look at your deployment output to see if the switch is getting passed to msbuild. If it is, then you are likely dealing with an unrelated issue.
skullycheese commentedon Nov 2, 2015
is there anything else that I should be looking for? I'm not really sure where to start.. it is an Orchard project which is a pretty big solution.. do you think that could be why it is taking this long?
davidebbo commentedon Nov 2, 2015
Did you verify that
/p:UseSharedCompilation=falsewas being passed? If so, please open a separate issue with more details about your situation. e.g. how long does it take? Did it use to be faster and then something happen? Where is the time spent in your deployment log? etc...skullycheese commentedon Nov 2, 2015
I did, okay I will open a new issue now.
etm-admin commentedon Aug 12, 2016
Is this issue resolved can we get rid of the
/p:UseSharedCompilation=false?davidebbo commentedon Aug 12, 2016
Yes, you should not have to pass this flag now.
pacodelacruz commentedon Dec 23, 2016
Hi @davidebbo, A similar behaviour is happening on our customer environment.
Kudu Extension Version is 59.51212.2600.
We've been advised by Azure Support that Site Precompilation should fix the problem. Interestingly they provided instructions for .NET 3.0?. We are deploying ASP.NET Web Apis (.NET Framework 4.5.2)
We are going to try pre-compilation, however, the customer would like to get a definitive answer from MS regarding this. Based on what you've seen, is pre-compilation the best approach? Is
p:UseSharedCompilation=falsestill a valid approach?Thanks,
davidebbo commentedon Dec 31, 2016
@pacodelacruz that is an old issue that is no longer relevant, so my guess is that you are running into an unrelated condition.
pacodelacruz commentedon Jan 9, 2017
Thanks @davidebbo. According to MS support, the problem is that vbcscompiler.exe is taking more than 60 mins to complete, and it is also occurring randomly, it does not happen on every instance. We'll update the thread if we find out the root cause.
pointing to new KuduScript that adds the switch UseSharedCompilation=…