Skip to content
This repository was archived by the owner on Sep 4, 2024. It is now read-only.
This repository was archived by the owner on Sep 4, 2024. It is now read-only.

Kudu suddenly taking a long time to deploy to temp #1693

Closed

Description

@AlanParr

Normally, from pushing to build complete takes about 5 minutes. In the last week or so, it is taking between 15 and 30 minutes.
Initially, I thought it was because I had overridden MSBUILD_PATH to use the C#6 msbuild, however I have removed this from one site and the build still takes the same amount of time. Reviewing the Kudu trace log shows it is taking nearly 20 minutes to copy the build output to temp, the solution isn't that big so it really shouldn't take very long.

Relevant snippet of Kudu Trace:

2015-09-14T11:40:23 Copying all files to temporary location below for package/publish:
2015-09-14T11:40:23 D:\local\Temp\8d2bcf7a334374d.
2015-09-14T11:57:47 KuduSync.NET from: 'D:\local\Temp\8d2bcf7a334374d' to: 'D:\home\site\wwwroot'

Activity

  1. davidebbo commented on Sep 14, 2015

    @davidebbo
    Member

    Update: please read if you are running into this issue

    I created a new entry in the Known Issues page to capture everything that someone needs to know about this issue: https://github.com/projectkudu/kudu/wiki/Known-issues


    Previous content:

    We deployed Kudu 47.40908.1797.0 last Tuesday (you can see version by going to root of Kudu). Try setting KUDU_EXTENSION_VERSION=47.40826.1772 in the Azure App Settings to go back to the previous build (verify that setting is effective by going to Kudu root again).

    This way we can isolate whether the issue is related to the new Kudu, or possibly some unrelated environmental thing.

  2. AlanParr commented on Sep 14, 2015

    @AlanParr
    Author

    I've just done a deployment and it's back down to 5 minutes using Kudu 47.40826.1772 so looks like it is most likely related to that.

    Are there release notes so I can see what I am missing out on by sticking back a version?

  3. davidebbo commented on Sep 14, 2015

    @davidebbo
    Member

    Hmmm, that's concerning, as nothing fundamental has changed. I guess the one thing is the switch to the new msbuild, which also implies the new Roslyn-based compiler, etc...

    Can you try going back to the newest Kudu, but set MSBUILD_PATH=D:\Windows\Microsoft.NET\Framework\v4.0.30319\MSBuild.exe. That would let us further isolate.

  4. davidebbo commented on Sep 14, 2015

    @davidebbo
    Member

    Also, you can try manually building clean from Kudu Console, comparing D:\Windows\Microsoft.NET\Framework\v4.0.30319\MSBuild.exe and D:\Program Files (x86)\MSBuild\14.0\Bin\MSBuild.exe

  5. AlanParr commented on Sep 14, 2015

    @AlanParr
    Author

    New Kudu with old msbuild is 5 minutes also.

  6. davidebbo commented on Sep 14, 2015

    @davidebbo
    Member

    It's concerning that the new msbuild/csc could be that much slower for your solution. Is there anything unusual about it? Can you describe what it's using? e.g. is it MVC, Web API, etc... and what version?

    Also, can you share your site name, either directly or indirectly? This may help us investigate.

  7. AlanParr commented on Sep 14, 2015

    @AlanParr
    Author

    It's just a bog-standard MVC4 app targeting .Net 4.5.
    I'm using VS2015 which would be using msbuild 14 and am experiencing no issues locally.

    Site is cms-staging-weu-site.
    I do have a custom deploy.cmd, although from what I can remember, the only thing I changed is setting the solution path.

  8. davidebbo commented on Sep 14, 2015

    @davidebbo
    Member

    Is the repo public? If not, any chance you can give me access to the repo such that I could try deploying it on my own test site?

    We have test repos with all kind of variations of MVC and framework, but we have not seen this kind of extreme slow perf.

  9. AlanParr commented on Sep 15, 2015

    @AlanParr
    Author

    Rather oddly, I thought I'd try reverting back to using the default msbuild today to check out the logs on a long build, and it seems to be fine now.
    Unfortunately, I didn't copy the full trace from when it was going slow before, although the fact that all the time was spent on copying to temp makes me think it was something post-build, so not necessarily msbuild related, perhaps that was a red herring.
    I'll continue doing deployments today and see if it occurs again.

    Forget the above, the problem is back again. Will try a few things and repost.

    The repo isn't public as it is a commercial application and I'm not sure our account levels at our source hosts would allow me to grant access to an external user. However, if you've got a private repo you could grant me temporary access to, I would be happy to push to that.

  10. AlanParr commented on Sep 15, 2015

    @AlanParr
    Author

    Some further testing:
    It looks like msbuild is running with really high memory usage, see attached screenshot of process explorer.

    This is occurring across 3 different site instances, located in West Europe, East US, and East Asia, so it isn't localised.

    Two of the deployments that have taken a long time are:

    • b36f0323a4f63bf92f88db29cfcb967ca2a924b8, took approx 50 minutes.
    • 230b443e6326ffbc411b5104ec61474b794b3657, in progress, up to 40 minutes so far. Build is 30, copying to temp has been in progress for 10 minutes.

    Example of a good deployment:

    • 53a699ab048989abf3025c9fd9046e374e403501, took approx 3 minutes.
    • I've tried not overriding msbuild, which was okay first thing this morning but after a few deployments everything reverted to long builds again.
    • Overriding msbuild with D:\Program Files (x86)\MSBuild\14.0\Bin\MSBuild.exe, which I understand is now the default anyway, results in long builds.
    • Overriding with the old msbuild works fine again.

    Comparing deployment logs between deployments 230b and 53a6, when overriding msbuild with msbuild 14, it outputs warnings, which didn't occur in the good deployment. There doesn't seem to be any single item that takes longer, every item just seems to take longer.

    kudu-procexp

  11. AlanParr commented on Sep 15, 2015

    @AlanParr
    Author

    It is worth noting that my repository also contains another MVC4 site and an asmx web service. These are deployed in to their own Web Apps on the same Shared server. When the main site is misbehaving, they take about 20 minutes to build. When the main site is using the framework msbuild, these are fine too without having their msbuilds overridden, so I suspect their speed problems are just the result of resource starvation due to whatever problem the main site is having.

  12. davidebbo commented on Sep 15, 2015

    @davidebbo
    Member

    I see. So the theory here is that the new Roslyn compiler (VBCSCompiler.exe) is not slower, but it eats up way more memory, causing the machine to thrash and be super slow if it doesn't have enough.

    Note that you can monitor the CPU and memory usage of your overall VM over the last 24 hours using https://portal.azure.com/:

    • Go to your Web App
    • Click on App Service Plan
    • Click on quotas

    This could confirm the theory if the slow builds correlate with high usage.

  13. AlanParr commented on Sep 15, 2015

    @AlanParr
    Author

    The CPU usage on the app service plan increased from 50% at 9am, peaking at 98% at 10am, and dropping down slowly after that. This approximately correlates with the big 50 minute build at 10 and a number of less slow, but still slower than they should be, builds either side of this.

    On an additional likely related note, I've just had one deployment fail with the following error:

    D:\Windows\Microsoft.NET\Framework\v4.0.30319\Microsoft.CSharp.targets(316,9): error MSB6006: "Csc.exe" exited with code -1. [D:\home\site\repository\Product\Production\Model\Model.csproj]
    Failed exitCode=-1, command="D:\Windows\Microsoft.NET\Framework\v4.0.30319\MSBuild.exe" "D:\home\site\repository\Product/Production/MVC/MVC.csproj" /nologo /verbosity:m /t:Build /t:pipelinePreDeployCopyAllFilesToOneFolder /p:_PackageTempDir="D:\local\Temp\8d2bdd3aecbffeb";AutoParameterizationWebConfigConnectionStrings=false /p:SolutionDir="D:\home\site\repository\Product" /p:Environment=staging;Configuration=staging
    Command 'starter.cmd deploy.cmd' was aborted due to no output nor CPU activity for 61 seconds. You can increase the SCM_COMMAND_IDLE_TIMEOUT app setting (or WEBJOBS_IDLE_TIMEOUT if this is a WebJob) if needed.\r\nstarter.cmd deploy.cmd
    One or more errors occurred.
    Command 'starter.cmd deploy.cmd' was aborted due to no output nor CPU activity for 61 seconds. You can increase the SCM_COMMAND_IDLE_TIMEOUT app setting (or WEBJOBS_IDLE_TIMEOUT if this is a WebJob) if needed.\r\nstarter.cmd deploy.cmd

  14. davidebbo commented on Sep 15, 2015

    @davidebbo
    Member

    This happened because it was going so slow that Kudu failed to detect activity for 60 seconds and aborted the build.

    Well, I think we understand the issue now, and your best bet is to use the pre-Roslyn compiler.

    We'll try to bring up the memory usage to the Roslyn team, but it's unlikely to be something they can just fix. They have a completely different architecture from before, and presumably it needs more memory to do its thing :(

  15. AlanParr commented on Sep 15, 2015

    @AlanParr
    Author

    I've just compared the csproj for the main site where I am having troubles and the MVC Rest site which is fine, below are the main differences that I can see.

    • Project ToolsVersion is 12.0 in main site, 4.0 in MVC Rest site
    • The following item is present in the main site but not the MVC Rest site:
    <Import Project="$(MSBuildExtensionsPath32)\Microsoft\VisualStudio\v$(VisualStudioVersion)\TypeScript\Microsoft.TypeScript.targets" Condition="Exists('$(MSBuildExtensionsPath32)\Microsoft\VisualStudio\v$(VisualStudioVersion)\TypeScript\Microsoft.TypeScript.targets')" />

    I am assuming that Visual Studio is not installed on the VM, so this could be our culprit? Any idea what a suitable alternative might be?

  16. 34 remaining items

  17. davidebbo commented on Sep 24, 2015

    @davidebbo
    Member

    To be more specific, the fix we will make is to modify kuduscript to always pass /p:UseSharedCompilation=false to the msbuild command line.

  18. skullycheese commented on Nov 2, 2015

    @skullycheese

    how exactly do I apply the fix for this? I tried adding /p:UseSharedCompilation=false as one of the paramters for msbuild in the deploy.cmd file but the deploy still takes about 20 minutes.

  19. skullycheese commented on Nov 2, 2015

    @skullycheese

    I've also added it as an app setting through the azure web app as well as setting the msbuild version back.. still having issues with the length of the deployment.. is there anything else I should be looking for?

  20. davidebbo commented on Nov 2, 2015

    @davidebbo
    Member

    Just look at your deployment output to see if the switch is getting passed to msbuild. If it is, then you are likely dealing with an unrelated issue.

  21. skullycheese commented on Nov 2, 2015

    @skullycheese

    is there anything else that I should be looking for? I'm not really sure where to start.. it is an Orchard project which is a pretty big solution.. do you think that could be why it is taking this long?

  22. davidebbo commented on Nov 2, 2015

    @davidebbo
    Member

    Did you verify that /p:UseSharedCompilation=false was being passed? If so, please open a separate issue with more details about your situation. e.g. how long does it take? Did it use to be faster and then something happen? Where is the time spent in your deployment log? etc...

  23. skullycheese commented on Nov 2, 2015

    @skullycheese

    I did, okay I will open a new issue now.

  24. etm-admin commented on Aug 12, 2016

    @etm-admin

    Is this issue resolved can we get rid of the /p:UseSharedCompilation=false?

  25. davidebbo commented on Aug 12, 2016

    @davidebbo
    Member

    Yes, you should not have to pass this flag now.

  26. pacodelacruz commented on Dec 23, 2016

    @pacodelacruz

    Hi @davidebbo, A similar behaviour is happening on our customer environment.
    Kudu Extension Version is 59.51212.2600.

    We've been advised by Azure Support that Site Precompilation should fix the problem. Interestingly they provided instructions for .NET 3.0?. We are deploying ASP.NET Web Apis (.NET Framework 4.5.2)

    We are going to try pre-compilation, however, the customer would like to get a definitive answer from MS regarding this. Based on what you've seen, is pre-compilation the best approach? Is p:UseSharedCompilation=false still a valid approach?

    Thanks,

  27. davidebbo commented on Dec 31, 2016

    @davidebbo
    Member

    @pacodelacruz that is an old issue that is no longer relevant, so my guess is that you are running into an unrelated condition.

  28. pacodelacruz commented on Jan 9, 2017

    @pacodelacruz

    Thanks @davidebbo. According to MS support, the problem is that vbcscompiler.exe is taking more than 60 mins to complete, and it is also occurring randomly, it does not happen on every instance. We'll update the thread if we find out the root cause.

  29. added a commit that references this issue on Oct 30, 2024
    9a4c81d
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions

    ⌂ 换网址