Skip to content

Conversation

jonathanpeppers
Copy link
Member

@jonathanpeppers jonathanpeppers commented Oct 22, 2020

Deploying a project that uses two $(RuntimeIdentifiers) will fail on
our new fast deploy system with:

Xamarin.Android.Common.Debugging.targets(589,5): Access to the path 'obj/Debug/net5.0-android/android/assets/' is denied.
    at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String path, OpenFlags flags, Int32 mode)
    at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options)
    at Xamarin.Android.Tasks.FastDeploy.DeployFileWithFastDevTool(ITaskItem file, String toolPath, String overridePath, Byte[] buffer, Byte[] compressed, LZ4Level lz4level) in /Users/builder/azdo/_work/18/s/xamarin-android/external/monodroid/tools/msbuild/Tasks/FastDeploy.cs:line 579
    at Xamarin.Android.Tasks.FastDeploy.DeployFastDevFiles(String toolPath, String overridePath) in /Users/builder/azdo/_work/18/s/xamarin-android/external/monodroid/tools/msbuild/Tasks/FastDeploy.cs:line 491
    at Xamarin.Android.Tasks.FastDeploy.RunTaskAsync() in /Users/builder/azdo/_work/18/s/xamarin-android/external/monodroid/tools/msbuild/Tasks/FastDeploy.cs:line 175

Reviewing the log, there are some weird things happening. Some
incorrect @(_ResolvedSymbols) items are passed to the
<FastDeploy/> MSBuild task:

_ResolvedSymbols
    obj/Debug/net5.0-android/android/assets/
    obj/Debug/net5.0-android/android/assets/

This is caused by the contents of the @(ResolvedFileToPublish) item
group that is populated by the ComputeFilesToPublish MSBuild target
in the dotnet/sdk:

ResolvedFileToPublish
    obj/Debug/net5.0-android/HelloForms.dll
    obj/Debug/net5.0-android/android.21-arm64/HelloForms.pdb
    obj/Debug/net5.0-android/android.21-x86/HelloForms.pdb

The HelloForms.pdb files listed here do not actually exist, the
path should be: obj/Debug/net5.0-android/HelloForms.pdb.
This is caused by the way our "outer" build has no $(RuntimeIdentifer),
while we have an "inner" build that runs per-RID.

The "inner" build needs to fix up the @(_DebugSymbolsIntermediatePath)
item group to account for this:

https://github.com/dotnet/sdk/blob/ee455863aa6ddf13108bb54b37a5a47fb12fe39e/src/Tasks/Microsoft.NET.Build.Tasks/targets/Microsoft.NET.Publish.targets#L522-L555

We were already fixing up @(IntermediateAssembly).

When HelloForms.dll did not map to a single HelloForms.pdb, the
%(DestinationSubPath) item metadata is not added by the
<ProcessAssemblies/> MSBuild task in Xamarin.Android.

And so the following item transform generated an incorrect result:

<_ResolvedSymbols Include="@(ResolvedSymbols->'$(MonoAndroidIntermediateAssemblyDir)%(DestinationSubPath)')" />

To fix this:

  • @(_ResolvedSymbols) should check if %(DestinationSubPath) is blank.
  • Call ->Distinct() on input to the <ProcessAssemblies/> MSBuild task.
  • Patch up the @(_DebugSymbolsIntermediatePath) item group, so that
    it uses the proper value for @(IntermediateOutputPath) of the
    outer build.

See dotnet/msbuild for the definition of @(_DebugSymbolsIntermediatePath): https://github.com/dotnet/msbuild/blob/65d31a0151b072b910d9ea16e152265d467e3239/src/Tasks/Microsoft.Common.CurrentVersion.targets#L374

After these changes, the @(ResolvedFileToPublish) item group appears
correct now:

ResolvedFileToPublish
    obj/Debug/net5.0-android/HelloForms.dll
    obj/Debug/net5.0-android/HelloForms.pdb

HelloForms.pdb now makes it to the .apk file as well.

Other changes:

  • Updated a test to look for UnnamedProject.pdb during a multiple
    RID build.
  • I removed the Removing duplicate: ... log message, as it was
    printing hundreds of log messages that aren't particularly useful.

@jonathanpeppers jonathanpeppers force-pushed the dotnet-fix-symbol-files branch from 7ed6c31 to 4d34ee0 Compare October 23, 2020 16:15
Deploying a project that uses two `$(RuntimeIdentifiers)` will fail on
our new fast deploy system with:

    Xamarin.Android.Common.Debugging.targets(589,5): Access to the path 'obj/Debug/net5.0-android/android/assets/' is denied.
        at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String path, OpenFlags flags, Int32 mode)
        at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options)
        at Xamarin.Android.Tasks.FastDeploy.DeployFileWithFastDevTool(ITaskItem file, String toolPath, String overridePath, Byte[] buffer, Byte[] compressed, LZ4Level lz4level) in /Users/builder/azdo/_work/18/s/xamarin-android/external/monodroid/tools/msbuild/Tasks/FastDeploy.cs:line 579
        at Xamarin.Android.Tasks.FastDeploy.DeployFastDevFiles(String toolPath, String overridePath) in /Users/builder/azdo/_work/18/s/xamarin-android/external/monodroid/tools/msbuild/Tasks/FastDeploy.cs:line 491
        at Xamarin.Android.Tasks.FastDeploy.RunTaskAsync() in /Users/builder/azdo/_work/18/s/xamarin-android/external/monodroid/tools/msbuild/Tasks/FastDeploy.cs:line 175

Reviewing the log, there are some weird things happening. Some
incorrect `@(_ResolvedSymbols)` items are passed to the
`<FastDeploy/>` MSBuild task:

    _ResolvedSymbols
        obj/Debug/net5.0-android/android/assets/
        obj/Debug/net5.0-android/android/assets/

This is caused by the contents of the `@(ResolvedFileToPublish)` item
group:

    ResolvedFileToPublish
        obj/Debug/net5.0-android/HelloForms.dll
        obj/Debug/net5.0-android/android.21-arm64/HelloForms.pdb
        obj/Debug/net5.0-android/android.21-x86/HelloForms.pdb

Since `HelloForms.dll` doesn't map to a single `HelloForms.pdb`, the
`%(DestinationSubPath)` item metadata is not added by the
`<ProcessAssemblies/>` MSBuild task.

And so the following item transform generates an incorrect result:

    <_ResolvedSymbols Include="@(ResolvedSymbols->'$(MonoAndroidIntermediateAssemblyDir)%(DestinationSubPath)')" />

To fix this:

* `@(_ResolvedSymbols)` should check if `%(DestinationSubPath)` is blank.
* Call `->Distinct()` on input to the `<ProcessAssemblies/>` MSBuild task.
* Patch up the `@(_DebugSymbolsIntermediatePath)` item group, so that
  it uses the proper value for `@(IntermediateOutputPath)` of the
  outer build.

See: https://github.com/dotnet/msbuild/blob/65d31a0151b072b910d9ea16e152265d467e3239/src/Tasks/Microsoft.Common.CurrentVersion.targets#L374

After these changes, the `@(ResolvedFileToPublish)` item group appears
correct now:

    ResolvedFileToPublish
        obj/Debug/net5.0-android/HelloForms.dll
        obj/Debug/net5.0-android/HelloForms.pdb

`HelloForms.pdb` now makes it to the `.apk` file as well.

Other changes:

* Updated a test to look for `UnnamedProject.pdb` during a multiple
  RID build.
* I removed the `Removing duplicate: ...` log message, as it was
  printing hundreds of log messages that aren't particularly useful.
@jonathanpeppers jonathanpeppers force-pushed the dotnet-fix-symbol-files branch from 4d34ee0 to 6434fc4 Compare October 27, 2020 01:30
@jonathanpeppers jonathanpeppers marked this pull request as ready for review October 27, 2020 13:31
<ProcessAssemblies
InputAssemblies="@(_ResolvedAssemblyFiles)"
ResolvedSymbols="@(_ResolvedSymbolFiles)"
InputAssemblies="@(_ResolvedAssemblyFiles->Distinct())"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is @(_ResolvedAssemblyFiles->Distinct()) potentially stored in a file anywhere? Presumably this expression controls @(_ProcessedShrunkAssemblies)? .Distinct() may change the ordering, and I don't know if that ordering is consistent from one process run to the next, so if this is "somehow" stored in a file anywhere, it may result in bad dependency tracking.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

->Distinct() in MSBuild preserves ordering, I asked Rainer about this before:

https://github.com/dotnet/msbuild/blob/6605cdf6e969fa670bab736a0aca415da2080cdb/src/Build/Evaluation/Expander.cs#L2517-L2555

@(_ResolvedAssemblyFiles) won't be used anywhere else, but the outputs of <ProcessAssemblies/> will be.

@jonathanpeppers
Copy link
Member Author

jonathanpeppers commented Oct 28, 2020

@jonpryor I put several more details in the PR text above we can use in the final commit message. If I end up making another change, I'll copy it to the commit message.

@jonpryor jonpryor merged commit 57c5a5f into dotnet:master Oct 28, 2020
@jonathanpeppers jonathanpeppers deleted the dotnet-fix-symbol-files branch October 28, 2020 19:11
jonathanpeppers added a commit to dotnet/maui-samples that referenced this pull request Oct 29, 2020
Context: dotnet/android#5236

Fixes problem with Fast Deployment on Android caused by incorrect
paths in `@(ResolvedFilesToPublish)` from the dotnet/sdk.

Hoping it might help the random iOS failure, too? We saw things in the
build log such as:

    asset directory did not exist: /Users/runner/work/1/s/HelloAndroid/obj/Debug/net5.0-android/assets/ (TaskId:253)
    Executing link --manifest /Users/runner/work/1/s/HelloAndroid/obj/Debug/net5.0-android/android/manifest/AndroidManifest.xml [...]
    ILLINK : error MT2301: The linker step 'Setup' failed during processing. [/Users/runner/work/1/s/HelloiOS/HelloiOS.csproj]

`net6-samples.sln` will build projects in parallel, so we're not sure
if the Android log message is affecting the iOS build or not.
jonathanpeppers added a commit to dotnet/maui-samples that referenced this pull request Oct 30, 2020
Context: dotnet/android#5236

Fixes problem with Fast Deployment on Android when building a
project with multiple `$(RuntimeIdentifiers)`. There were incorrect
paths in `@(ResolvedFilesToPublish)` coming from the dotnet/sdk.
@github-actions github-actions bot locked and limited conversation to collaborators Jan 24, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants