Skip to content

Add more build artifacts and automatically publish to BAR. #1395

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 21 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions .vsts.pipelines/builds/ci-internal.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,3 +3,21 @@ jobs:
parameters:
windowsPoolName: NetCoreInternal-Pool
windowsQueueName: buildpool.windows.10.amd64.vs2017

- template: ../jobs/collect-artifacts.yml
parameters:
dependsOn:
- centos71
- debian9
- fedora30
- ubuntu1804
pool:
vmImage: vs2017-win2016

- template: /eng/common/templates/job/publish-build-assets.yml
parameters:
dependsOn:
- Collect_Artifacts
enablePublishBuildArtifacts: true
pool:
vmImage: vs2017-win2016
3 changes: 3 additions & 0 deletions .vsts.pipelines/builds/matrix.yml
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ jobs:
matrix:
Production: {}
Online: { type: Online }
Offline: { type: Offline }

- template: ../jobs/ci-linux.yml
parameters:
Expand All @@ -43,6 +44,8 @@ jobs:
parameters:
job: ubuntu1804
imageName: mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-f90bc20-20180320154721
matrix:
Offline: { type: Offline }

- template: ../jobs/ci-local.yml
parameters:
Expand Down
39 changes: 37 additions & 2 deletions .vsts.pipelines/jobs/ci-linux.yml
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,18 @@ jobs:
displayName: Build source-build
timeoutInMinutes: 150

- script: |
set -ex
df -h
$(docker.run) $(docker.drop.map) $(docker.src.map) $(docker.src.work) $(imageName) ./build.sh \
--collect-artifacts \
/p:ArtifactOutput=/drop/artifacts/source-build/ \
/bl
du -h $(rootDirectory) | sort -h | tail -n 50
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if it would be better to have in the build itself... this seems like a useful diagnosis artifact someone can give us when source-build doesn't work for them.

The paths also might also be clearer in that context--using the project props rather than writing out full names in multiple places.

displayName: Copy source-build artifacts
condition: always()
continueOnError: true

# Generate prebuilt burndown data
- script: |
set -ex
Expand Down Expand Up @@ -137,7 +149,8 @@ jobs:
-path './bin/aspnet-debug/*' -o \
-iname '*.binlog' -o \
-iname '*.log' \) \
-exec cp {} --parents /logs/source-build/logs \;"
-exec cp {} --parents /logs/source-build/logs \;
find . > /logs/source-build/logs/file-listing.txt"
du -h $(rootDirectory) | sort -h | tail -n 50
displayName: Copy source-build production build logs
condition: always()
Expand Down Expand Up @@ -197,6 +210,18 @@ jobs:
timeoutInMinutes: 120
condition: and(succeeded(), eq(variables['sb.tarball'], true))

- script: |
set -ex
df -h
$(docker.run) $(docker.drop.map) $(docker.tb.map) $(docker.tb.work) $(imageName) "$(tarballName)/build.sh" \
--collect-artifacts \
/p:ArtifactOutput=/drop/artifacts/tarball/ \
/bl
du -h $(rootDirectory) | sort -h | tail -n 50
displayName: Copy tarball artifacts
condition: eq(variables['sb.tarball'], true)
continueOnError: true

# Run smoke tests.
- script: |
set -ex
Expand Down Expand Up @@ -228,7 +253,8 @@ jobs:
-path './bin/aspnet-debug/*' -o \
-iname '*.binlog' -o \
-iname '*.log' \) \
-exec cp {} --parents /logs/tarball/logs \;"
-exec cp {} --parents /logs/tarball/logs \;
find . > /logs/tarball/logs/file-listing.txt"
du -h $(rootDirectory) | sort -h | tail -n 50
displayName: Copy tarball logs
condition: eq(variables['sb.tarball'], true)
Expand All @@ -252,6 +278,7 @@ jobs:
# Copy artifacts to staging - Copy to VSTS owned folder is done outside of docker so copied files
# have correct ownership so VSTS can clean them up later.
- task: CopyFiles@2
displayName: Copy artifacts from Docker-owned staging directory to AzDO-agent-owned artifacts directory
condition: always()
continueOnError: true
inputs:
Expand All @@ -275,6 +302,14 @@ jobs:
PathtoPublish: $(Build.ArtifactStagingDirectory)/drop/tarball
ArtifactName: Tarball $(artifactName)
ArtifactType: Container
- task: PublishBuildArtifacts@1
displayName: Publish build artifacts
condition: always()
continueOnError: true
inputs:
PathtoPublish: $(Build.ArtifactStagingDirectory)/drop/artifacts
ArtifactName: Artifacts $(artifactName)
ArtifactType: Container

# Publish prebuilt report data to blob storage.
- template: ../steps/publish-prebuilt-data-sh.yml
Expand Down
145 changes: 145 additions & 0 deletions .vsts.pipelines/jobs/collect-artifacts.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,145 @@
jobs:
- job: Collect_Artifacts

dependsOn: ${{ parameters.dependsOn }}

displayName: Collect Artifacts

pool: ${{ parameters.pool }}

steps:
- ${{ if and(ne(variables['System.TeamProject'], 'public'), notin(variables['Build.Reason'], 'PullRequest')) }}:
- task: DownloadBuildArtifacts@0
displayName: Download CentOS artifacts
inputs:
artifactName: Tarball centos71 Offline
downloadPath: '$(Build.StagingDirectory)/centos'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMO you should use an each and pass a list of objects to this template to keep this out of the implementation yaml. You can get this down to only have a a single input list of objects that you reprocess to create the dependsOn and the sequence of download steps. The overall point being to keep the platform listings in a single file.


- task: Bash@3
inputs:
targetType: 'inline'
workingDirectory: '$(Build.StagingDirectory)/centos'
script: |
rm */tarball*.tar.gz

- task: DownloadBuildArtifacts@0
displayName: Download Debian artifacts
inputs:
artifactName: Tarball debian9 Offline
downloadPath: '$(Build.StagingDirectory)/debian'

- task: Bash@3
inputs:
targetType: 'inline'
workingDirectory: '$(Build.StagingDirectory)/debian'
script: |
rm */tarball*.tar.gz

- task: DownloadBuildArtifacts@0
displayName: Download Fedora artifacts
inputs:
artifactName: Tarball fedora30 Offline
downloadPath: '$(Build.StagingDirectory)/fedora'

- task: Bash@3
inputs:
targetType: 'inline'
workingDirectory: '$(Build.StagingDirectory)/fedora'
script: |
rm */tarball*.tar.gz

- task: DownloadBuildArtifacts@0
displayName: Download Ubuntu artifacts
inputs:
artifactName: Tarball ubuntu1804 Offline
downloadPath: '$(Build.StagingDirectory)/ubuntu'

- task: Bash@3
inputs:
targetType: 'inline'
workingDirectory: '$(Build.StagingDirectory)/ubuntu'
script: |
rm */tarball*.tar.gz

- task: DownloadBuildArtifacts@0
displayName: Download portable artifacts
inputs:
artifactName: Tarball centos71 Offline Portable
downloadPath: '$(Build.StagingDirectory)/portable'

- task: Bash@3
inputs:
targetType: 'inline'
workingDirectory: '$(Build.StagingDirectory)/portable'
script: |
rm */tarball*.tar.gz

- task: Bash@3
inputs:
targetType: 'inline'
workingDirectory: '$(Build.StagingDirectory)'
script: |
version=`echo centos/*/Private.SourceBuilt.Artifacts*.tar.gz | sed 's/\.tar\.gz$//' | sed 's/^centos\/.*\/Private\.SourceBuilt\.Artifacts\.//'`
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

$( is preferred over ` generally. But this is also quite complex to be here and something that might belong as build logic (where I would think version would be accessible?).

echo "Artifacts version: $version"
mkdir collected
cd collected
# do CentOS last to prefer those versions of any duplicates
for os in portable debian fedora ubuntu centos; do
# DownloadArtifact creates another directory - use '*' to not deal with this
tar xzf ../$os/*/Private.SourceBuilt.Artifacts*.tar.gz
rm -rf ../$os
done
echo $(Build.SourceVersion) > .version
echo $(Build.BuildNumber) > .buildnumber
echo $(Build.BuildId) > .buildid
tar czf ../Private.SourceBuilt.Artifacts.$version.tar.gz * .version .buildnumber .buildid
rm -rf *
mv ../Private.SourceBuilt.Artifacts.*.tar.gz .
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It isn't clear to me what this artifact will be used for... sounds like more than diag if it's ending in BAR, so I'm a little concerned since point of source-build is that you can't get a combined artifact like this.

displayName: Collect RID-specific packages into one tarball
condition: succeeded()

- task: PowerShell@2
inputs:
targetType: 'filePath'
filePath: $(Build.SourcesDirectory)\publish.ps1
arguments: >-
/p:OfficialBuildId=$(Build.BuildNumber)
/p:AzureAccountName=$(publish.blobStorage.account)
/p:ContainerName=$(publish.blobStorage.container)
"/p:AzureAccessToken=$(publish.blobStorage.accessToken)"
"/p:ManifestName=$(BUILD.REPOSITORY.NAME)"
"/p:ManifestBuildId=$(BUILD.BUILDID)"
"/p:ManifestBranch=$(BUILD.SOURCEBRANCHNAME)"
/p:ManifestCommit=$(Build.SourceVersion)
/p:DotNetPublishToBlobFeed=true
workingDirectory: '$(Build.SourcesDirectory)'
displayName: Publish collected artifacts
condition: succeeded()

- task: PublishBuildArtifacts@1
displayName: Uploaded collected artifacts
condition: always()
continueOnError: true
inputs:
PathToPublish: $(Build.SourcesDirectory)/artifacts/tarball/
ArtifactName: Collected artifacts
ArtifactType: Container

- task: PublishBuildArtifacts@1
displayName: Uploaded asset manifests
condition: always()
continueOnError: true
inputs:
PathToPublish: $(Build.SourcesDirectory)/artifacts/asset-manifests/
ArtifactName: AssetManifests
ArtifactType: Container

- task: PublishBuildArtifacts@1
displayName: Upload publish logs
condition: always()
continueOnError: true
inputs:
PathToPublish: $(Build.SourcesDirectory)/artifacts/log/
ArtifactName: Publishing logs
ArtifactType: Container

4 changes: 4 additions & 0 deletions build.proj
Original file line number Diff line number Diff line change
Expand Up @@ -245,6 +245,10 @@
Overwrite="true" />
</Target>

<Target Name="CopyArtifactsToOutputFolder">
<MSBuild Projects="eng/publishPackages/collectArtifacts.proj" Targets="CopyArtifactsToOutputFolder" Properties="ArtifactOutput=$(ArtifactOutput)" />
</Target>

<Import Project="$([MSBuild]::GetDirectoryNameOfFileAbove($(MSBuildThisFileDirectory), dir.targets))/dir.targets" />

</Project>
3 changes: 3 additions & 0 deletions build.sh
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,9 @@ for arg do
(--generate-prebuilt-data) set -- "$@" "/t:GeneratePrebuiltBurndownData"
alternateTarget=true
;;
(--collect-artifacts) set -- "$@" "/t:CopyArtifactsToOutputFolder"
alternateTarget=true
;;
(*) set -- "$@" "$arg" ;;
esac
done
Expand Down
2 changes: 2 additions & 0 deletions eng/Tools.props
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
<?xml version="1.0" encoding="utf-8"?>
<Project/>
7 changes: 5 additions & 2 deletions eng/Versions.props
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,11 @@
<Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<!-- Repo Version Information -->
<PropertyGroup>
<VersionPrefix>0.1.0</VersionPrefix>
<PreReleaseVersionLabel>alpha.1</PreReleaseVersionLabel>
<MajorVersion>3</MajorVersion>
<MinorVersion>1</MinorVersion>
<ExtraSuffix Condition="'$(BUILD_BUILDNUMBER)' != ''">$(BUILD_BUILDNUMBER)</ExtraSuffix>
<ExtraSuffix Condition="'$(BUILD_SOURCEVERSION)' != ''">$(VersionSuffix)+$(BUILD_SOURCEVERSION)</ExtraSuffix>
<ExtraSuffix Condition="'$(BUILD_SOURCEVERSION)' == '' and '$(GitCommitHash)' != ''">$(VersionSuffix)+$(GitCommitHash)</ExtraSuffix>
</PropertyGroup>
<!-- Production Dependencies -->
<PropertyGroup>
Expand Down
6 changes: 4 additions & 2 deletions eng/common/PublishToSymbolServers.proj
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,8 @@
<PropertyGroup>
<DotNetSymbolExpirationInDays Condition="'$(DotNetSymbolExpirationInDays)' == ''">3650</DotNetSymbolExpirationInDays>
<PublishToSymbolServer>true</PublishToSymbolServer>
<PublishToSymWeb Condition="'$(PublishToSymWeb)' == ''">true</PublishToSymWeb>
<PublishToMSDL Condition="'$(PublishToMSDL)' == ''">true</PublishToMSDL>
<PublishToSymbolServer Condition="'@(FilesToPublishToSymbolServer)' == '' and '@(PackagesToPublishToSymbolServer)' == ''">false</PublishToSymbolServer>
</PropertyGroup>

Expand All @@ -56,7 +58,7 @@
DryRun="false"
ConvertPortablePdbsToWindowsPdbs="false"
PdbConversionTreatAsWarning=""
Condition="$(PublishToSymbolServer)"/>
Condition="$(PublishToSymbolServer) and $(PublishToMSDL)"/>

<!--
Symbol Uploader: SymWeb
Expand All @@ -73,7 +75,7 @@
DryRun="false"
ConvertPortablePdbsToWindowsPdbs="false"
PdbConversionTreatAsWarning=""
Condition="$(PublishToSymbolServer)"/>
Condition="$(PublishToSymbolServer) and $(PublishToSymWeb)"/>
</Target>

<ItemGroup>
Expand Down
Loading