Add SQL DB instances into single script

You can add other databases to add them into “$databaseServerPairs” in below code.

• https://github.com/Microsoft/sql-server-samples/blob/master/samples/manage/azure-automation-automated-export/AutoExport.ps1#L18

And please use other credentials if you use other SQL Database servers.

• https://github.com/Microsoft/sql-server-samples/blob/master/samples/manage/azure-automation-automated-export/AutoExport.ps1#L27

Export error when SQL DB instances so large

Please read this section when you get below error.

The error message is caused by below line.
- https://github.com/Microsoft/sql-server-samples/blob/master/samples/manage/azure-automation-automated-export/AutoExport.ps1#L115

The error is caused by below, so it seems to take too long time to copy DB data.

  if((-not $? -and $global:retryLimit -ile $dbObj.RetryCount) -or ($currentTime - $dbObj.OperationStartTime).TotalMinutes -gt $global:waitInMinutes)

Please change variable “$waitInMinutes = 30;” from 30 minutes to long time.

• https://github.com/Microsoft/sql-server-samples/blob/master/samples/manage/azure-automation-automated-export/AutoExport.ps1#L37

In order to execute the runbook do I need to have the automation account to have the ability to “Run As account”?

“Azure Run As account” is needed, because we can't execute Runbook scripts without this. It needs to enable Azure Active Directory to register applications.
https://docs.microsoft.com/en-us/azure/automation/automation-create-aduser-account#create-an-automation-account-in-the-azure-portal

" 429 Too many requests" error in Runbook Job log when exporting large SQL Database instances

You will get below error when you execute long jobs.

Get-AzureSqlDatabaseImportExportStatus : A task was canceled.
At line:181 char:11
+ ...    $check = Get-AzureSqlDatabaseImportExportStatus -Request $dbObj.Ex ...
+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Get-AzureSqlDatabaseImportExportStatus], TaskCanceledException
    + FullyQualifiedErrorId : 
Microsoft.WindowsAzure.Commands.SqlDatabase.Database.Cmdlet.GetAzureSqlDatabaseImportExportStatus

• https://github.com/Microsoft/sql-server-samples/blob/master/samples/manage/azure-automation-automated-export/AutoExport.ps1#L211

The error is caused by frequent requests using “Get-AzureSqlDatabaseImportExportStatus”, so it need to insert “Start-Sleep” in the script to reduce Azure Management API calls internally.

• https://docs.microsoft.com/en-us/previous-versions/windows/it-pro/windows-powershell-1.0/ee177002(v=technet.10)
• Azure Management API - 429 Too many requests - Stack Overflow

Revert with "Start-ServiceFabricApplicationUpgrade"

Service Fabric retains old application packages for a while like below. As far as I confirmed, it should retain at least a few hours.

Meanwhile the retainment, you can revert from new deployment to old one with below PowerShell commands.

Login-AzureRmAccount

$applicationName = 'fabric:/FabricApp01'

$connectArgs = @{  ConnectionEndpoint = "'<your cluster name'".westus.cloudapp.azure.com:19000';  
                   X509Credential = $True;  
                   StoreLocation = "CurrentUser";  
                   StoreName = "My";  
                   ServerCommonName = "'your cluster name'.westus.cloudapp.azure.com";  
                   FindType = 'FindByThumbprint';  
                   # "Client certificates" thumbprint. Pick up this value from "security" item in your cluster on Azure Portal
                   FindValue = "YYYYYYYYYY7e3372bc1ed5cf62b435XXXXXXXXXX"; 
                   # "Cluster certificates" thumbprint.  Pick up this value from "security" item in your cluster on Azure Portal
                   ServerCertThumbprint = "YYYYYYYYYY2E67D7E54647A12B7787XXXXXXXXXX" } 
Connect-ServiceFabricCluster @connectArgs

$app = Get-ServiceFabricApplication -ApplicationName $applicationName
$app 
$table = @{}
$app.ApplicationParameters | ForEach-Object { $table.Add( $_.Name, $_.Value)}
Start-ServiceFabricApplicationUpgrade -ApplicationName $applicationName -ApplicationTypeVersion "1.0.2.52" -ApplicationParameter $table -UnmonitoredAuto

You can watch it progress in Service Fabric Explorer like below.

Revert with new Release using existing build packages

I believe you have already made some build packages for deployment into Service Fabric. You can create new Release in your VSTS using the packages like below.

Create your Azure Automation Account

At first, note when you create Azure Automation account. And you must create "Azure Run As account" like below, because it's mandatory to execute your Azure Automation scripts called as "Runbook". This probably need your Azure Active Directory privilege of App Registration.

Create your Runbook

Create a Runbook used to execute your scripts. Choose "Runbook" from left side of you Azure Automation account and click "Add a runbook" like below.

And input your Runbook name and choose "PowerShell" as your Runbook type.

Create your scripts into your Runbook

Open your Runbook and click "Edit" to create your script. Update your script like below.

$Conn = Get-AutomationConnection -Name AzureRunAsConnection
Add-AzureRMAccount -ServicePrincipal -Tenant $Conn.TenantID -ApplicationId $Conn.ApplicationID -CertificateThumbprint $Conn.CertificateThumbprint
 
get-azurermresourcegroup | ForEach-Object { $_.ResourceGroupName }

The name of "AzureRunAsConnection" should be created in "'your Azure Automation account name'- Connections". Once again, this is mandatory to execute your script. Confirm it like below if you need.

After updating the script, click "Test pane" to test your script. You can execute your script by clicking "Start" button, so you can take result like below.

Now you can publish your script by clicking "Publish" button to schedule and collaborate with other Runbooks. After publishing that, confirm the status like below.

Schedule your Runbook

Go back to top of your Azure Automation account and choose "Schedule" and click "Add a schedule" like below.

In this example, I setup my schedule as weekly like below.

Finally, you have to associate with your Runbook and Schedule. Go back to your Runbook, choose "Schedule" and click "Add a schedule". Associate your schedule like below.

Now, you can execute your script based on your schedule.

Create your new Runbook as "PowerShell"

Create your new "PowrShell" Runbook under your Azure Automation account and edit it like below. This Runbook output your Azure resources in a location specified by a parameter.

Param
(
    [Parameter (Mandatory = $true)]
    [String] $Location = 'Japan East'
)

# Setup Authentication
$Conn = Get-AutomationConnection -Name AzureRunAsConnection
Add-AzureRMAccount -ServicePrincipal -Tenant $Conn.TenantID -ApplicationId $Conn.ApplicationID -CertificateThumbprint $Conn.CertificateThumbprint

Get-AzureRmResourceGroup -Location $Location | ForEach-Object { Write-Output $_.ResourceGroupName }

You can specify parameters with "Param" keyword like above. "PowerShell Workflow" created in next section can call "PowerShell" Runbook, so you have to create your Runbooks as "PowerShell".

Create new Runbook as "PowerShell Workflow"

You can find how to pass your parameters and how to get output with your "PowerShell" Runbook.

workflow workflow-sample
{
    Param
    (
        [Parameter (Mandatory = $true)]
        [String] $Location01 = "West US",
        [Parameter (Mandatory = $true)]
        [String] $Location02 = "West Central US"
    )

    # settings
    $automationAccountName = "mytest-automation"
    $resourceGroupName = "mytest-automation-rg"

    # Setup Authentication
    $Conn = Get-AutomationConnection -Name AzureRunAsConnection
    Add-AzureRMAccount -ServicePrincipal -Tenant $Conn.TenantID -ApplicationId $Conn.ApplicationID -CertificateThumbprint $Conn.CertificateThumbprint

    ## backup Runbook
    echo '#1 runbook starts'
    $params = @{ 'Location'=$Location01 }
    $runbookName = 'execute-azure-cmdlet'
    $job = Start-AzureRmAutomationRunbook -AutomationAccountName $automationAccountName -Name $runbookName -ResourceGroupName $resourceGroupName -Parameters $params
    $doLoop = $true
    While ($doLoop) {
        $job = Get-AzureRmAutomationJob –AutomationAccountName $automationAccountName -Id $job.JobId -ResourceGroupName $resourceGroupName
        $status = $job.Status
        if($status -eq "Failed") {
            Write-Error "Error in $runbookName"
            Write-Error $job.Exception
            throw $job.Exception
        }
        $doLoop = (($status -ne "Completed") -and ($status -ne "Suspended") -and ($status -ne "Stopped"))
        Start-Sleep -Seconds 2
    }
    echo '################# output start #################'
    $record = Get-AzureRmAutomationJobOutput –AutomationAccountName $automationAccountName -Id $job.JobId -ResourceGroupName $resourceGroupName –Stream Any | Get-AzureRmAutomationJobOutputRecord
    $record # for example
    echo '                               #################'
    $record | Where-Object { $_.Value.value -NE $null} | ForEach-Object { Write-Output $_.Value.value }
    echo '################# output end #################'
    echo '#1 runbook is ended'

    ## 
    echo '#2 runbook is starts'
    $params = @{ 'Location'=$Location02 }
    $runbookName = 'execute-azure-cmdlet'
    $job = Start-AzureRmAutomationRunbook -AutomationAccountName $automationAccountName -Name $runbookName -ResourceGroupName $resourceGroupName -Parameters $params
    $doLoop = $true
    While ($doLoop) {
        $job = Get-AzureRmAutomationJob –AutomationAccountName $automationAccountName -Id $job.JobId -ResourceGroupName $resourceGroupName
        $status = $job.Status
        if($status -eq "Failed") {
            Write-Error "Error in $runbookName"
            Write-Error $job.Exception
            throw $job.Exception
        }
        $doLoop = (($status -ne "Completed") -and ($status -ne "Suspended") -and ($status -ne "Stopped"))
        Start-Sleep -Seconds 2
    }
    echo '################# output start #################'
    $record = Get-AzureRmAutomationJobOutput –AutomationAccountName $automationAccountName -Id $job.JobId -ResourceGroupName $resourceGroupName –Stream Any | Get-AzureRmAutomationJobOutputRecord
    $record | Where-Object { $_.Value.value -NE $null} | ForEach-Object { Write-Output $_.Value.value }
    echo '################# output end #################'
    echo '#2 runbook is ended'
}

Output logs with Workflow

You can execute your Workflow and find output logs like below.

PSComputerName        : localhost
PSSourceJobInstanceId : 256fbcbd-f339-4ce5-b75b-0dc973dd0f2a
Environments          : {AzureCloud, AzureChinaCloud, AzureUSGovernment}
Context               : Microsoft.Azure.Commands.Profile.Models.PSAzureContext




#1 runbook starts

################# output start #################

PSComputerName        : localhost

PSSourceJobInstanceId : 256fbcbd-f339-4ce5-b75b-0dc973dd0f2a
Value                 : {Environments, Context}
ResourceGroupName     : mytest-automation-rg
AutomationAccountName : mytest-automation
JobId                 : eb19892d-8e2d-4572-862f-9205ca6e89fc
StreamRecordId        : eb19892d-8e2d-4572-862f-9205ca6e89fc:00636563050813081260:00000000000000000001
Time                  : 03/10/2018 18:58:01 +00:00
Summary               : 
Type                  : Output

PSComputerName        : localhost
PSSourceJobInstanceId : 256fbcbd-f339-4ce5-b75b-0dc973dd0f2a
Value                 : {value}
ResourceGroupName     : mytest-automation-rg
AutomationAccountName : mytest-automation
JobId                 : eb19892d-8e2d-4572-862f-9205ca6e89fc
StreamRecordId        : eb19892d-8e2d-4572-862f-9205ca6e89fc:00636563050827143533:00000000000000000002
Time                  : 03/10/2018 18:58:02 +00:00
Summary               : normalian-datacatalog-rg
Type                  : Output

PSComputerName        : localhost
PSSourceJobInstanceId : 256fbcbd-f339-4ce5-b75b-0dc973dd0f2a
Value                 : {value}
ResourceGroupName     : mytest-automation-rg
AutomationAccountName : mytest-automation
JobId                 : eb19892d-8e2d-4572-862f-9205ca6e89fc
StreamRecordId        : eb19892d-8e2d-4572-862f-9205ca6e89fc:00636563050827612512:00000000000000000003
Time                  : 03/10/2018 18:58:02 +00:00
Summary               : sqldb-rg
Type                  : Output

                               #################

normalian-datacatalog-rg

sqldb-rg

################# output end #################

#1 runbook is ended

#2 runbook is starts

################# output start #################

demo-automation-rg

mytest-automation-rg

################# output end #################

#2 runbook is ended

References

• Runbook execution in Azure Automation | Microsoft Docs
• Runbook Output and Messages in Azure Automation | Microsoft Docs
• Runbook input parameters | Microsoft Docs

In VSTS Build Process

Create "Azure PowerShell script" task and "Azure Deployment: Create Or Update Resource Group Action" like below.

Azure PowerShell script: inline Script – inline script
Edit "Azure PowerShell script" task like below.

$context = New-AzureStorageContext -StorageAccountName 'your storage account name' -StorageAccountKey 'your storage access key'
$sasUrl = New-AzureStorageContainerSASToken -Container templates -Permission rwdl -Context $context 
Write-Output ("##vso[task.setvariable variable=SasUrl;]$sasUrl")

You can store generated values with VSTS variables like above.

Azure Resource Group Deployment - Override template parameters
Edit "Azure Deployment: Create Or Update Resource Group Action" like below.

-SASToken $(SasUrl)

Part of ARM template

Now you can use SAS token to specify your linked templates like below. Refer this sample if you need.

"variables": {"sharedTemplateUrl": "[concat('https://'your storage account name'.blob.core.windows.net/templates/blank-azuredeploy.json', parameters('SASToken') )]",
      "sharedParametersUrl": "[concat('https://'your storage account name'.blob.core.windows.net/templates/blank-azuredeploy.parameters.json', parameters('SASToken'))]"
    },

References

• GitHub - normalian/ARMTemplate-SASToken-InVSTS-Sample
• Setting VSTS variables from PowerShell
• Build variables | Microsoft Docs

Example to match OS Versions

It's important to match your Service Fabric cluster "Operating System" and base OS version specified in Dockerfile "FROM" keyword. And I put ServiceManifest and ApplicationManifest.xml just in case.

Example - part of Dockerfile

# This base OS for "WindowsServer 2016-Datacenter-with-Containers"
#FROM microsoft/aspnetcore-build:2.0.5-2.1.4-nanoserver-sac2016 AS base
# This base OS for "WindowsServerSemiAnnual Datacenter-Core-1709-with-Containers"
FROM microsoft/aspnetcore:2.0-nanoserver-1709 AS base
WORKDIR /app
EXPOSE 80

# This base OS for "WindowsServer 2016-Datacenter-with-Containers"
#FROM microsoft/aspnetcore-build:2.0.5-2.1.4-nanoserver-sac2016 AS build
# This base OS for "WindowsServerSemiAnnual Datacenter-Core-1709-with-Containers"
FROM microsoft/aspnetcore-build:2.0-nanoserver-1709 AS build
WORKDIR /src
COPY *.sln ./
COPY NetCoreWebApp/NetCoreWebApp.csproj NetCoreWebApp/
RUN dotnet restore
COPY . .
WORKDIR /src/NetCoreWebApp
RUN dotnet build -c Release -o /app

FROM build AS publish
RUN dotnet publish -c Release -o /app

FROM base AS final
WORKDIR /app
COPY --from=publish /app .
ENTRYPOINT ["dotnet", "NetCoreWebApp.dll"]

Example - part of ApplicationManifest.xml

<ServiceManifestImport><ServiceManifestRef ServiceManifestName="GuestContainer1Pkg"ServiceManifestVersion="1.0.0" /><ConfigOverrides /><Policies><ContainerHostPolicies CodePackageRef="Code"><RepositoryCredentials AccountName="Username of your Container registry"Password="password of your Container registry"PasswordEncrypted="false"/><PortBinding ContainerPort="80"EndpointRef="GuestContainer1TypeEndpoint"/></ContainerHostPolicies></Policies></ServiceManifestImport>

Example - part of ServiceManifest.xml

<EntryPoint><!-- Follow this link for more information about deploying Windows containers to Service Fabric: https://aka.ms/sfguestcontainers --><ContainerHost><ImageName>"Username of your Container registry".azurecr.io/sample/helloworldapp:latest</ImageName></ContainerHost></EntryPoint>

References

• There was an error during download.Failed to download container image · Issue #752 · Azure/service-fabric-issues · GitHub
• Create an Azure Service Fabric container application | Microsoft Docs
• Windows Container Version Compatibility | Microsoft Docs

What's happened if you build ASP.NET Framework Docker images with default settings

You can choose "Add - Docker Support" like below if you have setup "Visual Studio Tools for Docker" in your machine. The tool is really useful and it will generate below Dockerfile in your ASP.NET Framework application.

FROM microsoft/aspnet:4.7.1-windowsservercore-1709
ARG source
WORKDIR /inetpub/wwwroot
COPY ${source:-obj/Docker/publish} .

Last "COPY" command line is a little bit tricky, because copy sources directory will change with ${source} variable value. The directory will be ${source} variable value if the value is set, but the directory will be obj/Docker/publish if ${source} variable isn't set.

This Dockerfile should work on your machine, but it won't work on VSTS build tasks like below.

2018-03-20T16:23:53.7207717Z ##[section]Starting: Build an image
2018-03-20T16:23:53.7213905Z ==============================================================================
2018-03-20T16:23:53.7214387Z Task         : Docker
2018-03-20T16:23:53.7214924Z Description  : Build, tag, push, or run Docker images, or run a Docker command. Task can be used with Docker or Azure Container registry.
2018-03-20T16:23:53.7215463Z Version      : 0.3.10
2018-03-20T16:23:53.7215860Z Author       : Microsoft Corporation
2018-03-20T16:23:53.7216376Z Help         : [More Information](https://go.microsoft.com/fwlink/?linkid=848006)
2018-03-20T16:23:53.7216914Z ==============================================================================
2018-03-20T16:23:54.9185089Z [command]"C:\Program Files\Docker\docker.exe" build -f C:\agent\_work\1\s\Trunk\SFwithASPNetApp\ASPNetApp01\Dockerfile -t xxxxxxxxxxxxxxxxxxxister.azurecr.io/yyyyyyyyyyy-demo-projects:80 C:\agent\_work\1\s\Trunk\SFwithASPNetApp\ASPNetApp01
2018-03-20T16:23:55.0276465Z Sending build context to Docker daemon  3.072kB
2018-03-20T16:23:55.0278113Z 
2018-03-20T16:23:55.0302771Z Step 1/4 : FROM microsoft/aspnet:4.7.1-windowsservercore-1709
2018-03-20T16:23:55.0313682Z  ---> dc3f4d701ead
2018-03-20T16:23:55.0315249Z Step 2/4 : ARG source
2018-03-20T16:23:55.0326718Z  ---> Using cache
2018-03-20T16:23:55.0328908Z  ---> 9a10d9b50bc9
2018-03-20T16:23:55.0329320Z Step 3/4 : WORKDIR /inetpub/wwwroot
2018-03-20T16:23:55.0340016Z  ---> Using cache
2018-03-20T16:23:55.0342052Z  ---> 28d5a9cc0dd0
2018-03-20T16:23:55.0342974Z Step 4/4 : COPY ${source:-obj/Docker/publish} .
2018-03-20T16:23:55.0348449Z COPY failed: GetFileAttributesEx \\?\C:\Windows\TEMP\docker-builder521937930\obj\Docker\publish: The system cannot find the path specified.
2018-03-20T16:23:55.0575409Z ##[error]C:\Program Files\Docker\docker.exe failed with return code: 1
2018-03-20T16:23:55.0588673Z ##[section]Finishing: Build an image

What is workaround of the issue?

You need to setup both Visual Studio and VSTS in this case. Please follow below steps .

Setup in your ASP.NET Framework project
At first you need to setup new pubxml file in your ASP.NET Framework project to ensure output binaries into "obj\Docker\publish" directory, so choose "Publish" from right click menu of your ASP.NET Framework project like below.

Choose "Folder" as target and edit "Choose a folder" as "obj\Docker\publish" like below.

After setup the pubxml file, you can find it in your ASP.NET Framework like below. In this case, the filename is "FolderProfile.pubxml".

Setup in your VSTS tasks
You need to update "Build Solution" tasks at first. Please note to add "/p:PublishProfile=FolderProfile.pubxml" and remove "/p:WebPublishMethod=Package".

• "MSBuild Arguments" - before change

/p:DeployOnBuild=true /p:WebPublishMethod=Package /p:PackageAsSingleFile=true /p:SkipInvalidConfigurations=true /p:PackageLocation="$(build.artifactstagingdirectory)\\"

• "MSBuild Arguments" - after change

/p:DeployOnBuild=true /p:PackageAsSingleFile=true /p:SkipInvalidConfigurations=true /p:PackageLocation="$(build.artifactstagingdirectory)\\" /p:PublishProfile=FolderProfile.pubxml 

After setup "Build Solution" tasks correctly, you can just add "Build an Image" task and specify "Docker file" correctly like below.

And you also need to add "Push an image" to store your Docker images into some registries. Note to specify as "Push an Image" not "Push images" like below.

You can find your Docker images in your Container registry if the build process on VSTS work correctly.

References

• Does not work out of the box. "The system cannot find the file specified" · Issue #65 · Microsoft/aspnet-docker · GitHub
• Visual Studio Tools for Docker with ASP.NET Core | Microsoft Docs
• docker - Dockerfile COPY {source-...} - Stack Overflow

Step by Step to step Windows Server version 1709 based VM as Private Agent

You need to create new Virtual Machine on Azure Portal. Choose "Windows Server, version 1709 with Containers" for base VM, because it contains "docker.exe" command. But keep in mind the image doesn't contain "docker-compose".

You don't need add special setting when you create the VM, but don't setup Network Security Group as completely closed to enable accessible VSTS service.

Access the VM using Remote Desktop and install Visual Studio 2017 into the VM, because the VM doesn't contain MSBuild and other commands for VSTS Build/Release processes. Follow below commands.

C:\Users\azureuser>powershell
Windows PowerShell
Copyright (C) Microsoft Corporation. All rights reserved.

PS C:\Users\azureuser> curl https://aka.ms/vs/15/release/vs_community.exe -O vs_community.exe
PS C:\Users\azureuser> dir


    Directory: C:\Users\azureuser


Mode                LastWriteTime         Length Name
----                -------------         ------ ----
d-r---        3/22/2018   4:19 PM                3D Objects
d-r---        3/22/2018   4:19 PM                Contacts
d-r---        3/22/2018   4:19 PM                Desktop
d-r---        3/22/2018   4:19 PM                Documents
d-r---        3/22/2018   4:19 PM                Downloads
d-r---        3/22/2018   4:19 PM                Favorites
d-r---        3/22/2018   4:19 PM                Links
d-r---        3/22/2018   4:19 PM                Music
d-r---        3/22/2018   4:19 PM                Pictures
d-r---        3/22/2018   4:19 PM                Saved Games
d-r---        3/22/2018   4:19 PM                Searches
d-r---        3/22/2018   4:19 PM                Videos
-a----        3/22/2018   4:24 PM        1180608 vs_community.exe

PS C:\Users\azureuser> .\vs_community.exe

I chose below settings in my case, but change the settings for your environment if you need.

After Visual Studio installation has completed, add MSBuild execution folder path into PATH environment variable like below.

PS C:\Users\azureuser> setx /M PATH "%PATH%;C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\MSBuild\15.0\B
in"

SUCCESS: Specified value was saved.
PS C:\Users\azureuser> 

Next, you also need to add "docker-compose" to build with it, because this base VM contains only just "docker" command. Follow below commands to install it.

PS C:\> [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
PS C:\> Invoke-WebRequest "https://github.com/docker/compose/releases/download/1.20.0/docker-compose-Windows-x86_64.exe" -UseBasicParsing -OutFile $Env:ProgramFiles\docker\docker-compose.exe
PS C:\>

What will be error messages if you haven't complete docker-compose

You will get "##[error]Unhandled: Failed which: Not found docker: null" message from your VSTS build task.

2018-03-17T21:07:37.4182183Z ##[section]Starting: Build an image
2018-03-17T21:07:37.4186946Z ==============================================================================
2018-03-17T21:07:37.4187316Z Task         : Docker
2018-03-17T21:07:37.4187693Z Description  : Build, tag, push, or run Docker images, or run a Docker command. Task can be used with Docker or Azure Container registry.
2018-03-17T21:07:37.4188244Z Version      : 0.3.10
2018-03-17T21:07:37.4188534Z Author       : Microsoft Corporation
2018-03-17T21:07:37.4188879Z Help         : [More Information](https://go.microsoft.com/fwlink/?linkid=848006)
2018-03-17T21:07:37.4189247Z ==============================================================================
2018-03-17T21:07:37.6890544Z ##[error]Unhandled: Failed which: Not found docker: null
2018-03-17T21:07:37.6953107Z ##[section]Finishing: Build an image

What will be error messages if you build Windows Server ver 1709 images with "Hosted VS2017" agent

You will get "The following Docker images are incompatible with the host operating system: [microsoft/aspnet:4.7.1-windowsservercore-1709]. Update the Dockerfile to specify a different base image." message from your VSTS build task.

2018-03-17T21:02:41.2336315Z 
2018-03-17T21:02:41.2336964Z Build FAILED.
2018-03-17T21:02:41.4137038Z 
2018-03-17T21:02:41.4138305Z "D:\a\1\s\Trunk\SFwithASPNetApp\SFwithASPNetApp.sln" (default target) (1) ->
2018-03-17T21:02:41.4139002Z "D:\a\1\s\Trunk\SFwithASPNetApp\docker-compose.dcproj" (default target) (3) ->
2018-03-17T21:02:41.4139575Z (DockerComposeBuild target) -> 
<b>2018-03-17T21:02:41.4141190Z   C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\MSBuild\Sdks\Microsoft.Docker.Sdk\build\Microsoft.Docker.targets(111,5): error : The following Docker images are incompatible with the host operating system: [microsoft/aspnet:4.7.1-windowsservercore-1709]. Update the Dockerfile to specify a different base image. See http://aka.ms/DockerToolsTroubleshooting for more details. [D:\a\1\s\Trunk\SFwithASPNetApp\docker-compose.dcproj]</b>
2018-03-17T21:02:41.4142524Z 
2018-03-17T21:02:41.4142996Z     0 Warning(s)
2018-03-17T21:02:41.4143435Z     1 Error(s)
2018-03-17T21:02:41.4143702Z 
2018-03-17T21:02:41.4144142Z Time Elapsed 00:14:00.47
2018-03-17T21:02:43.2578358Z ##[error]Process 'msbuild.exe' exited with code '1'.
2018-03-17T21:02:44.0944814Z ##[section]Finishing: Build solution
2018-03-17T21:02:44.1143215Z ##[section]Starting: Post Job Cleanup
2018-03-17T21:02:44.1300074Z Cleaning any cached credential from repository: US-Crackle-Demo-Projects (Git)
2018-03-17T21:02:44.1413004Z ##[command]git remote set-url origin https://daisami-online.visualstudio.com/_git/US-Crackle-Demo-Projects
2018-03-17T21:02:44.3340613Z ##[command]git remote set-url --push origin https://daisami-online.visualstudio.com/_git/US-Crackle-Demo-Projects
2018-03-17T21:02:44.3757483Z ##[section]Finishing: Post Job Cleanup
2018-03-17T21:02:44.4763369Z ##[section]Finishing: Job

Finally, you need to setup VSTS Private Agent. Refer to How to setup your CentOS VMs as VSTS Private Agent - normalian blog to pick up "access token" for setup Private Agent.

PS C:\agent> .\config.cmd

>> Connect:

Enter server URL > https://"your vsts account name".visualstudio.com
Enter authentication type (press enter for PAT) >
Enter personal access token > ****************************************************
Connecting to server ...

>> Register Agent:

Enter agent pool (press enter for default) > "Your Agent Pool Name"
Enter agent name (press enter for VSTSPAVM01) >
Scanning for tool capabilities.
Connecting to the server.
Successfully added the agent
Testing agent connection.
Enter work folder (press enter for _work) >
2018-03-18 05:09:24Z: Settings Saved.
Enter run agent as service? (Y/N) (press enter for N) > Y
Enter User account to use for the service (press enter for NT AUTHORITY\NETWORK SERVICE) > NT AUTHORITY\SYSTEM
Granting file permissions to 'NT AUTHORITY\SYSTEM'.
Service vstsagent.daisami-online.VSTSPAVM01 successfully installed
Service vstsagent.daisami-online.VSTSPAVM01 successfully set recovery option
Service vstsagent.daisami-online.VSTSPAVM01 successfully configured
Service vstsagent.daisami-online.VSTSPAVM01 started successfully
PS C:\agent>

Now, you can choose your Private Agent in your VSTS Build/Release processes.

What will be error messages if you setup Private Agent account as "NT AUTHORITY\NETWORK SERVICE"

Your VM can't access" //./pipe/docker_engine: " and the build tasks will be failed.

DockerGetServiceReferences:
docker-compose -f "C:\agent\_work\1\s\Trunk\SFwithASPNetApp\docker-compose.yml" -f "C:\agent\_work\1\s\Trunk\SFwithASPNetApp\docker-compose.override.yml" -p dockercompose13733567670188849996 --no-ansi config
C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\MSBuild\Sdks\Microsoft.Docker.Sdk\build\Microsoft.VisualStudio.Docker.Compose.targets(195,5): Error MSB4018: The "GetServiceReferences" task failed unexpectedly.
C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\MSBuild\Sdks\Microsoft.Docker.Sdk\build\Microsoft.VisualStudio.Docker.Compose.targets(195,5): error MSB4018: The "GetServiceReferences" task failed unexpectedly. [C:\agent\_work\1\s\Trunk\SFwithASPNetApp\docker-compose.dcproj]
C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\MSBuild\Sdks\Microsoft.Docker.Sdk\build\Microsoft.VisualStudio.Docker.Compose.targets(195,5): error MSB4018: Microsoft.Docker.Utilities.CommandLineClientException: error during connect: Get http://%2F%2F.%2Fpipe%2Fdocker_engine/v1.30/version: open //./pipe/docker_engine: Access is denied. In the default daemon configuration on Windows, the docker client must be run elevated to connect. This error may also indicate that the docker daemon is not running.. [C:\agent\_work\1\s\Trunk\SFwithASPNetApp\docker-compose.dcproj]
C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\MSBuild\Sdks\Microsoft.Docker.Sdk\build\Microsoft.VisualStudio.Docker.Compose.targets(195,5): error MSB4018: [C:\agent\_work\1\s\Trunk\SFwithASPNetApp\docker-compose.dcproj]
C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\MSBuild\Sdks\Microsoft.Docker.Sdk\build\Microsoft.VisualStudio.Docker.Compose.targets(195,5): error MSB4018: For more troubleshooting information, go to http://aka.ms/DockerToolsTroubleshooting ---> Microsoft.Docker.Utilities.CommandLineClientException: error during connect: Get http://%2F%2F.%2Fpipe%2Fdocker_engine/v1.30/version: open //./pipe/docker_engine: Access is denied. In the default daemon configuration on Windows, the docker client must be run elevated to connect. This error may also indicate that the docker daemon is not running. [C:\agent\_work\1\s\Trunk\SFwithASPNetApp\docker-compose.dcproj]

References

• Install Docker Compose | Docker Documentation
• Use command-line parameters to install Visual Studio - Visual Studio | Microsoft Docs

Example to match OS Versions

It's important to match your Service Fabric cluster "Operating System" and base OS version specified in Dockerfile "FROM" keyword. And I put ServiceManifest and ApplicationManifest.xml just in case.

Example - part of Dockerfile

# This base OS for "WindowsServer 2016-Datacenter-with-Containers"
#FROM microsoft/aspnetcore-build:2.0.5-2.1.4-nanoserver-sac2016 AS base
# This base OS for "WindowsServerSemiAnnual Datacenter-Core-1709-with-Containers"
FROM microsoft/aspnetcore:2.0-nanoserver-1709 AS base
WORKDIR /app
EXPOSE 80

# This base OS for "WindowsServer 2016-Datacenter-with-Containers"
#FROM microsoft/aspnetcore-build:2.0.5-2.1.4-nanoserver-sac2016 AS build
# This base OS for "WindowsServerSemiAnnual Datacenter-Core-1709-with-Containers"
FROM microsoft/aspnetcore-build:2.0-nanoserver-1709 AS build
WORKDIR /src
COPY *.sln ./
COPY NetCoreWebApp/NetCoreWebApp.csproj NetCoreWebApp/
RUN dotnet restore
COPY . .
WORKDIR /src/NetCoreWebApp
RUN dotnet build -c Release -o /app

FROM build AS publish
RUN dotnet publish -c Release -o /app

FROM base AS final
WORKDIR /app
COPY --from=publish /app .
ENTRYPOINT ["dotnet", "NetCoreWebApp.dll"]

Example - part of ApplicationManifest.xml

<ServiceManifestImport><ServiceManifestRef ServiceManifestName="GuestContainer1Pkg"ServiceManifestVersion="1.0.0" /><ConfigOverrides /><Policies><ContainerHostPolicies CodePackageRef="Code"><RepositoryCredentials AccountName="Username of your Container registry"Password="password of your Container registry"PasswordEncrypted="false"/><PortBinding ContainerPort="80"EndpointRef="GuestContainer1TypeEndpoint"/></ContainerHostPolicies></Policies></ServiceManifestImport>

Example - part of ServiceManifest.xml

<EntryPoint><!-- Follow this link for more information about deploying Windows containers to Service Fabric: https://aka.ms/sfguestcontainers --><ContainerHost><ImageName>"Username of your Container registry".azurecr.io/sample/helloworldapp:latest</ImageName></ContainerHost></EntryPoint>

References

• There was an error during download.Failed to download container image · Issue #752 · Azure/service-fabric-issues · GitHub
• Create an Azure Service Fabric container application | Microsoft Docs
• Windows Container Version Compatibility | Microsoft Docs

References

• Create an Azure Service Fabric container application | Microsoft Docs
• Virtual Machine series | Microsoft Azure
• What is workaround when your got error messages "There was an error during download.Failed" while downloading container images - normalian blog

How to override environment variable values on VSTS tasks

You need to add "PowerShell" tasks into your build process, specify its "Type" as "Inline Script" and edit "Inline Script" like below.

$LowerBuildRepositoryName = "$(Build.Repository.Name)".ToLower()
Write-Output ("##vso[task.setvariable variable=Build.Repository.Name;]$LowerBuildRepositoryName")

Write-Host "Build.Repository.Name variable updates"

You can override any variables as you like to edit the inline script.

References

• Build variables | Microsoft Docs
• Variables in Release Management | Microsoft Docs

How to use "Replace Tokens" on VSTS

Input "Replace Tokens" into search box when you add new tasks in your VSTS Build/Release process and click "Install" to initialize it.

After adding "Replace Tokens" task in your process, change "Root directory" and "Target files" to specify which files you want to change. In below example, I specify *.xml files in my "SFwithASPNetApp" project.

And finally refer below a part of Service Fabric ServiceManifest.xml. This xml file uses "Build.Repository.Name" and "Build.BuildId" environment variables to specify Docker image name.

<CodePackage Name="Code"Version="1.0.0"><EntryPoint><!-- Follow this link for more information about deploying Windows containers to Service Fabric: https://aka.ms/sfguestcontainers --><ContainerHost><ImageName>mynormalianregister.azurecr.io/#{Build.Repository.Name}#:#{Build.BuildId}#</ImageName></ContainerHost></EntryPoint>

The Docker image name will replace from "mynormalianregister.azurecr.io/#{Build.Repository.Name}#:#{Build.BuildId}#" into "mynormalianregister.azurecr.io/us-customer-demo-projects:111" in this case.

Note that you must put "}#" not "}" as suffix token.

References

• VSTS Build: Replacing token with build number - Stack Overflow
• Variables in Release Management | Microsoft Docs
• Build variables | Microsoft Docs

How to setup Build tasks on VSTS

You need to add 5 tasks after your "Push an image" task, but you also need to add "PowerShell Script" task if you might need to use capital letters in your project names or something. Refer to How to override values of environment variables on VSTS tasks - normalian blog to override environment variables.

Now, we will introduce how to setup the tasks.

• Replace Tokens
• Build solution
• Update Service Fabric Manifests
• Copy Files
• Publish Build Artifacts

<?xml version="1.0" encoding="utf-8"?>
<ServiceManifest Name="GuestContainer1Pkg"
                 Version="1.0.0"
                 xmlns="http://schemas.microsoft.com/2011/01/fabric"
                 xmlns:xsd="http://www.w3.org/2001/XMLSchema"
                 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <ServiceTypes>
    <!-- This is the name of your ServiceType.
         The UseImplicitHost attribute indicates this is a guest service. -->
    <StatelessServiceType ServiceTypeName="GuestContainer1Type" UseImplicitHost="true" />
  </ServiceTypes>

  <!-- Code package is your service executable. -->
  <CodePackage Name="Code" Version="1.0.0">
    <EntryPoint>
      <!-- Follow this link for more information about deploying Windows containers to Service Fabric: https://aka.ms/sfguestcontainers -->
      <ContainerHost>
        <ImageName>"your acr account name".azurecr.io/#{Build.Repository.Name}#:#{Build.BuildId}#</ImageName>
      </ContainerHost>
    </EntryPoint>
    <!-- Pass environment variables to your container: -->
    <!--
    <EnvironmentVariables>
      <EnvironmentVariable Name="VariableName" Value="VariableValue"/>
    </EnvironmentVariables>
    -->
  </CodePackage>
  .....

How to confirm build result

You can watch your build result logs in VSTS Build page like below, and you also can find your build number like below. The number is used for Docker images tags.

References

• Replace configuration files with environment variables on VSTS tasks - normalian blog
• How to override values of environment variables on VSTS tasks - normalian blog
• How to setup VSTS Private Agent for build Windows Server ver 1709 base Docker images - normalian blog
• How to build ASP.NET Framework Docker images on VSTS build tasks - normalian blog
• How to setup Service Fabric connections on VSTS - normalian blog

Create a Release Definition to use artifacts created by your Build Definition

Choose "Releases" item from the top of VSTS menus, click "+" icon from left side and choose create "Create release definition", so you can find below diagrams.

Next, click "Add artifact", choose "Build" as "Source type" and setup your "Project" and "Source(Build definition)" which you have already created before following this article. Refer to below image if you need.

Next, choose "Add environment" box and choose "Azure Service Fabric Deployment". Note that you need to complete
How to setup Service Fabric connections on VSTS - normalian blog to setup this. Refer to below image and table as you need.

Execute Release definition

After completion to create the Release Definition, click "+Rlease" link on VSTSportal.

Next, you can execute your deployment process by clicking "Deploy" on VSTSportal like below. After execution of the process, you also be able to watch process progress by choosing "Logs" tab on VSTSportal.

This is "Logs" tab on VSTS.

Reference

• Deploy an Azure Service Fabric application with continuous integration (Team Services) | Microsoft Docs
• How to build ASP.NET Framework Docker images on VSTS build tasks - normalian blog
• Create Service Fabric Deployment Package with Docker images on VSTS Build Task - normalian blog
• How to setup Service Fabric connections on VSTS - normalian blog

Overview of Service Fabric, VSTS and Windows Container architecture

At first, refer to the architecture diagram below.

You have to create below resources to setup this architecture.

• Service Fabric cluster
• VSTS Proejct, Build Process and Release Process
• Azure Container Registry
• Virtual Machines for VSTS Private Agent

In this article, you can find references to Service Fabric cluster, VSTS and Virtula Machines. But please create Azure Container Registry for yourself and it should be quite easy.

Edit ServiceManifest.xml of your Serviec Fabric project

At first, make a REST API application and deploy it into your Service Fabric cluster. Note to edit "ServiceManifest.xml" file in your Service Fabric project not to specify actual port like below. This setup is needed to collaborate API Management and Service Fabric.

<Resources><Endpoints><Endpoint Protocol="http"Name="ServiceEndpoint"Type="Input" /></Endpoints></Resources>

Download certificate file to access Service Fabric cluster

You probably created the certificate automatically when you made your Service Fabric cluster. Go to your Service Fabric cluster, choose "security" tab and pick up the certificate thumbprint like below.

Next, download the certificate file as pfx into your machine. Go to KeyVault, choose "certificate" tab, select your certificate and choose "Download in PFX/PEM format" like below.

Save thumbprint and pfx file to use ARM Template in later section.

Deploy new API Management instance by using ARM Template

Download apim.json and apim.parameters.json ARM Templates from service-fabric-api-management/apim.json at master · Azure-Samples/service-fabric-api-management · GitHub. And add '"validateCertificateChain": false into apim.json' if you will use self-signed certificate file like below .

"apiVersion": "2017-03-01",
            "type": "Microsoft.ApiManagement/service/backends",
            "name": "[concat(parameters('apimInstanceName'), '/', parameters('service_fabric_backend_name'))]",
            "dependsOn": ["[resourceId('Microsoft.ApiManagement/service', parameters('apimInstanceName'))]",
                "[resourceId('Microsoft.ApiManagement/service/certificates', parameters('apimInstanceName'), parameters('serviceFabricCertificateName'))]"
            ],
            "properties": {"description": "My Service Fabric backend",
                "url": "fabric:/fake/service",
                "protocol": "http",
                "resourceId": "[parameters('clusterHttpManagementEndpoint')]",
                "tls":{"validateCertificateChain": false},
                "properties": {"serviceFabricCluster": {"managementEndpoints": ["[parameters('clusterHttpManagementEndpoint')]"
                        ],
                        "clientCertificateThumbprint": "[parameters('serviceFabricCertificateThumbprint')]",
                        "serverCertificateThumbprints": ["[parameters('serviceFabricCertificateThumbprint')]"
                        ],
                        "maxPartitionResolutionRetries": 5}}}
        },

Update apim.parameters.json like below.

{"$schema": "http://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {"apimInstanceName": {"value": "sfapim01"
        },
        "subnetName": {"value": "API-Subnet"
        },
        "apimPublisherEmail": {"value": "mymail@address.com"
        },
        "apimSku": {"value": "Developer"
        },
        "serviceFabricCertificateName": {"value": "Daichi Isami"
        },
        "serviceFabricCertificate": {"value": "base64 encoded string of your pfx file. don't insert breaklines"
        },
        "certificatePassword": {"value": ""
        },
        "serviceFabricCertificateThumbprint": {"value": "your Cluster certificates thumbprint"
        },
        "url_path": {"value": "/api/values"
        },
        "clusterHttpManagementEndpoint": {"value": "https://'your cluster name'.westus.cloudapp.azure.com:19080"
        },
        "inbound_policy":{"value": "<policies>\r\n<inbound>\r\n<base />\r\n<set-backend-service backend-id=\"servicefabric\" sf-service-instance-name=\"fabric:/SFApiApp/Web1\" sf-resolve-condition=\"@((int)context.Response.StatusCode != 200)\" />\r\n</inbound>\r\n<backend>\r\n<base />\r\n</backend>\r\n<outbound>\r\n<base />\r\n</outbound>\r\n<on-error>\r\n<base />\r\n</on-error>\r\n</policies>"
        },
        "policies_policy_name": {"value": "policy"
        },
        "apis_service_fabric_app_name": {"value": "service-fabric-app"
        },
        "apim_service_fabric_product_name": {"value": "service-fabric-api-product"
        },
        "service_fabric_backend_name": {"value": "servicefabric"
        },
        "apis_service_fabric_app_name_operation": {"value": "service-fabric-app-operation"
        },
        "vnetName": {"value": "VNet-sf-sample01-1709cluster"
        },
        "vnetVersion": {"value": "2017-03-01"
        },
        "networkSecurityGroupName": {"value": "apim-vnet-security-03"
        },
        "networkSecurityGroupVersion": {"value": "2017-03-01"
        }}

You can put blank for certificatePasswordvalue if you created your certificate file automatically. Refer to commands for base64encode for your certificate below if you need.

$bytes = [System.IO.File]::ReadAllBytes("C:\temp\yourpfxfile.pfx")
$b64 = [System.Convert]::ToBase64String($bytes);
$b64 

It should takes 30 or 40 minutes to complete this deployment.

Access your Service Fabric application via API Management

Go to "Developer Portal" of your API Management, choose "Service Fabric App" among APIs and click "Try it" button. Now, you can send requests to your API application via API Management like below.

Troubleshoot - "Service Fabric exception when trying to resolve partition: A Security error has occurred, failed to verify remote certificate"

You might get error messages below if you use self-signed certificate file.

service-fabric-backend (1371 ms)
{
    "message": "Service Fabric exception when trying to resolve partition: A Security error has occurred, failed to verify remote certificate.",
    "serviceName": {},
    "resourceId": "https://sf-sample01-1709cluster.westus.cloudapp.azure.com:19080",
    "managementEndpoint": [
        "https://sf-sample01-1709cluster.westus.cloudapp.azure.com:19080"
    ]
}

You should forget to update apim.json. Refer to " Deploy new API Management instance by using ARM Template" section in this article.

References

• Integrate Azure Service Fabric with API Management | Microsoft Docs
• GitHub - Azure-Samples/service-fabric-api-management: Azure Resource Manager templates to create a Service Fabric cluster with Azure API Management.

References

• Unable to RDP to Virtual Machine: CredSSP Encryption Oracle Remediation – Azure Virtual Machines

Execute PowerShell scripts inside Azure VMs from Azure Portal

Go to Azure Portal and choose an Azure VM among your VMs, so you can find "Run command" menu from left menus. Next, choose "RunPowerShellScript" menu, so you can execute PowerShell scripts like below.

I have already located a text file into "F;\temp\hello.txt" path on Azure VM before executing above scripts to take this screenshot. This means you can manage files inside Azure VMs.
Here is diagram for this scenario. We send HTTP requests as REST API call to VM Agent and the agent execute your PowerShell scripts inside the VM.

Execute PowerShell scripts inside Azure VMs from client machines

You can execute the scripts with PowerShell command named Invoke-AzureRmVMRunCommand. Here is diagram for this scenario.
Follow commands snippet below in your local machine PowerShell ISE to execute your scripts inside VM.

$rgname = 'your vm resource group'
$vmname = 'your vm name'
$localmachineScript = 'PowerShell script file on your local machine like script-test.ps1'
Invoke-AzureRmVMRunCommand -ResourceGroupName $rgname -Name $vmname -CommandId 'RunPowerShellScript' -ScriptPath $localmachineScript -Parameter @{"arg1" = "var1";"arg2" = "var2"} -Debug 

Confirm your Azure PowerShell module version and authentication if the script doesn't work well.

Execute PowerShell scripts inside Azure VMs from Azure Automation

At first, update Azure Automation modules for your Azure Automation account. Go to your Azure Automation account and choose "Modules" menu from left menu and click "Update Azure Modules" to be latest versions like below.

Next, It's needed to locate your script into downloadable place such like Azure Storage, because we can't locate any files into Azure Automation runtime environment. In this case, I have located a script file into "https://change-your-storage-account-name.blob.core.windows.net/scripts/script-test.ps1" and the content is same with above. Here is diagram for this scenario.

Finally, create a Runbook for the script like below and execute it. This need to authenticate to Azure AD.

$connection = Get-AutomationConnection -Name "AzureRunAsConnection"
Write-Output $connection
Add-AzureRMAccount -ServicePrincipal -Tenant $connection.TenantID -ApplicationId $connection.ApplicationID -CertificateThumbprint $connection.CertificateThumbprint

$rgname = 'your vm resource group'
$vmname = 'your vm name'
$localmachineScript = 'PowerShell script file on your local machine like script-test.ps1'
wget "https://automationbackupstorage.blob.core.windows.net/scripts/$localmachineScript" -outfile $localmachineScript 
Invoke-AzureRmVMRunCommand -ResourceGroupName $rgname -Name $vmname -CommandId 'RunPowerShellScript' -ScriptPath $localmachineScript -Parameter @{"arg1" = "var1";"arg2" = "var2"} -Debug 

How to handle exceptions for the scripts inside Azure VMs

It's needed to handle errors when you will integrate this scripts execution into your workflow. I have updated 'script-test.ps1' script like below,

cd F:\temp
type hello.txt
throw "Error trying to do a task @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@"

Here is a result of the script execution.

PS D:\temp> $rgname = 'your vm resource group'
$vmname = 'your vm name'
$localmachineScript = 'script-test.ps1'
$result = Invoke-AzureRmVMRunCommand -ResourceGroupName $rgname -Name $vmname -CommandId 'RunPowerShellScript' -ScriptPath $localmachineScript -Parameter @{"arg1" = "var1";"arg2" = "var2"} -Debug 
DEBUG: 6:28:11 PM - InvokeAzureRmVMRunCommand begin processing with ParameterSet 'DefaultParameter'.

...

DEBUG: ============================ HTTP REQUEST ============================

HTTP Method:
POST

Absolute Uri:
...

Headers:
x-ms-client-request-id        : f0edfe29-5abf-4d7f-9d83-8c98b3e59891
accept-language               : en-US

Body:
{
  "commandId": "RunPowerShellScript",
  "script": [
    "cd F:\\temp",
    "type hello.txt",
    "throw \"Error trying to do a task @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@\""
  ],
  "parameters": [
    {
      "name": "arg1",
      "value": "var1"
    },
    {
      "name": "arg2",
      "value": "var2"
    }
  ]
}


DEBUG: ============================ HTTP RESPONSE ============================

Status Code:
OK

...

Body:
{
  "startTime": "2018-06-26T18:28:14.5508701-07:00",
  "endTime": "2018-06-26T18:28:36.3646186-07:00",
  "status": "Failed",
  "error": {
    "code": "VMExtensionProvisioningError",
    "message": "VM has reported a failure when processing extension 'RunCommandWindows'. Error message: \"Finished executing command\"."
  },
  "name": "bc901040-54ad-4ff1-a8ee-c9794b7a34cb"
}


DEBUG: AzureQoSEvent: CommandName - Invoke-AzureRmVMRunCommand; IsSuccess - True; Duration - 00:00:33.0677753; Exception - ;
DEBUG: Finish sending metric.
DEBUG: 6:28:45 PM - InvokeAzureRmVMRunCommand end processing.
DEBUG: 6:28:45 PM - InvokeAzureRmVMRunCommand end processing.
Invoke-AzureRmVMRunCommand : Long running operation failed with status 'Failed'. Additional Info:'VM has reported a failure when processing extension 'RunCommandWindows'. Error message: 
"Finished executing command".'
ErrorCode: VMExtensionProvisioningError
ErrorMessage: VM has reported a failure when processing extension 'RunCommandWindows'. Error message: "Finished executing command".
StartTime: 6/26/2018 6:28:14 PM
EndTime: 6/26/2018 6:28:36 PM
OperationID: bc901040-54ad-4ff1-a8ee-c9794b7a34cb
Status: Failed
At line:1 char:1
+ Invoke-AzureRmVMRunCommand -ResourceGroupName $rgname -Name $vmname - ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : CloseError: (:) [Invoke-AzureRmVMRunCommand], ComputeCloudException
    + FullyQualifiedErrorId : Microsoft.Azure.Commands.Compute.Automation.InvokeAzureRmVMRunCommand
 
DEBUG: AzureQoSEvent: CommandName - Invoke-AzureRmVMRunCommand; IsSuccess - False; Duration - 00:00:33.0677753; Exception - Microsoft.Azure.Commands.Compute.Common.ComputeCloudException: Long
 running operation failed with status 'Failed'. Additional Info:'VM has reported a failure when processing extension 'RunCommandWindows'. Error message: "Finished executing command".'
ErrorCode: VMExtensionProvisioningError
ErrorMessage: VM has reported a failure when processing extension 'RunCommandWindows'. Error message: "Finished executing command".
StartTime: 6/26/2018 6:28:14 PM
EndTime: 6/26/2018 6:28:36 PM
OperationID: bc901040-54ad-4ff1-a8ee-c9794b7a34cb
Status: Failed ---> Microsoft.Rest.Azure.CloudException: Long running operation failed with status 'Failed'. Additional Info:'VM has reported a failure when processing extension 'RunCommandWi
ndows'. Error message: "Finished executing command".'
   at Microsoft.Rest.ClientRuntime.Azure.LRO.AzureLRO`2.CheckForErrors()
   at Microsoft.Rest.ClientRuntime.Azure.LRO.AzureLRO`2.<StartPollingAsync>d__17.MoveNext()
...
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Management.Compute.VirtualMachinesOperationsExtensions.RunCommand(IVirtualMachinesOperations operations, String resourceGroupName, String vmName, RunCommandInput paramet
ers)
   at Microsoft.Azure.Commands.Compute.Automation.InvokeAzureRmVMRunCommand.<ExecuteCmdlet>b__0_0()
   at Microsoft.Azure.Commands.Compute.ComputeClientBaseCmdlet.ExecuteClientAction(Action action)
   --- End of inner exception stack trace ---
   at Microsoft.Azure.Commands.Compute.ComputeClientBaseCmdlet.ExecuteClientAction(Action action)
   at Microsoft.WindowsAzure.Commands.Utilities.Common.AzurePSCmdlet.ProcessRecord();
DEBUG: Finish sending metric.
DEBUG: 6:28:47 PM - InvokeAzureRmVMRunCommand end processing.
DEBUG: 6:28:47 PM - InvokeAzureRmVMRunCommand end processing.

PS D:\temp> $result

It seems to be difficult to handle the errors, because there are no contents including message of "Error trying to do a task @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@". In this case, -ErrorVariable option should be useful. Update the script again and execute it like below.

PS D:\temp> $rgname = 'your vm resource group'
$vmname = 'your vm name'
$localmachineScript = 'script-test.ps1'
Invoke-AzureRmVMRunCommand -ResourceGroupName $rgname -Name $vmname -CommandId 'RunPowerShellScript' -ScriptPath $localmachineScript -Parameter @{"arg1" = "var1";"arg2" = "var2"} -ErrorVariable result
echo "============================="
$result.Count
echo "============================="
$result
echo "============================="
$result[1]

Invoke-AzureRmVMRunCommand : Long running operation failed with status 'Failed'. Additional Info:'VM has reported a failure when processing extension 'RunCommandWindows'. Error message: 
"Finished executing command".'
ErrorCode: VMExtensionProvisioningError
ErrorMessage: VM has reported a failure when processing extension 'RunCommandWindows'. Error message: "Finished executing command".
StartTime: 6/26/2018 6:35:02 PM
EndTime: 6/26/2018 6:35:17 PM
OperationID: 2ea46d42-2523-4f23-9135-9a595f62f656
Status: Failed
At line:1 char:1
+ Invoke-AzureRmVMRunCommand -ResourceGroupName $rgname -Name $vmname - ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : CloseError: (:) [Invoke-AzureRmVMRunCommand], ComputeCloudException
    + FullyQualifiedErrorId : Microsoft.Azure.Commands.Compute.Automation.InvokeAzureRmVMRunCommand
 
=============================
1
=============================
Invoke-AzureRmVMRunCommand : Long running operation failed with status 'Failed'. Additional Info:'VM has reported a failure when processing extension 'RunCommandWindows'. Error message: 
"Finished executing command".'
ErrorCode: VMExtensionProvisioningError
ErrorMessage: VM has reported a failure when processing extension 'RunCommandWindows'. Error message: "Finished executing command".
StartTime: 6/26/2018 6:35:02 PM
EndTime: 6/26/2018 6:35:17 PM
OperationID: 2ea46d42-2523-4f23-9135-9a595f62f656
Status: Failed
At line:1 char:1
+ Invoke-AzureRmVMRunCommand -ResourceGroupName $rgname -Name $vmname - ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : CloseError: (:) [Invoke-AzureRmVMRunCommand], ComputeCloudException
    + FullyQualifiedErrorId : Microsoft.Azure.Commands.Compute.Automation.InvokeAzureRmVMRunCommand
 
=============================

PS D:\temp> $result

Unfortunately, we can't take error messages from inside PowerShell scripts but we can find there are errors or not. We should use both Azure Automation logs and logs inside Azure VMs.

References

• https://www.petri.com/execute-scripts-inside-azure-vms-portal
• https://docs.microsoft.com/en-us/azure/virtual-machines/windows/run-command
• https://docs.microsoft.com/en-us/powershell/module/azurerm.compute/invoke-azurermvmruncommand?view=azurermps-6.3.0
• https://blogs.msdn.microsoft.com/powershell/2006/11/02/erroraction-and-errorvariable/

Step by Step to step Windows Server version 1709 based VM as Private Agent

You need to create new Virtual Machine on Azure Portal. Choose "Windows Server, version 1709 with Containers" for base VM, because it contains "docker.exe" command. But keep in mind the image doesn't contain "docker-compose".

You don't need to add special settings when you create VMs, but don't setup Network Security Group as completely closed to enable accessible VSTS service.

Access the VM using Remote Desktop and install Visual Studio 2017 into the VM, because the VM doesn't contain MSBuild and other commands for VSTS Build/Release processes. Follow below commands.

C:\Users\azureuser>powershell
Windows PowerShell
Copyright (C) Microsoft Corporation. All rights reserved.

PS C:\Users\azureuser> curl https://aka.ms/vs/15/release/vs_community.exe -O vs_community.exe
PS C:\Users\azureuser> dir


    Directory: C:\Users\azureuser


Mode                LastWriteTime         Length Name
----                -------------         ------ ----
d-r---        3/22/2018   4:19 PM                3D Objects
d-r---        3/22/2018   4:19 PM                Contacts
d-r---        3/22/2018   4:19 PM                Desktop
d-r---        3/22/2018   4:19 PM                Documents
d-r---        3/22/2018   4:19 PM                Downloads
d-r---        3/22/2018   4:19 PM                Favorites
d-r---        3/22/2018   4:19 PM                Links
d-r---        3/22/2018   4:19 PM                Music
d-r---        3/22/2018   4:19 PM                Pictures
d-r---        3/22/2018   4:19 PM                Saved Games
d-r---        3/22/2018   4:19 PM                Searches
d-r---        3/22/2018   4:19 PM                Videos
-a----        3/22/2018   4:24 PM        1180608 vs_community.exe

PS C:\Users\azureuser> .\vs_community.exe

I chose below settings in my case, but change the settings for your environment if you need.

After Visual Studio installation has completed, add MSBuild execution folder path into PATH environment variable like below.

PS C:\Users\azureuser> setx /M PATH "%PATH%;C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\MSBuild\15.0\B
in"

SUCCESS: Specified value was saved.
PS C:\Users\azureuser> 

Next, you also need to add "docker-compose" to build with it, because this base VM contains only just "docker" command. Follow below commands to install it.

PS C:\> [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
PS C:\> Invoke-WebRequest "https://github.com/docker/compose/releases/download/1.20.0/docker-compose-Windows-x86_64.exe" -UseBasicParsing -OutFile $Env:ProgramFiles\docker\docker-compose.exe
PS C:\>

Finally, you need to setup VSTS Private Agent. Refer to How to setup your CentOS VMs as VSTS Private Agent - normalian blog to pick up "access token" for setup Private Agent. Note you need to setup user account of this Private Agent as "NT AUTHORITY\SYSTEM".

PS C:\agent> .\config.cmd

>> Connect:

Enter server URL > https://"your vsts account name".visualstudio.com
Enter authentication type (press enter for PAT) >
Enter personal access token > ****************************************************
Connecting to server ...

>> Register Agent:

Enter agent pool (press enter for default) > "Your Agent Pool Name"
Enter agent name (press enter for VSTSPAVM01) >
Scanning for tool capabilities.
Connecting to the server.
Successfully added the agent
Testing agent connection.
Enter work folder (press enter for _work) >
2018-03-18 05:09:24Z: Settings Saved.
Enter run agent as service? (Y/N) (press enter for N) > Y
Enter User account to use for the service (press enter for NT AUTHORITY\NETWORK SERVICE) > NT AUTHORITY\SYSTEM
Granting file permissions to 'NT AUTHORITY\SYSTEM'.
Service vstsagent.daisami-online.VSTSPAVM01 successfully installed
Service vstsagent.daisami-online.VSTSPAVM01 successfully set recovery option
Service vstsagent.daisami-online.VSTSPAVM01 successfully configured
Service vstsagent.daisami-online.VSTSPAVM01 started successfully
PS C:\agent>

Now, you can choose your Private Agent in your VSTS Build/Release processes.

Note: What will be error messages if you haven't complete docker-compose

You will get "##[error]Unhandled: Failed which: Not found docker: null" message from your VSTS build task.

2018-03-17T21:07:37.4182183Z ##[section]Starting: Build an image
2018-03-17T21:07:37.4186946Z ==============================================================================
2018-03-17T21:07:37.4187316Z Task         : Docker
2018-03-17T21:07:37.4187693Z Description  : Build, tag, push, or run Docker images, or run a Docker command. Task can be used with Docker or Azure Container registry.
2018-03-17T21:07:37.4188244Z Version      : 0.3.10
2018-03-17T21:07:37.4188534Z Author       : Microsoft Corporation
2018-03-17T21:07:37.4188879Z Help         : [More Information](https://go.microsoft.com/fwlink/?linkid=848006)
2018-03-17T21:07:37.4189247Z ==============================================================================
2018-03-17T21:07:37.6890544Z ##[error]Unhandled: Failed which: Not found docker: null
2018-03-17T21:07:37.6953107Z ##[section]Finishing: Build an image

Note: What will be error messages if you build Windows Server ver 1709 images with "Hosted VS2017" agent

You will get "The following Docker images are incompatible with the host operating system: [microsoft/aspnet:4.7.1-windowsservercore-1709]. Update the Dockerfile to specify a different base image." message from your VSTS build task.

2018-03-17T21:02:41.2336315Z 
2018-03-17T21:02:41.2336964Z Build FAILED.
2018-03-17T21:02:41.4137038Z 
2018-03-17T21:02:41.4138305Z "D:\a\1\s\Trunk\SFwithASPNetApp\SFwithASPNetApp.sln" (default target) (1) ->
2018-03-17T21:02:41.4139002Z "D:\a\1\s\Trunk\SFwithASPNetApp\docker-compose.dcproj" (default target) (3) ->
2018-03-17T21:02:41.4139575Z (DockerComposeBuild target) -> 
<b>2018-03-17T21:02:41.4141190Z   C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\MSBuild\Sdks\Microsoft.Docker.Sdk\build\Microsoft.Docker.targets(111,5): error : The following Docker images are incompatible with the host operating system: [microsoft/aspnet:4.7.1-windowsservercore-1709]. Update the Dockerfile to specify a different base image. See http://aka.ms/DockerToolsTroubleshooting for more details. [D:\a\1\s\Trunk\SFwithASPNetApp\docker-compose.dcproj]</b>
2018-03-17T21:02:41.4142524Z 
2018-03-17T21:02:41.4142996Z     0 Warning(s)
2018-03-17T21:02:41.4143435Z     1 Error(s)
2018-03-17T21:02:41.4143702Z 
2018-03-17T21:02:41.4144142Z Time Elapsed 00:14:00.47
2018-03-17T21:02:43.2578358Z ##[error]Process 'msbuild.exe' exited with code '1'.
2018-03-17T21:02:44.0944814Z ##[section]Finishing: Build solution
2018-03-17T21:02:44.1143215Z ##[section]Starting: Post Job Cleanup
2018-03-17T21:02:44.1300074Z Cleaning any cached credential from repository: US-Crackle-Demo-Projects (Git)
2018-03-17T21:02:44.1413004Z ##[command]git remote set-url origin https://daisami-online.visualstudio.com/_git/US-Crackle-Demo-Projects
2018-03-17T21:02:44.3340613Z ##[command]git remote set-url --push origin https://daisami-online.visualstudio.com/_git/US-Crackle-Demo-Projects
2018-03-17T21:02:44.3757483Z ##[section]Finishing: Post Job Cleanup
2018-03-17T21:02:44.4763369Z ##[section]Finishing: Job

Note: What will be error messages if you setup Private Agent account as "NT AUTHORITY\NETWORK SERVICE"

Your VM can't access" //./pipe/docker_engine: " and the build tasks will be failed.

DockerGetServiceReferences:
docker-compose -f "C:\agent\_work\1\s\Trunk\SFwithASPNetApp\docker-compose.yml" -f "C:\agent\_work\1\s\Trunk\SFwithASPNetApp\docker-compose.override.yml" -p dockercompose13733567670188849996 --no-ansi config
C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\MSBuild\Sdks\Microsoft.Docker.Sdk\build\Microsoft.VisualStudio.Docker.Compose.targets(195,5): Error MSB4018: The "GetServiceReferences" task failed unexpectedly.
C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\MSBuild\Sdks\Microsoft.Docker.Sdk\build\Microsoft.VisualStudio.Docker.Compose.targets(195,5): error MSB4018: The "GetServiceReferences" task failed unexpectedly. [C:\agent\_work\1\s\Trunk\SFwithASPNetApp\docker-compose.dcproj]
C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\MSBuild\Sdks\Microsoft.Docker.Sdk\build\Microsoft.VisualStudio.Docker.Compose.targets(195,5): error MSB4018: Microsoft.Docker.Utilities.CommandLineClientException: error during connect: Get http://%2F%2F.%2Fpipe%2Fdocker_engine/v1.30/version: open //./pipe/docker_engine: Access is denied. In the default daemon configuration on Windows, the docker client must be run elevated to connect. This error may also indicate that the docker daemon is not running.. [C:\agent\_work\1\s\Trunk\SFwithASPNetApp\docker-compose.dcproj]
C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\MSBuild\Sdks\Microsoft.Docker.Sdk\build\Microsoft.VisualStudio.Docker.Compose.targets(195,5): error MSB4018: [C:\agent\_work\1\s\Trunk\SFwithASPNetApp\docker-compose.dcproj]
C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\MSBuild\Sdks\Microsoft.Docker.Sdk\build\Microsoft.VisualStudio.Docker.Compose.targets(195,5): error MSB4018: For more troubleshooting information, go to http://aka.ms/DockerToolsTroubleshooting ---> Microsoft.Docker.Utilities.CommandLineClientException: error during connect: Get http://%2F%2F.%2Fpipe%2Fdocker_engine/v1.30/version: open //./pipe/docker_engine: Access is denied. In the default daemon configuration on Windows, the docker client must be run elevated to connect. This error may also indicate that the docker daemon is not running. [C:\agent\_work\1\s\Trunk\SFwithASPNetApp\docker-compose.dcproj]

References

• Install Docker Compose | Docker Documentation
• Use command-line parameters to install Visual Studio - Visual Studio | Microsoft Docs

Jenkins Setup - if you need

This step isn't needed if you have already setup Jenkins in your environment. Refer to contents below if you want to setup it on Microsoft Azure.

• Jenkins on Azure | Microsoft Docs
• Tutorial - Create a development pipeline in Azure with Jenkins | Microsoft Docs

Install XFrame Filter Plugin into Jenkin and enable to use iFrame

Install a plugin called "XFrame Filter Plugin" into your Jenkins, because it needs to enable iFrame to embed your Jenkins portal into VSTSdashboard.
Go to your Jenkins portal and choose "Manage Jekins" - "Manage Plugins" like below.

Next, click "Available" and input "XFrame" to find "XFrame Filter Plugin". You can install the plugin easily just enable checkbox and click "Downlaod now and install after restart".

• CollabNet User Information Center
• https://wiki.jenkins.io/display/JENKINS/XFrame+Filter+Plugin

After completion of the install, you need to configure the plugin. Go to your Jenkins portal again and choose "Manage Jekins" - "Configure System" like below.

Find the plugin among them, enable the feature and input your VSTS account URL into "X-Frame-Options Options" box like below.

Embed Jenkins portal using by “Embedded Webpage” into VSTSDashboard

Next, you need to go to your VSTSdashboard and add "Embedd Webpage" like below,

Configure "Embedd Webpage" to input your Jenkins URL like below.

I believe your browser doesn't trust your Jenkins URL. so you also need to enable untrusted contents like below,

Finally, you can watch Jenkins portal on VSTSdashboard, so you can watch both VSTS and Jenkins pipeline like below.