vcpkg/scripts/azure-pipelines/osx

vcpkg-eg-mac VMs

Table of Contents

• vcpkg-eg-mac VMs
  • Table of Contents
  • Basic Usage
    • Creating a new Vagrant box
      • VM Software Versions
    • Creating a New Azure Agent Pool
    • Running the VM
  • Getting an Azure Pipelines PAT
  • Setting up a new macOS machine
  • Troubleshooting
  • (Internal) Accessing the macOS fileshare

Basic Usage

The most common operation here is to set up a new VM for Azure pipelines; we try to make that operation as easy as possible. It should take all of three steps, assuming the machine is already set up (or read these instructions for how to set up a machine):

1. Create a new vagrant box
2. Create a new agent pool
3. Setup and run the vagrant VM
4. Update azure-pipelines.yml and azure-pipelines-osx.yml to point to the new macOS pool.

Creating a new Vagrant box

Whenever we want to install updated versions of the command line tools, or of macOS, we need to create a new vagrant box. This is pretty easy, but the results of the creation are not public, since we're concerned about licensing. However, if you're sure you're following Apple's licensing, you can set up your own vagrant boxes that are the same as ours by doing the following:

You'll need some prerequisites:

• An Xcode installer - you can get this from Apple's developer website, although you'll need to sign in first: https://developer.apple.com/downloads
• The software installed by Install-Prerequisites.ps1

If you're updating the CI pool, make sure you update macOS.

First, you'll need to create a base VM; this is where you determine what version of macOS is installed. Follow the Parallels process for creating a macOS VM; this involves updating to whatever version, and then scrolling right until you find "Install macOS from recovery partition".

Once you've done this, you can run through the installation of macOS onto a new VM. You should set the username to vagrant.

Once it's finished installing, make sure to turn on the SSH server. Open System Preferences, then go to Sharing > Remote Login, and turn it on. You'll then want to add the vagrant SSH keys to the VM's vagrant user. Open the terminal application and run the following:

$ # basic stuff
$ date | sudo tee '/etc/vagrant_box_build_time'
$ printf 'vagrant\tALL=(ALL)\tNOPASSWD:\tALL\n' | sudo tee -a '/etc/sudoers.d/vagrant'
$ sudo chmod 0440 '/etc/sudoers.d/vagrant'
$ # then install vagrant keys
$ mkdir -p ~/.ssh
$ curl -fsSL 'https://raw.github.com/mitchellh/vagrant/master/keys/vagrant.pub' >~/.ssh/authorized_keys
$ chmod 0600 ~/.ssh/authorized_keys

Finally, you'll need to install the Parallel Tools. From your host, in the top bar, go to Actions > Install Parallels Tools..., and then follow the instructions.

Now, let's package the VM into a base box. (The following instructions are adapted from these official instructions).

Run the following commands:

$ cd ~/Parallels
$ echo '{ "provider": "parallels" }' >metadata.json
$ tar zcvf <macos version>.box ./metadata.json ./<name of VM>.pvm

This will create a box file which contains all the necessary data. You can delete the metadata.json file after.

Once you've done that, you can upload it to the fileshare, under share/boxes/macos-base, add it to share/boxes/macos-base.json, and finally add it to vagrant:

$ vagrant box add ~/vagrant/share/boxes/macos-base.json

Then, we'll create the final box, which contains all the necessary programs for doing CI work. Copy configuration/Vagrantfile-box.rb as Vagrantfile, and configuration/vagrant-box-configuration.json into a new directory; into that same directory, download the Xcode command line tools dmg, and name it clt.dmg. Then, run the following in that directory:

$ vagrant up
$ vagrant package

This will create a package.box, which is the box file for the base VM. Once you've created this box, if you're making it the new box for the CI, upload it to the fileshare, under share/boxes/macos-ci. Then, add the metadata about the box (the name and version) to share/boxes/macos-ci.json. Once you've done that, add the software versions under VM Software Versions.

VM Software Versions

• 2022-02-04 (minor update to 2022-01-03)
  • macOS: 12.1
  • Xcode CLTs: 13.2
• 2022-01-03:
  • macOS: 12.1
  • Xcode CLTs: 13.2
• 2021-07-27:
  • macOS: 11.5.1
  • Xcode CLTs: 12.5.1
• 2021-04-16:
  • macOS: 11.2.3
  • Xcode CLTs: 12.4
• 2020-09-28:
  • macOS: 10.15.6
  • Xcode CLTs: 12

Creating a New Azure Agent Pool

When updating the macOS machines to a new version, you'll need to create a new agent pool for the machines to join. The standard for this is to name it PrOsx-YYYY-MM-DD, with YYYY-MM-DD the day that the process is started.

In order to create a new agent pool, go to the vcpkg/public project; go to Project settings, then go to Agent pools under Pipelines. Add a new self-hosted pool, name it as above, and make certain to check the box for "Grant access permission to all pipelines".

Once you've done this, you are done; you can start adding new machines to the pool!

Running the VM

First, make sure that your software is up to date:

$ cd ~/vcpkg
$ git fetch
$ git switch -d origin/master
$ ./scripts/azure-pipelines/osx/Install-Prerequisites.ps1

as well as checking to make sure macOS is up to date.

Then, follow the instructions for accessing ~/vagrant/share.

And finally, grab a PAT, update the vagrant box, set up the VM, and run it:

$ vagrant box remove -f vcpkg/macos-ci # This won't do anything if the machine never had a box before
$ vagrant box add ~/vagrant/share/boxes/macos-ci.json
$ ~/vcpkg/scripts/azure-pipelines/osx/Setup-VagrantMachines.ps1 -Date <box version YYYY-MM-DD> -DevopsPat <PAT>
$ cd ~/vagrant/vcpkg-eg-mac
$ vagrant up # if this fails, reboot through the kvm and/or log in interactively, then come back here

Getting an Azure Pipelines PAT

Personal Access Tokens are an important part of this process, and they are fairly easy to generate. On ADO, under the correct project (in vcpkg's case, "vcpkg"), click on the "User Settings" icon, then go to "Personal access tokens". It is the icon to the left of your user icon, in the top right corner.

Then, create a new token, give it a name, make sure it expires quickly, and give it a custom defined scope that includes the "Agent pools: Read & manage" permission (you'll need to "Show all scopes" to access this). You can now copy this token and use it to allow machines to join.

Setting up a new macOS machine

Before anything else, one must download brew and powershell.

$ /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)"
$ brew cask install powershell

Then, we need to download the vcpkg repository:

$ git clone https://github.com/microsoft/vcpkg

Then, we need to mint an SSH key:

$ ssh-keygen
$ cat .ssh/id_rsa.pub

Add that SSH key to authorized_keys on the file share machine with the base box.

Next, install prerequisites:

$ cd vcpkg/scripts/azure-pipelines/osx
$ ./Install-Prerequisites.ps1 -Force

And finally, make sure you can access ~/vagrant/share.

Troubleshooting

The following are issues that we've run into:

• (with a Parallels box) vagrant up doesn't work, and vagrant gives the error that the VM is 'stopped'.
  • Try logging into the GUI with the KVM, and retrying vagrant up.
• (when running a powershell script) The error Failed to initialize CoreCLR, HRESULT: 0x8007001F is printed.
  • Reboot the machine; run

  $ sudo shutdown -r now
  

  and wait for the machine to start back up. Then, start again from where the error was emitted.

(Internal) Accessing the macOS fileshare

The fileshare is located on vcpkgmm-01, under the fileshare user, in the share directory. In order to get sshfs working on the physical machine, You can run Install-Prerequisites.ps1 to grab the right software, then either:

$ mkdir -p ~/vagrant/share
$ sshfs fileshare@vcpkgmm-01:share ~/vagrant/share

If you get an error, that means that gatekeeper has prevented the kernel extension from loading, so you'll need to access the GUI of the machine, go to System Preferences, Security & Privacy, General, unlock the settings, and allow system extensions from the osxfuse developer to run. Then, you'll be able to add ~/vagrant/share as an sshfs.