iOS Automated Builds with Xcode4
I just spent the last (frustrating) week or more at work setting up our build server for some iPhone / iPad projects. It’s so complicated, and there’s so much misinformation out there, that it’s worth sharing some tips here which will hopefully be useful for us to read back on. And maybe this’ll save someone some hassles out there too.
In this article, i won’t give you a complete build script, as everyone’s environments are subtly different, but instead i hope to impart the knowledge and tips and pitfalls that’ll make it reasonably straightforwards for you to create your own script without all the horrible hassles I had.
Aims
Our final goal was to have two files arrive in our Dropbox each morning: zip files containing a dev branch build and a stage branch build. These zip files are to contain Xcode archives (for later app-store submission) and the ipa files (for TestFlight).
Our setup and requirements can be described thus:
- A ruby script is used to control the build process.
- Two Git branches: dev and stage. The idea is that all new features get added to the dev branch. Any bug fixes get added to the stage branch. Every two weeks (end of the sprint), with the testers' blessing, we merge all the new features from dev into stage, and then merge all the bug fixes from stage into dev.
- We use TestFlight to provide the builds to the testers. For this, we need signed IPA files to submit to test flight.
- We want to produce an Xcode archive (.xcarchive) – this is used for submission to the app store, and for sending to clients. With this file, we can double click it on any mac, which will import that exact build into Xcode, and we can then re-sign it with the App-store distribution profile and submit to the app store (so long as the Bundle Seed ID is the same – more on this later). This is so that we can ensure that what we submit to the app store is exactly the same build that was tested.
- We’re using Xcode 4, and don’t need to support pre-iOS 4, so that simplifies things a lot (e.g. no need for CODE_SIGN_ENTITLEMENTS plists or Ad-hoc build configurations).
Provisioning
It’s worth going over provisioning: certificates, keys, profiles, etc. I’ll try to describe how it affects you, but you should really also read Technote 2250.
In short, it works like this:
Your private key signs a certificate, which validates provisioning profiles.
Keys and Certificates
Private keys are generated using the Keychain on your mac. Using this key, you request a certificate. So certificates only work if you have the matching private key installed – be aware of this, you cannot simply download the certificate from the iOS provisioning portal if you don’t have the private key. To copy a key to another computer, e.g. your build machine, right click > Export the key from within the Keychain, and make sure you export it as a .p12 file.
If, in the Keychain, you can’t see a little triangle next to your certificate, then it means your private key isn’t installed. Delete the certificate (it’s useless without the key), export the key from the original mac it was generated on, and import it on the other machine before you try importing the certificate.
You may see ‘Code sign identity’ mentioned in the project settings later on. This is a synonym for your certificate.
You’ll only need one distribution certificate, it’ll work for all your provisioning profiles (both adhoc and appstore).
Keep in mind that everything i’m discussing is to do with distribution, not development. Always ensure you’re in the ‘distribution’ tab in the iOS provisioning portal.)
Provisioning Profiles
You need two profiles: ad-hoc (TestFlight) and app-store.
When you generate a profile, it will be linked to your certificate, and will only work on a mac that has the cert (and key) installed. When you generate new profiles, however, you don’t need to re-generate the certificate – it isn’t necessary.
Profiles only work for a specific app id (discussed later), or they can work for a wildcard id. For this article, i’ll only be discussing those with specific app id’s.
When profiles are generated, they are locked to a Bundle Seed ID and a Bundle ID. When you compile an app, you must use a profile with a matching bundle id, or the profile won’t work. Your app’s bundle ID can be found in your Info.plist file.
Once an app has been signed with one profile (e.g. your ad-hoc TestFlight profile), you can re-sign it with another profile (e.g. your app-store profile) – IF the Bundle Seed ID’s match.
Provisioning profiles have a unique UUID to identify themselves, eg: 44A1166C-C096-4112-B3E0-9081520CBABC. This UUID can be extracted in the build script, as it changes each time you add a new device to your profile, and you don’t want to have to hunt down the UUID each time you re-generate the profile. It’s far easier to simply check the *..mobileprovision file into your repo, and let the build script look up the UDID, install the profile, and update the build settings accordingly.
App ID’s
An App ID = the Bundle Seed ID + ‘.’ + Bundle ID.
The Bundle Seed ID is also known as the App ID prefix, or the Team ID. When you generate a profile, it will be given your Team ID as it’s Bundle Seed ID. The Bundle Seed ID looks like: 5M3M5W7ABC.
Nowhere in your app’s configuration do you need to specify a bundle seed id, and the only time it matters is for app store submission time: when you want to submit the exact same build that the testers tested. When you go to re-sign the archive with your app store distribution profile, it’ll need to have the same Bundle Seed ID as in the ad-hoc profile you originally signed it with for testing.
The Bundle ID looks like: ‘com.splinter.myawesomeapp’. This is specified in your app’s Info.plist file. When you do a release/archive compile, you’ll need to sign against a provisioning profile that matches this bundle id. Eg, if you have two apps, with the following bundle id’s:
- com.splinter.world-domination-app
- com.splinter.servitude-app
In that case, you’ll need different profiles, with bundle id’s to match. (Or you can use wildcard profiles, but i won’t go into them here). The important thing to remember is: the bundle id in the profile must match your app’s bundle id, or it won’t compile.
Server setup
The server I use is a mac mini, with Bamboo. Personally, i don’t really recommend Bamboo as it over-complicates the git repository setup which makes it difficult to push commits up to your git server – which we use to make version number commits. You can use cron jobs, or Hudson, or any other CI setup you wish.
TestFlight
TestFlight has a brilliantly simply upload API that uses Curl from the command line to submit newer versions of your app. The only complication is if you wish to have both dev and stage branches available to your testers: since TestFlight groups builds by bundle name, only the most recent build will be available.
So, to get around this, I use ‘PlistBuddy’ to modify the app name and bundle ID prior to building the dev branch version. Please note that if you do this, you’ll also need to use a provisioning profile that matches the new bundle id.
Build Steps
The steps I follow for making a build are as follows:
- Increment the version numbers in the Info.plist file
- Check into git the updated Info.plist
- Override the bundle id and name, if it is the dev branch
- Install the provisioning profile
- Use zerg_xcode to update the project file, to use the desired certificate and provisioning profile
- Build the .xcarchive
- Create the .ipa file
- Upload to TestFlight
Incrementing versions
Our app’s versioning (for internal development and testing) follows a common major.minor.buildnumber strategy. Eg: 1.4.123. Each time the build runs, it increments the build number, updates the Info.plist, and checks it into git. The git commit uses the version number as the commit message so that later on we can see the exact code that was used to produce a given build.
To do this, PlistBuddy is used to extract the old version number, which is then incremented and saved to the Info.plist. In the ruby script, it looks like this:
oldVersion = `/usr/libexec/Plistbuddy -c "print :CFBundleVersion" Info.plist`.strip
This returns a string, such as ‘1.2.3’. We then need to increment this, which the following does:
components = oldVersion.split('.')
newBuild = components.pop.to_i + 1
components.push(newBuild).join('.')
We then need to save this new version number back to the Info.plist, which is a simple matter of using Plistbuddy again. So the complete function to read, increment, and save the file is as follows:
def incrementBundleVersion
oldVersion = `/usr/libexec/Plistbuddy -c "print :CFBundleVersion" Info.plist`.strip
components = oldVersion.split('.')
newBuild = components.pop.to_i + 1
version = components.push(newBuild).join('.')
system("/usr/libexec/PlistBuddy -c \"Set :CFBundleVersion #{version}\" Info.plist")
system("/usr/libexec/PlistBuddy -c \"Set :CFBundleShortVersionString #{version}\" Info.plist")
end
Don’t forget to add the code to check this back into Git after doing this.
Overriding bundle id and names
At this step, you can override the bundle id and bundle names. This is useful if you want to have different branches of your code available to your testers on TestFlight, such as dev and stage builds. Since TestFlight groups builds with the same bundle id/name, you’ll have to change it to do this.
PlistBuddy is used for this:
def overridePlistBundleName(infoPlist, name)
system("/usr/libexec/PlistBuddy -c \"Set :CFBundleDisplayName #{name}\" \"#{infoPlist}\"")
system("/usr/libexec/PlistBuddy -c \"Set :CFBundleName #{name}\" \"#{infoPlist}\"")
end
def overridePlistBundleId(infoPlist, id)
system("/usr/libexec/PlistBuddy -c \"Set :CFBundleIdentifier #{id}\" \"#{infoPlist}\"")
end
Keep in mind of course that if you change the bundle id, you’ll need to use a different provisioning profile that matches the new bundle id.
Installing profiles
Your *.mobileprovision files should be checked into your repo, so that whenever you need to add a new device to your profile it is a simple enough job to update the profile on the iOS Provisioning Portal, download the updated profile, and check it in.
However, your profile will need to be installed. To do this, you first need to get the UUID out of the file, with code such as follows:
# Gets the uuid from a provisioning profile
def getProfileId(provisioningProfile)
File.open(provisioningProfile,"r") {|f|
raw = f.read
nice = raw.gsub("\n","").gsub("\r","").
gsub("\t","").gsub(" ","")
matches = nice.scan(/UUID<\/key>(.*)<\/string>/)
return matches.first.first
}
abort("Couldn't find UUID in: " + provisioningProfile)
end
Once you have the profile’s UUID, you can install it automatically, so that nobody has to log into the build server to install new profiles every time that someone updates the profiles. The code to install a profile:
# Installs a profile so we can compile against it and returns it's uuid
def installProfile(provisioningProfile)
uuid = getProfileId(provisioningProfile)
dest = File.expand_path("~/Library/MobileDevice/Provisioning Profiles/#{uuid}.mobileprovision")
system("cp \"#{provisioningProfile}\" \"#{dest}\"")
uuid
end
After installing the profile, you’ll need to store it’s UUID somewhere for later when it comes to updating the project file so that this profile is explicitly selected.
Zerg
Zerg_xcode is a ruby gem that is used to modify the Xcode project file so that we can explicitly set the provisioning profile and code signing identity (aka certificate).
It is unfortunate that we cannot set these as command line arguments and have to resort to updating the project files, but xcodebuild ignores build settings when archiving.
It is important to manually override the project file, as it gives us much more control over selecting the correct profile and certificate. This is necessary when we need to use different profiles for the different branches, because the bundle id’s are different for TestFlight’s sake (also, it prevents us from submitting a dev build to the app store!).
Here is the code I use to update the project file to manually set the certificate name (aka code sign identity) and profile’s UUID. Note that the certificate name is something like ‘iPhone Distribution: MY COMPANY’, which you can find in your keychain.
require 'rubygems'
require 'zerg_xcode' # https://github.com/ddribin/zerg_xcode
# Update the project to set the profile etc because
# xcodebuild only pays lip service to command line args
def doctorProject(target, identity, profileUuid)
project = ZergXcode.load("MyProject.xcodeproj")
configuration = 'Release'
build_configurations = project["buildConfigurationList"]
["buildConfigurations"]
configuration_object = build_configurations.select {|item|
item['name'] == configuration }[0]
configuration_object["buildSettings"]
["PROVISIONING_PROFILE"] = profileUuid
configuration_object["buildSettings"]
["PROVISIONING_PROFILE[sdk=iphoneos*]"] = profileUuid
configuration_object["buildSettings"]
["CODE_SIGN_IDENTITY"] = identity
configuration_object["buildSettings"]
["CODE_SIGN_IDENTITY[sdk=iphoneos*]"] = identity
target = project["targets"].select {|item|
item['name'] == target }[0]
build_configurations = target["buildConfigurationList"]
["buildConfigurations"]
configuration_object = build_configurations.select {|item|
item['name'] == configuration }[0]
configuration_object["buildSettings"]
["PROVISIONING_PROFILE[sdk=iphoneos*]"] = profileUuid
configuration_object["buildSettings"]
["CODE_SIGN_IDENTITY[sdk=iphoneos*]"] = identity
project.save!
end
Build the .xcarchive
This is where the rubber hits the road: building the archive. Thankfully, with all the prerequisites taken care of by now, it’s a simple matter of the following:
xcodebuild -target "My Target" -scheme "My Scheme" -configuration Release clean archive
Note that we’re doing an ‘archive’, not a ‘build. It’s important to note the difference here. An archive builds a ’.xcarchive', whereas a build produces a .app and a .dsym. However, the .app and .dsym are fairly limited on their own – there’s not much you can do with them. An xcarchive is much more useful: you can import it into Xcode on another mac, re-sign it, and submit to the app store.
It is worth noting than an xcarchive is simply a package folder which contains the .app, .dsym, and a plist descriptor. So if you need the .dsym to symbolicate crashes later on, you can get it out of the xcarchive.
Xcodebuild will put the .xcarchive in a folder such as:
~/Library/Developer/Xcode/Archives/yyyy-mm-dd/MyArchive.xcarchive
To make it easier to find the just-generated archive, in my build script, before the xcodebuild step, I rename the Archives folder. Then after running the build, there will only be one xcarchive file inside the archives folder, which i grab, and then rename the Archives folder back as it was before building. This means that you can only run one build at a time on your build server, but it’s the best outcome given that you can’t control where the file is output.
The .xcarchive is really a folder, which contains the .app folder, among others. It contains symlink(s), which you have to be careful to preserve when copying, moving, and zipping it. If you break the symlink, you will get confusing code signing issues later on when you try to use it.
To preserve the symlinks, I make sure to use ‘mv’ to move the .xcarchive to my desired folder, as I could not get ‘cp’ to maintain relative symlinks. And when zipping it up, ensure that you use the -y option to preserve the symlinks, e.g.:
zip -rqy "MyOutput.zip" "/Folder/With/My/ArchiveAndIpa")
# -r means to recurse through folders, -q quiets the output,
# -y to preserve symlinks so as to not break codesigning
Create the .ipa file
Now that we have the .xcarchive, we need to produce the .ipa which we send to TestFlight for our testers to use. Use a utility called PackageApplication for this. But first you’ll need to find the path to the .app file within the .xcarchive first. I use the following in my ruby build script to do that:
app = Dir[outputFolder+'/*.xcarchive/Products/Applications/*.app'].first
And to generate the IPA filename that’ll go in my output folder:
ipa = outputFolder + '/' + File.basename(app).gsub('.app', '.ipa')
Once you have found those, perform something like the following:
/usr/bin/xcrun -sdk iphoneos PackageApplication
"/Path/To/MyApp.app" -o "/Path/To/MyOutput.ipa"
One interesting point that took a while to realise, is that a lot of examples you’ll see on the internet for how to use PackageApplication are misleading: they show command line arguments for signing with a certificate and profile, but this is simply unnecessary, as your code was already signed as part of the archive process.
Also, be aware that PackageApplication only works if you pass it full, expanded paths for the .app and .ipa. So relative paths, or any path with ~ in it, will fail with no error message, confusingly. So be careful of that pitfall.
Upload to TestFlight
Once you have your ipa file, it is a simple matter of uploading it to TestFlight using their convenient curl api:
system("curl -F file=@\"#{ipa}\" -F api_token='#{token}'
-F team_token='#{team}' -F notes=\"#{notes}\" -F notify=True
-F distribution_lists='#{list}'
\"http://testflightapp.com/api/builds.json\"")
This api is very simple and better documented on TestFlight’s site anyway, so i won’t get into it much here. It’s just worth reiterating that if you want dev and stage builds to be separately available on TestFlight for your testers, you’ll have to change the bundle id and bundle name as described in the section ‘Overriding bundle id and names’.
