Uploading to an S3 bucket from iOS

Sometimes it can be useful to upload files directly to an S3 bucket from an iOS app. Perhaps you’re creating a video app and you need to upload videos to a staging area before your server can grab them. Or perhaps you’re working on an app that is used for managing an S3 bucket’s contents. Or perhaps you don’t want to upload, but you want to use the AWS S3 API for other purposes, such as listing a bucket’s contents.

In this article, I aim to explain how you can talk to the S3 API directly using Objective-C, for you to use in either an iOS or Mac app.

This is all based around the work I’ve done in S3Sync, a Mac console app for uploading to an S3 bucket. You can view it on github as a good example of how to do all this.

Request signing

The S3 api is a fairly typical RESTful api, but the main stumbling block is that each of your requests needs to be signed with an HMAC involving your amazon secret key. Luckily, there is the ever-helpful AFAmazonS3Client which can do the signing for us, when combined with AFNetworking.

Prerequisites

You’ll need AFNetworking (version 2) and AFAmazonS3Client (also version 2) in your app. Cocoapods is probably your best bet for this, or you can download the appropriate tagged versions from their respective github repos and include the files directly, if you’re not comfortable with the modifications that Cocoapods makes to your project files.

Setup

Firstly, you’ll need to instantiate a manager object to use for later requests. You can do this as follows:

AFAmazonS3Manager *s3Manager =
    [[AFAmazonS3Manager alloc] initWithAccessKeyID:@"MYAMAZONKEYID"
                                            secret:@"MYAMAZONSECRET"];
s3Manager.requestSerializer.region = AFAmazonS3USStandardRegion;
s3Manager.requestSerializer.bucket = @"MYBUCKETNAME";

You don’t need to set the region, but perhaps by setting it you can make things faster if you’re eg using a Sydney bucket.

If your bucket is being used to host a website, it will be something like “www.splinter.com.au”.

Note that you’ll probably want to obfuscate the key id and secret in your app, I wrote about this here: Storing Secret Keys.

Getting a list of files from a bucket

To get a list of files (objects, in S3-terminology), you can use the following:

/// Marker is nil for when you call this, it is only used for recursive calls.
- (void)getS3ListFromMarker:(NSString *)marker {
    NSDictionary *params = marker ? @{@"marker": marker} : nil;
    
    [s3Manager GET:@"/" parameters:params
        success:^(AFHTTPRequestOperation *operation, NSXMLParser *responseObject) {

        // Parse the response.
        ListObjects *myParsedObject = ...

        // TODO keep track of the files listed so far, add them to previous.
        
        // Fetch more files if necessary.
        if (myParsedObject.isTruncated) {
            S3Object *lastObject = myParsedObject.objects.lastObject;
            [self getS3ListFromMarker:lastObject.key];
        } else {
            // TODO finished!
        }
    } failure:^(AFHTTPRequestOperation *operation, NSError *error) {
        NSLog(@"Error: %@", error);
    }];
}

I haven’t described how you’ll parse the response, it’s formatted in XML (not JSON, surprisingly!) which is a bit trickier. But it’s fairly straightforwards, and is left as an exercise for the reader (or you could look at S3Sync, link is above). The two main things you need to parse are the list of objects, and the ‘IsTruncated’ field.

The main thing to note here is that the response only returns up to 1000 files at a time. If there are more than that many files, IsTruncated will be true and you’ll need to recursively request again, with the ‘marker’ parameter set to the key of the last object returned in the previous call.

Uploading a file

To upload a file, you’ll firstly need the mime type. If you don’t already know it, a clever way to guess the mime type is to use NSURLConnection with a file url, as below:

// Get the file using url loading mechanisms to get the mime type.
NSMutableURLRequest *fileRequest = [NSMutableURLRequest requestWithURL:
    [NSURL fileURLWithPath:myLocalFileFullPath]];
fileRequest.cachePolicy = NSURLCacheStorageNotAllowed;
NSURLResponse *fileResponse = nil;
NSData *data = [NSURLConnection sendSynchronousRequest:fileRequest
                                     returningResponse:&fileResponse 
                                                 error:nil];
// Mime type is now in fileResponse.MIMEType.

It’s also highly recommended to get the MD5 of the file, and Base-64 encode it, to ensure nothing gets corrupted while uploading. You’ll need to #import <CommonCrypto/CommonCrypto.h> too:

unsigned char md5Buffer[CC_MD5_DIGEST_LENGTH];
CC_MD5(data.bytes, (CC_LONG)data.length, md5Buffer);
NSData *md5data = [NSData dataWithBytes:md5Buffer length:sizeof(md5Buffer)];
NSString *md5Base64 = [md5data base64EncodedStringWithOptions:0];

Next, you need to generate the request. Since we’ll be mutating parameters in the url request, we won’t be able to use AFNetworking’s handy single-shot GET or PUT methods, instead we’ll need to form requests as below:

// Build the un-authed request.
NSURL *url = [s3Manager.baseURL URLByAppendingPathComponent:
    @"somefolder/subfolder/video_to_upload.mp4"];
NSMutableURLRequest *originalRequest = [[NSMutableURLRequest alloc] initWithURL:url];
originalRequest.HTTPMethod = @"PUT";
originalRequest.HTTPBody = data;
[originalRequest setValue:md5Base64 forHTTPHeaderField:@"Content-MD5"];
[originalRequest setValue:fileResponse.MIMEType forHTTPHeaderField:@"Content-Type"];

Note the path component above is the remote path to upload to on the bucket.

Once we’ve created the request, it needs to be ‘signed’ using your secret key. Basically it sets an HTTP header field. This is where AFAmazonS3Client does its magic:

// Sign it.
NSURLRequest *request = [s3Manager.requestSerializer
    requestBySettingAuthorizationHeadersForRequest:originalRequest 
                                             error:nil];

And once that is all done, you can create the AFNetworking request operation and add it to the queue, which starts it:

// Upload it.
AFHTTPRequestOperation *operation = [s3Manager HTTPRequestOperationWithRequest:request success:^(AFHTTPRequestOperation *operation, id responseObject) {
    // Success!
} failure:^(AFHTTPRequestOperation *operation, NSError *error) {
    NSLog(@"Error uploading %@", error);
}];
[s3Manager.operationQueue addOperation:operation];

If you want the code in one chunk, rather than split up and explained here, please check out S3Sync (linked above).

Also, if you want to upload a file > 5MB, you can use S3’s Multipart upload, but I have not covered that here.

That’s all for today, I hope this has been helpful.

Chris Hulbert

(Comp Sci, Hons - UTS)

Software Developer (Freelancer / Contractor) in Australia.

I have worked at places such as Google, Cochlear, Assembly Payments, News Corp, Fox Sports, NineMSN, FetchTV, Coles, Woolworths, Trust Bank, and Westpac, among others. If you're looking for help developing an iOS app, drop me a line!

Get in touch:
[email protected]
github.com/chrishulbert
linkedin

 Subscribe via RSS