iOS Chunked Upload
I'm trying to stream contacts from the user's address book to our server. Pulling all the contacts into memory at once can crash or make the device unresponsive. I don't want to incur the overhead of writing all the contacts to a file and uploading a file. I can see the data being sent across the wire, but it looks like it's in an invalid format. The server doesn't recognize a request body.
I'm reading contacts from the address book and writing them to an NSOutputStream. This NSOutputStream shares a buffer with an NSInputStream via this code
Buffering NSOutputStream used as NSInputStream?
//
// NSStream+BoundPairAdditions.m
// WAControls
//
//
#import "NSStream+BoundPairAdditions.h"
#include <sys/socket.h>
static void CFStreamCreateBoundPairCompat(
CFAllocatorRef alloc,
CFReadStreamRef * readStreamPtr,
CFWriteStreamRef * writeStreamPtr,
CFIndex transferBufferSize
)
// This is a drop-in replacement for CFStreamCreateBoundPair that is necessary because that
// code is broken on iOS versions prior to iOS 5.0 <rdar://problem/7027394> <rdar://problem/7027406>.
// This emulates a bound pair by creating a pair of UNIX domain sockets and wrapper each end in a
// CFSocketStream. This won't give great performance, but it doesn't crash!
{
#pragma unused(transferBufferSize)
int err;
Boolean success;
CFReadStreamRef readStream;
CFWriteStreamRef writeStream;
int fds[2];
assert(readStreamPtr != NULL);
assert(writeStreamPtr != NULL);
readStream = NULL;
writeStream = NULL;
// Create the UNIX domain socket pair.
err = socketpair(AF_UNIX, SOCK_STREAM, 0, fds);
if (err == 0) {
CFStreamCreatePairWithSocket(alloc, fds[0], &readStream, NULL);
CFStreamCreatePairWithSocket(alloc, fds[1], NULL, &writeStream);
// If we failed to create one of the streams, ignore them both.
if ( (readStream == NULL) || (writeStream == NULL) ) {
if (readStream != NULL) {
CFRelease(readStream);
readStream = NULL;
}
if (writeStream != NULL) {
CFRelease(writeStream);
writeStream = NULL;
}
}
assert( (readStream == NULL) == (writeStream == NULL) );
// Make sure that the sockets get closed (by us in the case of an error,
// or by the stream if we managed to create them successfull).
if (readStream == NULL) {
err = close(fds[0]);
assert(err == 0);
err = close(fds[1]);
assert(err == 0);
} else {
success = CFReadStreamSetProperty(readStream, kCFStreamPropertyShouldCloseNativeSocket, kCFBooleanTrue);
assert(success);
success = CFWriteStreamSetProperty(writeStream, kCFStreamPropertyShouldCloseNativeSocket, kCFBooleanTrue);
assert(success);
}
}
*readStreamPtr = readStream;
*writeStreamPtr = writeStream;
}
// A category on NSStream that provides a nice, Objective-C friendly way to create
// bound pairs of streams.
@implementation NSStream (BoundPairAdditions)
+ (void)createBoundInputStream:(NSInputStream **)inputStreamPtr outputStream:(NSOutputStream **)outputStreamPtr bufferSize:(NSUInteger)bufferSize
{
CFReadStreamRef readStream;
CFWriteStreamRef writeStream;
assert( (inputStreamPtr != NULL) || (outputStreamPtr != NULL) );
readStream = NULL;
writeStream = NULL;
#if defined(__MAC_OS_X_VERSION_MIN_REQUIRED) && (__MAC_OS_X_VERSION_MIN_REQUIRED < 1070)
#error If you support Mac OS X prior to 10.7, you must re-enable CFStreamCreateBoundPairCompat.
#endif
#if defined(__IPHONE_OS_VERSION_MIN_REQUIRED) && (__IPHONE_OS_VERSION_MIN_REQUIRED < 50000)
#error If you support iOS prior to 5.0, you must re-enable CFStreamCreateBoundPairCompat.
#endif
if (NO) {
CFStreamCreateBoundPairCompat(
NULL,
((inputStreamPtr != nil) ? &readStream : NULL),
((outputStreamPtr != nil) ? &writeStream : NULL),
(CFIndex) bufferSize
);
} else {
CFStreamCreateBoundPair(
NULL,
((inputStreamPtr != nil) ? &readStream : NULL),
((outputStreamPtr != nil) ? &writeStream : NULL),
(CFIndex) bufferSize
);
}
if (inputStreamPtr != NULL) {
*inputStreamPtr = CFBridgingRelease(readStream);
}
if (outputStreamPtr != NULL) {
*outputStreamPtr = CFBridgingRelease(writeStream);
}
}
@end
Here I build the request body by handling the NSOutputStream delegation.
- (void)stream:(NSStream *)stream handleEvent:(NSStreamEvent)eventCode {
switch(eventCode) {
case NSStreamEventHasSpaceAvailable: {
if(self.contactIndex == 0 && [self.producerStream hasSpaceAvailable]) {
NSMutableData *data = [[NSMutableData alloc] init];
[data appendData:[@"rnrnrn" dataUsingEncoding:NSUTF8StringEncoding]];
[data appendData:[@"{"contacts": [" dataUsingEncoding:NSUTF8StringEncoding]];
[self.producerStream write:[data bytes] maxLength:[data length]];
}
while([self.producerStream hasSpaceAvailable] && self.contactIndex < [self.dataContactIDs count]) {
NSMutableData *contactData = [[[self getNextContact] dataUsingEncoding:NSUTF8StringEncoding] mutableCopy];
if(self.contactIndex < [self.dataContactIDs count]) {
[contactData appendData:[@"," dataUsingEncoding:NSUTF8StringEncoding]];
}
[self.producerStream write:[contactData bytes] maxLength:[contactData length]];
}
if(self.contactIndex == self.dataContactIDs.count) {
NSMutableData *data = [[NSMutableData alloc] init];
[data appendData:[@"]}" dataUsingEncoding:NSUTF8StringEncoding]];
[data appendData:[@"rnrnrn" dataUsingEncoding:NSUTF8StringEncoding]];
[self.producerStream write:[data bytes] maxLength:[data length]];
[stream close];
[stream removeFromRunLoop:[NSRunLoop currentRunLoop] forMode:NSDefaultRunLoopMode];
stream = nil;
}
} break;
case NSStreamEventHasBytesAvailable: {
} break;
case NSStreamEventErrorOccurred: {
} break;
case NSStreamEventEndEncountered: {
[stream close];
[stream removeFromRunLoop:[NSRunLoop currentRunLoop] forMode:NSDefaultRunLoopMode];
stream = nil;
} break;
default: {
} break;
}
}
I'm using AFNetworking to do the networking. I set the request body stream to the NSInputStream.
NSMutableURLRequest *request = [[NSMutableURLRequest alloc] initWithURL:[NSURL URLWithString:url]];
[request setHTTPMethod:@"POST"];
[request setValue:@"application/json; charset=UTF-8" forHTTPHeaderField:@"Content-Type"];
[request setHTTPBodyStream:inputStream];
AFHTTPRequestOperation *op = [[AFHTTPRequestOperation alloc] initWithRequest:request];
op.responseSerializer = [AFHTTPResponseSerializer serializer];
[op setUploadProgressBlock:^(NSUInteger bytesWritten, long long totalBytesWritten, long long totalBytesExpectedToWrite) {
NSLog(@"PROGRESS %d %lld %lld", bytesWritten, totalBytesWritten, totalBytesExpectedToWrite);
}];
[op setCompletionBlockWithSuccess:^(AFHTTPRequestOperation *operation, id responseObject) {
[self processResponse:responseObject success:success error:error log:log];
} failure:^(AFHTTPRequestOperation *operation, NSError *e) {
[self processError:e op:operation error:error log:log];
}];
[[NSOperationQueue mainQueue] addOperation:op];
Then network request comes across like so: (captured using Wireshark)
POST /upload?token=dd224bceb02929b36d35&agent=iPhone%20Simulator&v=1.0 HTTP/1.1
Host: localhost:6547
Transfer-Encoding: Chunked
Accept-Encoding: gzip, deflate
Content-Type: application/json; charset=UTF-8
Accept-Language: en-us
Connection: keep-alive
Accept: */*
User-Agent: MyApp/2.0 CFNetwork/672.0.8 Darwin/13.0.0
9BD
{"contacts": [(valid json array)]}
0
I'm not sure why the 9BD and 0 are included in the request body. I think it's an error with how the buffers are setup and I believe this causes the server to disregard the http body because it's invalid. Does it look like I'm building the request correctly? Is there a better way to do this? I'm using pyramid/python to handle the request. The server receives the request okay, but the request body is empty.
Edit
If I don't send any contacts, the "9BD" goes away. If I change the contact data, the "9BD" changes to different characters. The "0" is always at the bottom.
Edit 2
Jim pointed out that the request is in a valid format. That means the server isn't handling the stream properly. The request is hitting the server okay, and the server is replying okay. However, I'm not seeing any of the request body. The server is running pyramid/python. On the server, request.body is empty.
Your stream handler delegate is not correct:
Here, when you write data into the producerStream:
[self.producerStream write:[data bytes] maxLength:[data length]];
it can happen that not all bytes from the NSData object could have been written into the stream. When this happens, you loose bytes.
In order to fix this, you need to check the return value of write:maxLength: , which equals the number of bytes written (or indicates an error). Then you need to save the state of the NSData object and the range of bytes from that data object which you have written into the stream. In the next event loop you need to check whether there are bytes left from the data and continue to write bytes until all bytes have been written.
Actually, a robust implementation of such kind of task is quite tricky and error prone.
I would like to share some code which copies one stream into another and is already tested:
RXStreamToStreamCopier
This code may give you a jump start. The class RXStreamToStreamCopier copies a source stream into an destination stream. The streams are scheduled on a run loop which you can specify. The class is like an NSOperation which can be started and cancelled.
Internally, the class uses a fixed size transfer buffer and a pull method to read from the source stream and a push method to write into the destination stream. You can override the push method to transform the source bytes.
You create a RXStreamToStreamCopier object with
- (id) initWithSourceStream:(NSInputStream*)sourceStream
destinationStream:(NSOutputStream*)destinationStream;
The source stream would be associated to your data source. The destination stream is usually one half of a bounded stream pair. The bounded stream pair's other end - an input stream - can then be finally set the property HTTPBodyStream of the request.
You might use it as it is, but it depends on a another library, RXPromise.
Hint for a workaround with Chunked Transfer Encoding:
If you know the size of the streamed bytes in advance, you can explicitly set the Content-Length header. This will cause NSURLConnection not to use chunked transfer encoding.
As @Rob stated in a comment, NSURLSession will behave differently: if a stream is set as input, a Content-Length header will be removed which results in a chunked transfer encoding.
That request is fine. Your request is chunked:
Transfer-Encoding: Chunked
The 9BD indicates the length of the next chunk. The zero at the end indicates that there are no more chunks.
See section 3.6.1 of RFC 2616 for details.
Your problem is probably that your server doesn't understand chunked requests.
链接地址: http://www.djcxy.com/p/17386.html上一篇: 生成大量独特的随机float32数字
下一篇: iOS大块上传