efficient way to truncate large array in Matlab

I have a large (multi-GB) array in Matlab, that I want to truncate¹. Naively, I thought that truncating can't need much memory, but then I realised that it probably can:

>> Z = zeros(628000000, 1, 'single');
>> Z(364000000:end) = [];
Out of memory. Type HELP MEMORY for your options.

Unless Matlab does some clever optimisations, before truncating Z , this code actually creates an array (of type double!) 364000000:628000000 . I don't need this array, so I can do instead:

>> Z = Z(1:363999999);

In this case, the second example works, and is fine for my purpose. But why does it work? If Z(364000000:end) = 0 fails due to the memory needed for the intermediate array 364000000:628000000 , then why does not Z = Z(1:363999999) fail due to the memory needed for the intermediate array 1:363999999 , that is larger? Of course, I don't need this intermediate array, and would be happy with either a solution that truncates my array without having any intermediate array, or, failing that, if Matlab optimises a particular method.

  • Is there any way to truncate an array without creating an intermediate indexing array?
  • If not, is either of the aforementioned methods more memory-efficient than the other (it appears ot is)? If so, why? Does Matlab really create intermediate arrays in both examples?

  • ¹Reason: I'm processing data but don't know how much to preallocate. I make an educated guess, often I'm allocating too much. I choose chunk size based on available memory, because splitting in fewer chunks means faster code. So I want to avoid any needless memory usage. See also this post on allocating by chunk.


    I ran both examples on a machine with 24GB of RAM with profile('-memory','on'); . This profiler option will show memory allocated and freed on each line of code. These are supposed to be gross not net amounts. I checked with a simple function that has net 0 free and alloc and it reported the gross amounts. However, it seems likely that builtin commands with no .m code to back them do not give fine-grained memory reporting to the profiler.

    I ran a couple tests for the following code:

    % truncTest.m
    N = 628000000;
    M = 364000000;
    
    clear Z
    Z = zeros(N,1,'single');
    Z(M:end) = [];
    Z(1) % just because
    
    clear Z
    Z = zeros(N,1,'single');
    Z = Z(1:M);
    Z(1)
    

    For what they are worth, the memory profiling results for this N and M are:

    在这里输入图像描述

    Well, both lines look the same in terms of memory allocated and freed. Maybe that's not the whole truth.

    So, out of curiosity I decreased M to 200 (just 200!) without changing N , did profile clear and reran. Profiling claims:

    在这里输入图像描述

    Interestingly, Z=Z(1:M); is practically instantaneous now, and Z(M:end)=[]; is a little faster. Both free about 2.4GB of memory, as expected.

    Finally, if we go the other direction and set M=600000000; :

    在这里输入图像描述

    Now even Z=Z(1:M); is slow, but about twice as fast as Z(M:end)=[]; .

    This suggests:

  • Z=Z(1:M); just grabs the indicated elements, stores them in a new buffer or temporary variable, releases the old buffer and assigns the new/temporary to the array Z . I was able to make my weaker 4GB machine go from 2.45 seconds to thrashing the page file for 5 minutes just by increasing M and leaving N alone. Definitely prefer this option for small M/N , probably in all cases.
  • Z(M:end)=[]; always rewrites the buffer, and execution time increases with M too. Actually always slower, and seems to increase exponentially, unlike Z=Z(1:M); .
  • Memory profiling does not give fine-grained information about these builtin operations and should not be misinterpreted as giving a total of memory freed and allocated over the commands execution, but rather a net change.
  • UPDATE 1 : Just for fun I timed the tests at a range of values of M :

    Clearly more informative than the profiling. Both methods are not no-ops, but Z=Z(1:M); is fastest, but it can use almost double the memory of Z for M/N near 1.

    UPDATE 2 :

    A relatively unknown feature called mtic (and mtoc ) were available in 32-bit Windows prior to R2008b. I still have it installed on one machine, so I decided to see if that provides any more insight, with the understanding that (a) much has changed since then and (b) it's a completely different memory manager used in 32-bit MATLAB. Still, I reduced the test size to N=128000000; M=101000000; N=128000000; M=101000000; and had a look. First, feature mtic for Z=Z(1:M-1);

    >> tic; feature mtic; Z=Z(1:M-1); feature mtoc, toc
    
    ans = 
    
          TotalAllocated: 808011592
              TotalFreed: 916009628
        LargestAllocated: 403999996
               NumAllocs: 86
                NumFrees: 77
                    Peak: 808002024
    
    Elapsed time is 0.951283 seconds.
    

    Clearing up, recreating Z , the other way:

    >> tic; feature mtic; Z(M:end) = []; feature mtoc, toc
    
    ans = 
    
          TotalAllocated: 1428019588
              TotalFreed: 1536018372
        LargestAllocated: 512000000
               NumAllocs: 164
                NumFrees: 157
                    Peak: 1320001404
    
    Elapsed time is 4.533953 seconds.
    

    In every metric ( TotalAllocated , TotalFreed , NumAllocs , etc.), Z(M:end) = []; is less efficient than Z=Z(1:M-1); . I expect it is possible to discern what is going on in memory by examining these numbers for these values of N and M , but we'd be guessing about an old MATLAB

    链接地址: http://www.djcxy.com/p/29160.html

    上一篇: 在MATLAB中比较两幅图像

    下一篇: 在Matlab中截取大数组的有效方法