TStringList and TThread that does not free all of its memory
Version used: Delphi 7.
I'm working on a program that does a simple for loop on a Virtual ListView. The data is stored in the following record:
type TList=record
Item:Integer;
SubItem1:String;
SubItem2:String;
end;
Item is the index. SubItem1 the status of the operations (success or not). SubItem2 the path to the file. The for loop loads each file, does a few operations and then, save it. The operations take place in a TStringList. Files are about 2mb each.
Now, if I do the operations on the main form, it works perfectly.
Multi-threaded, there is a huge memory problem. Somehow, the TStringList doesn't seem to be freed completely. After 3-4k files, I get an EOutofMemory exception. Sometimes, the software is stuck to 500-600mb, sometimes not. In any case, the TStringList always return an EOutofMemory exception and no file can be loaded anymore. On computers with more memory, it takes longer to get the exception.
The same thing happens with other components. For instance, if I use THTTPSend from Synapse, well, after a while, the software cannot create any new threads because the memory consumption is too high. It's around 500-600mb while it should be, max, 100mb. On the main form, everything works fine.
I guess the mistake is on my side. Maybe I don't understand threads enough. I tried to free everything on the Destroy event. I tried FreeAndNil procedure. I tried with only one thread at a time. I tried freeing the thread manually (no FreeOnTerminate...)
No luck.
So here is the thread code. It's only the basic idea; not the full code with all the operations. If I remove the LoadFile prodecure, everything works good. A thread is created for each file, according to a thread pool.
unit OperationsFiles;
interface
uses Classes, SysUtils, Windows;
type
TOperationFile = class(TThread)
private
Position : Integer;
TPath, StatusMessage: String;
FileStringList: TStringList;
procedure UpdateStatus;
procedure LoadFile;
protected
procedure Execute; override;
public
constructor Create(Path: String; LNumber: Integer);
end;
implementation
uses Form1;
procedure TOperationFile.LoadFile;
begin
try
FileStringList.LoadFromFile(TPath);
// Operations...
StatusMessage := 'Success';
except
on E : Exception do StatusMessage := E.ClassName;
end;
end;
constructor TOperationFile.Create(Path : String; LNumber: Integer);
begin
inherited Create(False);
TPath := Path;
Position := LNumber;
FreeOnTerminate := True;
end;
procedure TOperationFile.UpdateStatus;
begin
FileList[Position].SubItem1 := StatusMessage;
Form1.ListView4.UpdateItems(Position,Position);
end;
procedure TOperationFile.Execute;
begin
FileStringList:= TStringList.Create;
LoadFile;
Synchronize(UpdateStatus);
FileStringList.Free;
end;
end.
What could be the problem?
I thought at one point that, maybe, too many threads are created. If a user loads 1 million files, well, ultimately, 1 million threads is going to be created -- although, only 50 threads are created and running at the same time .
Thanks for your input.
There are (probably) no leaks in the code you show in the question.
I say probably because an exception raised during the Execute could result in a leak. The lifetime of the string list should be protected by a finally block.
FileStringList:= TStringList.Create;
try
LoadFile;
Synchronize(UpdateStatus);
finally
FileStringList.Free;
end;
That said, I expect the exception swallow in LoadFile means that you don't leak the string list.
You say that perhaps thousands of threads are created. Each thread reserves memory for its stack, and the default stack size is 1MB. Once you have thousands of 1MB stacks reserved, you can easily exhaust or fragment address space.
I've seen problems due to cavalier creation of threads in the past. For example I had a program that failed when it created and destroyed threads, with never more than 256 threads in existence. This was on a 16 core machine with 4GB address space. You probably have 2GB address space available.
Although you state that no more than 50 threads are in existence at any one moment, I'm not sure how you can be sure of that. Not least, because you have set FreeOnTerminate to True and thereby surrendered control over the lifetime of your threads.
My guess is that your problems are related to the number of threads you create. One thread per processor will suffice. Re-use your threads. It's expensive to create and destroy a thread for a small task.
If this is not enough to solve your problems then you will need to show the code that manages thread lifetime.
Finally, I wonder how much benefit you will extract from threading this app. If it is IO bound then the threaded version may well be slower!
Based on the information given, it's not possible to reproduce your error. Some hints are made by Remy and David which might help you.
Looking at the structure of your program, the flow can be divided into two classical solutions.
The first part where you are delegating tasks to different threads, is a Single-Producer-Multiple-Consumer problem. Here it can be solved by creating a small number of threads, passing them a thread-safe object queue. The main thread then pushes the task objects into the queue. The consumer threads takes care of the individual file checking tasks.
The second part where the result is to be transfered to the main thread is a Multiple-Producer-Single-Consumer problem. If you pass a second thread-safe object queue to the threads at initialization, they can easily put the results into the queue. Drain the result queue from the main thread within a timer event.
上一篇: 测量线程的内存消耗