Problem with trimming lines / link with objects in Delphi

Question

My application creates many objects in memory based on file names (among other string data). I was hoping to optimize memory usage by keeping the path and file name separate, and then dividing the path between objects on the same path. I did not try to use the string pool or anything else, basically my objects are sorted, so if I have 10 objects with the same path, I want objects 2-10 to have their path “pointed” in the path of the object 1 (for example, object [2] = .Path of object [1] .Path);

I have a problem, but I do not believe that my objects actually share the link to the same line after I think I tell them (object [2] .Path = object [1]. Path assignment) .

When I perform an experiment with a string list and set all the values ​​pointing to the first value in the list, I can see the “save memory” in action, but when I use objects, I absolutely do not see any changes, admittedly, I use the task manager (private working set) to view changes in memory usage.

Here is a contrived example, I hope this makes sense.

I have an object:

TfileObject=class(Tobject) FpathPart: string; FfilePart: string; end; 

Now I create 1,000,000 object instances using a new line for each of them:

 var x: integer; MyFilePath: string; fo: TfileObject; begin for x := 1 to 1000000 do begin // create a new string for every iteration of the loop MyFilePath:=ExtractFilePath(Application.ExeName); fo:=TfileObject.Create; fo.FpathPart:=MyFilePath; FobjectList.Add(fo); end; end; 

Run this and the task manager says I'm using 68 MB of memory or something like that. (Note that if I allocated MyFilePath outside the loop, I will save memory due to one instance of the string, but this is a contrived example, and not really how it will happen in the application).

Now I want to "optimize" the use of my memory if all the objects have the same instance of the path string, since this is the same value:

var x: integer; start for x: = 1 for FobjectList.Count-1 do start TfileObject (FobjectList [x]) FpathPart: .. = TfileObject (FobjectList [0]) FpathPart; end; end;

Task Manager shows absurdly no changes.

However, if I do something like this with a TstringList:

 var x: integer; begin for x := 1 to 1000000 do begin FstringList.Add(ExtractFilePath(Application.ExeName)); end; end; 

Task Manager talks about 60 MB memory usage.

Now optimize with:

 var x: integer; begin for x := 1 to FstringList.Count - 1 do FstringList[x]:=FstringList[0]; end; 

Task Manager shows the reduction in memory usage that I expect is now 10 MB.

So, I seem to be able to use strings in a list of strings, but not in objects. I obviously missed something conceptually, in code or both!

I hope this makes sense, I really can see the ability to save memory using this technique, since I have many objects with lots of string data, the data is sorted differently, and I would like to be able to iterate over this data after loading it into memory and free some of this memory back, dividing the lines this way.

Thanks in advance for any help you can offer.

PS: I use Delphi 2007, but I just tested on Delphi 2010, and the results are the same, except that Delphi 2010 uses twice as much memory due to unicode strings ...

Answer 1

When your Delphi program allocates and frees memory, it does this not directly using the Windows API functions, but through the memory manager. What you are observing here is that the memory manager does not free all allocated memory back to the OS when it is no longer needed in your program. It will save part or all of them allocated for later in order to speed up subsequent memory requests in the application. Therefore, if you use system tools, the memory will be displayed as allocated by the program, but it is not active, it is marked as available inside and stored in the lists of used memory blocks that MM will use for any additional distribution memory in your program before it goes over in the OS and will request more memory.

If you really want to check how any changes in your programs affect memory consumption, you should not rely on external tools, but should use the diagnostics provided by the memory manager. Download the full version of FastMM4 and use it in your program, putting it as the first block in the DPR file. You can get detailed information using the GetMemoryManagerState() function, which will tell you how many blocks of small, medium and large memory are used and how much memory is allocated for each block size. However, for a quick check (which will be sufficient), you can simply call the GetMemoryManagerUsageSummary() function. It will tell you the total allocated memory, and if you call it, you will see that reassigning FPathPart really frees up a few MB of memory.

When using TStringList you will observe different behavior, and all rows will be added sequentially. The memory for these lines will be allocated from larger blocks, and these blocks will not contain anything else, so they can be freed again when the items in the list of lines are freed. If OTOH you create your objects, then the lines will be distributed alternately with other data elements, so freeing them will create empty areas of memory in large blocks, but the blocks will not be released, because they still contain actual memory for other purposes. You have mostly increased memory fragmentation, which in itself can be a problem.

Answer 2

As noted in another answer, memory that is not used is not always immediately sent to the system by the Delphi memory manager.

Your code guarantees a large amount of such memory, dynamically increasing the list of objects.

A TObjectList (in conjunction with TList and TStringList ) uses an additional memory allocation block. A new instance of one of these containers starts with the memory allocated for 4 elements ( Capacity ). When the number of added elements exceeds Capacity , additional memory is allocated, initially doubling the capacity, and then, as soon as a certain number of elements is reached, increase the capacity by 25%.

Each time the Account exceeds Capacity , additional memory is allocated, the current memory is copied to the new memory and previously freed memory (this is memory that does not return immediately to the system).

When you know how many items need to be loaded into one of these types of lists, you can avoid this memory allocation (and achieve significant performance improvements) by first selecting the List capacity accordingly.

You don’t have to set the exact required capacity - the best guess is (which is likely to be closer or higher, the actual required rate will still be better than the initial default capacity of 4, if the number of elements is significantly> 64)

Answer 3

Because the task manager does not tell you the whole truth. Compare with this code:

 var x: integer; MyFilePath: string; fo: TfileObject; begin MyFilePath:=ExtractFilePath(Application.ExeName); for x := 1 to 1000000 do begin fo:=TfileObject.Create; fo.FpathPart:=MyFilePath; FobjectList.Add(fo); end; end; 

Answer 4

To share a link, strings must be assigned directly and be of the same type (obviously, you cannot use the link between UnicodeString and AnsiString).

The best way I can think of achieving what you want is to follow:

 var StrReference : TStringlist; //Sorted function GetStrReference(const S : string) : string; var idx : Integer; begin if not StrReference.Find(S,idx) then idx := StrReference.Add(S); Result := StrReference[idx]; end; procedure YourProc; var x: integer; MyFilePath: string; fo: TfileObject; begin for x := 1 to 1000000 do begin // create a new string for every iteration of the loop MyFilePath := GetStrReference(ExtractFilePath(Application.ExeName)); fo := TfileObject.Create; fo.FpathPart := MyFilePath; FobjectList.Add(fo); end; end; 

To verify that it worked correctly, you can call the StringRefCount (unit system) function. I don’t know which version of delphi was introduced, so the current implementation is here.

 function StringRefCount(const S: UnicodeString): Longint; begin Result := Longint(S); if Result <> 0 then Result := PLongint(Result - 8)^; end; 

Let me know if it worked the way you wanted.

EDIT: If you are afraid that the string list is too large, you can safely check it and remove from the list any string with StringRefCount of 1.

The list can also be cleared too ... But this will force the function to reserve a new copy of any new line passed to the function.