The hourglass is meaningless when Windows has marked the program as non-responsive. If a program is stuck in an infinite loop it will show an hourglass too, how is a user going to know the difference? When I see a program labeled as "Not Responding" I give it a few seconds to sort itself out and then I force kill it. This has been a solved problem for decades now, it's terrible user interface design, and it points to piss poor coding internally.
Do better then talk? Here is the code to deduplicate a TStringList while updating a progress bar and preventing your app from being labeled as non-responsive:
DedupStringList.Sorted := True;
DedupStringList.Duplicates := dupIgnore;
for i := 0 to OrigStringList.Count - 1 do
if i mod 1000 = 0 then
Brain dead simple, tested on a Pentium Dual Core T4500 laptop with 2GB of memory, sample data set was 250 unique first and last names randomly distributed over 6,000,000 entries creating an 84 MB text file. In a simple console application, the file was loaded to OrigStringList and deduplicated by transferring the to another TStringList with TStringList.Duplicates set to dupIgnore using TStringList.AddStrings(). This took 43 seconds and resulted in DedupStringList containing the original 250 unique strings. For the next test the same dataset was used in a graphical application using the code above to deduplicate the list. It took 46 seconds to process, updated the progress bar, and the application remained responsive allowing the window to be moved, minimized and maximized, as was not labelled "Not Responding" by the Task Manager. That can be further spread up by increasing the number of iterations between calls to Application.ProcessMessages and how often the progress bar is updated, that can be adjusted to the number of entries be processed, the type of progress bar being updated, etc etc. Using this method you'll want to either disable the form, or better yet, disable the individual controls that you don't want users clicking during the processing. You'll definitely want to disable the De-Deplicate button to prevent inpatient users from clicking it multiple times and stuffing the message queue.