Friday, December 01, 2006

Installing BDS Update 2 on Vista

Few days ago I had an opportunity to test BDS 2006 on a Vista RTM (I was writing an article on Vista for the Monitor magazine and I did some personal tests when all other work was finished). Installing BDS was not a problem, but Update 2 was stubbornly refusing to find the BDS installation. All I was getting was this error:

Setup cannot continue. This is Update 2 setup for Borland Developer
Studio 2006 Architect Edition. Please cancel this and install 
Edition of Update 2

(Sorry, no picture - I forgot to take a screen shot at that time.) There was a double space between 'install' and 'Edition' so obviously something went very wrong with the installer and it didn't know anymore what it's supposed to do.

Soon  I found out that I'm not the only one with the problem. I also found a workaround on Borland support site. As I was already very skeptical about the whole UAC thing, I switched it off without a thought.

That's how you do it: Click the Start button, type user in the search box and run User Accounts applet. Click the Turn User Account Control on or off link. Uncheck the User User Account Control (UAC) to help protect your computer. Click OK. Be aware that restart will be required to disable this setting.

After installing BDS 2006 Update 2 you can turn UAC on with the similar procedure - but that's up to you. I left it disabled. Hate the damn thing.

To return to my installation problems - this workaround didn't work for me. I was not getting that error anymore but it got even worse - Update 2 just closed and Vista reported something like "Microsoft Installer found a problem and cannot continue". I tried few times )(stubborn programmer) but the result was always the same. I even rebooted and retried and the I got lucky - Update 2 still closed but Vista displayed an error hint in the taskbar saying that my app got closed due to a DEP protection. (Yes, I was testing Vista on AMD 64 processor, which has hardware DEP support built in. No, I was running Vista 32-bit.)

That's something, but how do we disable DEP? Vista, like Windows XP, only has two options - enable DEP for all system programs (default) or enable DEP for all programs except the one in the exclusion list. (DEP settings are hidden in System Properties, Performance Options, in case you can't find them.)

I searched the MSDN and found out that I have to disable DEP during boot. Usually (since NT times) one would have to edit c:\boot.ini file. Surprise, surprise, this file is gone in Vista! To edit boot configurations, you have to use a command-line tool bcdedit. Documentation on it is not very well organized. The best starting point is probably Boot Options for Driver Testing and Debugging.

It turns out it's easiest to create new configuration with the /copy switch:

C:\Users\gabr>bcdedit /copy {current} /d "DEP disabled"
The entry was successfully copied to {69ef4ad0-8123-11db-aa33-0015c53ad3ed}.

I based the new configuration on the current configuration (which is the simplest approach to creating a new configuration). Bcdedit returned GUID of the new configuration. We'll need this GUID to modify the new configuration:

C:\Users\gabr>bcdedit /set {69ef4ad0-8123-11db-aa33-0015c53ad3ed} nx AlwaysOff
The operation completed successfully.

The 'nx AlwaysOff' setting disables the execution prevention (see BCDEdit /set for more parameters).

We can verify that configuration really got created:

Windows Boot Manager
--------------------
identifier              {bootmgr}
device                  partition=C:
description             Windows Boot Manager
locale                  en-US
inherit                 {globalsettings}
default                 {current}
resumeobject            {53a24ff1-7f3d-11db-9cd4-a2298b078311}
displayorder            {current}
                        {69ef4ad0-8123-11db-aa33-0015c53ad3ed}
toolsdisplayorder       {memdiag}
timeout                 30

Windows Boot Loader
-------------------
identifier              {current}
device                  partition=C:
path                    \Windows\system32\winload.exe
description             Microsoft Windows Vista
locale                  en-US
inherit                 {bootloadersettings}
osdevice                partition=C:
systemroot              \Windows
resumeobject            {53a24ff1-7f3d-11db-9cd4-a2298b078311}
nx                      OptIn

Windows Boot Loader
-------------------
identifier              {69ef4ad0-8123-11db-aa33-0015c53ad3ed}
device                  partition=C:
path                    \Windows\system32\winload.exe
description             DEP disabled
locale                  en-US
inherit                 {bootloadersettings}
osdevice                partition=C:
systemroot              \Windows
resumeobject            {53a24ff1-7f3d-11db-9cd4-a2298b078311}
nx                      AlwaysOff

 Alternatively, you can check the System Properties, Startup and Recovery.

 

(I apologize for the quality of the picture - that's purely Windows Liver Writer's doing.)

If you made those changes and reboot the system, it will display the usual NT style 'select your configuration' prompt. I did that, booted without a DEP and finally I could install Update 2. I rebooted back with DEP enabled and BDS 2006 works just fine.

Sadly, I had to return the test computer next day so now I'm running Vista only on my Dell notebook. At least BDS works fine here, too.

Wednesday, November 15, 2006

Bulk update

I have updated few of my BSD-licensed units. Enjoy.

GpLists 1.22a

GpHugeFile 4.0b

GpStructuredStorage 1.11

GpSync 1.19a

  • Raise exception if caller tries to post a message that is larger than the message queue.

GpTextFile 4.0a

  • Bug fixed [found by Brian D. Ogilvie]: When cfUseLF was set, /CR/ was used as line delimiter in Writeln (/LF/ should be used).

GpTextStream 1.04a

  • Bug fixed: When cfUseLF was set, /CR/ was used as line delimiter in Writeln (/LF/ should be used).

GpTimezone 1.21b

  • Modified UTCTo[TZ]LocalTime/[TZ]LocalTimeToUTC functions to automatically execute FixDT on the result.

Friday, October 27, 2006

Faster than the speed of binary search

[Updated 2006-10-28, new text and code at the end.]

Everybody (at least everybody competent enough to be worth mentioning) know how to search in sorted array with the binary search algorithm. But everybody (or almost everybody) thinks that this is the fastest approach to searching. Is everybody (or almost everybody) correct regarding that topic? It depends.

On general, the answer is true - binary search is fastest. But that only holds if we know nothing about the data stored in the array - in other words, if we pretend that it is random (but sorted, of course).

Want an example? Take an array [1..99] of byte. Fill it with numbers in the range 1..100 but leave (random) one out. Use each number only once. Sort it. You'll get an array with first N numbers in the same position as the array index (A[i] = i for all i in 1..N) and the rest of it will be off by one (A[i] = i+1 for all i in N+1..99). Do you think that binary search is fastest possible search in that case? Of course no - to find a random number (say m) you only have to check A[m] and A[m-1]. Two lookups, that's all.

Are there more general guidelines to speeding up a binary search. Yes, it's quite simple - we only have to be able to guess the position of the value we're searching for better than with the "it's somewhere in the middle" approach. In a way, that's exactly what we did in the example above.

This approach is very simple to use if data in the array is distributed uniformly. In that case, we can take the value at low and high end of the interval (not the indices as in the binary search, but values at those indices) and use a linear interpolation to guess a possible position of the value.

Something similar happend to me last week. I was solving some DVB-related problem and built an array of video fragments (video PES, if you know anything about the DVB internals) in a video file (actually I only stored file offset for each fragment). Later, I needed to find (many times) a fragment closest to some file offset (every time a different offset). Of course, I could use a simple binary search, but it was such a nice opportunity to try out a better approach and run some tests ...

For test data I used a small, 40 MB video file. There were 51737 entries in the array and the code executed 3128 searches. With standard bisection, the code did 15.45 rounds through the bisection loop per search (on average). With interpolated search, this number was decreased to 4.13 rounds per search. That's better by a factor 3.7! And in the real life, the file will be some 50x larger - around 2 GB - and difference will be even bigger.

The number of rounds/number of searches distribution for both approaches:

We can see that standard binary search needed 16 rounds for majority of the searches and almost no searches were completed in less than 15 rounds.

On the other hand, interpolated binary search shows almost gaussian distribution from 2 to 6 rounds.

But as the code is more complicated (see below), the total speedup was not that impressive, only somewhere around 1.6 times. Still, it was a personal satisfaction to see the theoretically better code run faster in reality, too :)

function TGpTSPIDIndex.Find(transportStreamOffset: int64; var index: integer): boolean;
var
L, H, I, C: integer;
begin
{More work is needed to check for boundary conditions}
if Count = 0 then begin
index := 0;
Result := false;
end
else if transportStreamOffset <= Chains[0].TransportStreamOffset then begin
index := 0;
Result := (transportStreamOffset = Chains[0].TransportStreamOffset);
end
else if transportStreamOffset = Chains[Count-1].TransportStreamOffset then begin
index := Count-1;
Result := true;
end
else if transportStreamOffset >= Chains[Count-1].TransportStreamOffset then begin
index := Count;
Result := false;
end
else begin
Result := false;
L := 0;
H := Count - 1;
while Chains[L].TransportStreamOffset <= Chains[H].TransportStreamOffset do begin
//instead of
//I := (L + H) shr 1;
//we're using
if L = H then
I := L
else begin
I := Round((transportStreamOffset - Chains[L].TransportStreamOffset) /
(Chains[H].TransportStreamOffset - Chains[L].TransportStreamOffset) * (H-L)) + L;
if I < L then
I := L
else if I > H then
I := H;
end;
//offsets are int64 numbers
C := Int64Compare(Chains[I].TransportStreamOffset, transportStreamOffset);
if C < 0 then
L := I + 1
else begin
H := I - 1;
if C = 0 then begin
Result := true;
L := I;
end;
end;
end;
index := L;
end;
end; { TGpTSPIDIndex.Find }


[Updated 2006-10-28]


While thinking on the comment posted by the anonymous I looked at the code from a new view and suddenly I realized that I'm an idiot. The 'while' condition in the bisection loop in my version should not differ from the condition in standard binary search! When I put this right, I could suddenly remove all extra tests at the beginning as they were no longer needed. New code (tested to work the same as the old code and as the standard binary search - at least on my data) is posted below.


A word of warning: all this works only if there are no repetitive elements in the sorted array! If this condition is not satisfied, you'll soon get a 'division by 0' exception in the calculation of the weighted midpoint.

function TGpTSPIDIndex.Find(transportStreamOffset: int64; var index: integer): boolean;
var
L, H, I, C: integer;
begin
Result := false;
L := 0;
H := Count - 1;
while L <= H do begin
//standard binary search
//I := (L + H) shr 1;
//weighted version
if L = H then
I := L
else begin
I := Round((transportStreamOffset - Chains[L].TransportStreamOffset) /
(Chains[H].TransportStreamOffset - Chains[L].TransportStreamOffset) * (H-L)) + L;
if I < L then
I := L
else if I > H then
I := H;
end;
C := Int64Compare(Chains[I].TransportStreamOffset, transportStreamOffset);
if C < 0 then
L := I + 1
else begin
H := I - 1;
if C = 0 then begin
Result := true;
L := I;
end;
end;
end;
index := L;
end; { TGpTSPIDIndex.Find }

Tuesday, October 17, 2006

Delphi to the rescue (and one little Win32 API call)

A new day, a new hardware to support ...

I plugged the board (DVB receiver/transmitter in case you are interested) into the computer, powered it up and got an "interesting" hardware wizard window

This one is localized into Slovenian, but you probably know it by heart - it is a standard Windows wizard for adding new hardware devices. Except that this time it came with a twist - for some reason the window was only 200 pixels high. Did I have to mention that this wizard is not resizable? Probably not. (I hate non-resizable applets!)

So what did I do? Reboot, of course. You can probably guess that it didn't help (or I wouldn't be writing this). HW wizard appeared after the next boot and it was completely the same as in the picture above.

An ordinary user would probably start kicking the computer, blaming it all on the Microsoft and calling his/her best friend that supposedly knows "everything" about computers. Me? I kicked the computer, blamed it all on the Microsoft and fired up Delphi 2006.

First, I started one of my old applications that lists all windows on the system (using EnumWindows API) and located window handle of the HW wizard. I could also do this with Microsofts Spy++ but at that moment I was positively mad at guys in Redmond and didn't want to use their tools.

Next I wrote a small app that called SetWindowPos API with this handle and bigger window boundaries. And, behold, HW wizard was magically enlarged! Sometimes it's really good to know your Win32 API ...

OK, so I lied a little. I didn't wrote a small app that called ... etc. as I already had something lying handly.  An old app of mine that searches for a window with specified caption and resizes it. As it is a burden to type "ÄŒarovnik za najdeno novo strojno opremo" into a command line (and I would be running into all sorts of problems with codepages in DOS window that way) I updated it to also accept window handle instead of a windows caption. (Yes, I could type everything into Start, Run without any codepage problems, but this solution was way more satisfying :) )

If you run into something like that in the future, feel free to use my little app.

program SetWindowPlacement;

{$APPTYPE CONSOLE}

uses
Windows,
SysUtils;

procedure SetWindowPos(windowTitle: string; windowLeft, windowTop,
windowWidth, windowHeight: integer);
var
flags : cardinal;
hWindow: THandle;
begin
if Copy(windowTitle, 1, 1) = '#' then begin
Delete(windowTitle, 1, 1);
hWindow := StrToInt(windowTitle);
end
else
hWindow := FindWindow(nil, PChar(windowTitle));
if hWindow = 0 then
Writeln('Window >', windowTitle, '< not found!')
else begin
flags := SWP_NOACTIVATE OR SWP_NOZORDER;
if (windowWidth = 0) and (windowHeight = 0) then
flags := flags OR SWP_NOSIZE;
Windows.SetWindowPos(hWindow, 0, windowLeft, windowTop, windowWidth, windowHeight,
flags);
end;
end; { SetWindowPos }

procedure Usage;
begin
Writeln('SetWindowPlacement');
Writeln('(c) 2006 Primoz Gabrijelcic');
Writeln;
Writeln('Usage: SetWindowPlacement (<window title>|#<window handle>) <left> <top> [<width> <height>]');
end; { Usage }

begin
if (ParamCount < 3) or (ParamCount > 5) then
Usage
else try
if ParamCount = 3 then
SetWindowPos(ParamStr(1), StrToInt(ParamStr(2)), StrToInt(ParamStr(3)),
0, 0)
else
SetWindowPos(ParamStr(1), StrToInt(ParamStr(2)), StrToInt(ParamStr(3)),
StrToInt(ParamStr(4)), StrToInt(ParamStr(5)));
except Usage; end;
end.

Monday, October 16, 2006

My very personal Algorithms and Data Structures

Thanks to the wonders of the printing on demand, I have received my very personal (printed just for me! merry!) copy of Julian M Bucknall's Algorithms and Data Structures. (previous post on that topic)

I won't comment on the content much - this is a great book and every serious Delphi programmer should have it on the shelf. Plus the Julian's writing is fluid and well organized and always to the point.

Printing could be improved a little, though. Font on inner pages is slightly too black for my taste and shadings under the code fragments are waaay too dense. Covers are weird - it seems that the font on the front and back cover was aliased to a different color than used on the cover for the background. It looks unsharp from the distance and halo'd if you look closer.

Still, the book is totally readable and printing is much better than I expected. Binding is fine, too. I have no worries that the book will fall apart in my hands after only a short use.

I hope Julian won't mind if I attach here few photos for the yet undecided. (Yes, the bottom right corner is slightly damaged, but that's courtesy of mail service. The book could be wrapped and packed better, though, so that such things wouldn't happen.)

front cover pages 80 and 81 pages 294 and 295 back cover

Wednesday, October 04, 2006

Algorithms and Data Structures

Julian Bucknall has made his excellent book The Tomes of Delphi: Algorithms and Data Structures available on Lulu.

If you don't have it yet, don't wait. Get your copy now! Every Delphi developer should have it!

[I don't know if it shows ;) but I'm a great fan of Julian's writing.]

Saturday, September 30, 2006

There's no firewall like PF

[No Delphi content here, sorry.]

If you are, by any chance, using Open/FreeBSD for firewalling, be sure to read three excellent articles on PF writen by the Daniel Hartmeier. Yes, the maker of PF himself.

Firewall Ruleset Optimization

Testing Your Firewall

Firewall Management

Tuesday, September 26, 2006

GpStructuredStorage internals

[This is a second article in the 'embedded file system' series. If you missed the first part, you can read it here.]

GpStructuredStorage compound file is organized in 1 KB blocks. First block contains header, then the content alternates between a file allocation table (FAT) fragment block and 256 blocks managed by the preceeding FAT fragment. Each block can be represented by a number - header block is block #0, first fat fragment is block #1 and so on.

[header:HEADER:1024]
[fat entry:FATENTRY:1024]
256 x [block:FOLDER/FILE:1024]
[fat entry:FATENTRY:1024]
256 x [block:FOLDER/FILE:1024]
...
[fat entry:FATENTRY:1024]
<=256 x [block:FOLDER/FILE:1024]

Header starts with a signature, which must always be 'GpStructuredStorage file'#13#10#26#0.

HEADER
[signature:32] // PChar
[unused:964]
[storage attribute file:4]
[storage attribute file size:4]
[first FAT block:4]
[first unused block:4]
[root folder:4]
[root folder size:4]
[version:4] // storage system version
Header ends in:
  • block number and size of the internal file containing global storage attributes
  • block number of the first FAT block (always 1)
  • block number of the first unused file block
  • block number and size of the root folder
  • storage system version (at the moment $01000200, or 1.0.2.0)

Each FAT fragment contains 256 32-bit numbers, one for each of the 256 following file blocks. This number is either 0 if block is last in the FAT chain, or it is a number of the next file block in the chain. Each block is linked into exactly one chain. It can either belong to a file/folder or to a empty block list. Address of the first block in the empty list is stored in the header ([first unused block] entry).

FATENTRY
256*[next block:4] // next-pointers for this block; 0 = unused
FAT structure defines the limitations of my file format - last block must have number that is less than 4294967294. As block are 1 KB in size, total file size cannot exceed 4294967294 * 1 KB, which is slightly less than 4 TB. Enough for all practical purposes, I think.

Folders are simple - each folder is just a file. It contains file information records and is terminated with two 0 bytes.

FOLDER //potentially split over several blocks
[FILE_INFO]
[FILE_INFO]
...
[FILE_INFO]
[0:2]
File information record is a variable length record containing file name (up to 64 KB), attributes, length, and address of the first file block (additional blocks are reached by following the FAT chain).
FILE_INFO
[file name length:2]
[file name:1..65535]
[file attributes:ATTRIBUTES:4]
[file length:4] // 4 GB per file
[first file block:4]
At the moment, only two attributes are defined. One specifies that file is actually a subfolder, and another designates a special file containing file attributes (for discussion of attributes see the previous article).
ATTRIBUTES
$0001 = attrIsFolder
$0002 = attrIsAttributeFile
That's just about everything that is to tell about the compound file format. Armed with this knowledge, one can easily write a compound file browser/repair utility. 

Friday, September 22, 2006

Writing an embedded file system

Once upon a time, Julian M. Bucknall wrote an interesting article for The Delphi Magazine. Well, he wrote more than one and they are all interesting and (that's the best part) he's still writing them, but I particularly liked that one.

The article name is Manage The Damage (it had to be Julian's Jimmy Somerville phase) and it was published in Issue 69 (May 2001). Yes, quite an old stuff - at least in computing terms. Inside, Julian described an implementation of an embedded file system i. e. a file system that rests inside another storage (a file in an ordinary file system, for example). That's such a useful concept that even Microsoft recognizes it (think OLE structured storage).

In short, embedded file system (or compound file, or structured storage) allows you to store many unconnected or loosely connected pieces of information inside one physical file in a structured way. For example, it can be used to store different parts of configuration settings for a large application, or it can be used for data acquisition to store many data packets inside one file, or it can be used as a simple database to store text fragments or code snippets.

Although I liked the idea behind Manage The Damage, I couldn't live with the implementation. It was too limited for my taste (embedded file system was limited to 32 MB) and implementation was a mess of pointers, which made the code almost nonportable to .NET.

And then, as all fairy tales start, I decided to write my own implementation.

Why not use OLE structured storage, you'd ask? Well, I like to know how my data is stored (Have you ever tried to look inside an OLE structure storage file with a hex editor? I did. Not a pretty sight.), I want the implementation to be fast and simple to use from Win32 and potentially .NET. Besides that, it sounded like an interesting challenge.

So how did I fare? Good, if I'm the one to answer. There were no pointers killed while writing the code, total size of the file system is limited only to 4 TB (4 GB for files stored inside the compound file) and file internals are easy to understand (well, at least to me ;) ).

The code was used in some commercial projects. Also, GExperts use it for snippet storage (CodeLibrarian.fs file). It seems to be stable and mostly bug-free and as such I'm releasing it to the public, with the usual string attached.

For the brave, here's the code and test suite: GpStructuredStorage.

If you're still interested, continue reading. I'll show few examples on how the GpStructuredStorage can be used.

A small how-to

Next fragment creates compound file and then reopens it for reading. Note that the compound file is implemented as an interface and doesn't need explicit destructor calls as such.

Instead of a file name, one can also send a TStream or descendant to the .Initialize method.

storage: IGpStructuredStorage;
storage := CreateStructuredStorage;
storage.Initialize(CStorageFile, fmCreate);
// write and read here
storage := CreateStructuredStorage;
storage.Initialize(CStorageFile, fmOpenRead);
// from now on, only reading is allowed

Now that we have the storage interface, we can create a file and then read it.

strFile: TStream
strFile := storage.OpenFile('/folder/file.dat', fmCreate);
try
// write to strFile
finally FreeAndNil(strFile); end;
strFile := storage.OpenFile('/folder/file.dat', fmOpenRead);
try
// read from strFile
finally FreeAndNil(strFile); end;

There is no need to create a /folder manually - every file access automagically creates all required folders.


Still, you are free to do it the old way.

storage.CreateFolder('/folder/subfolder');
if not storage.FolderExists('/folder/subfolder') then
//panic

Of course, there is also a FileExists function.


File enumeration is simplified to the max.

files: TStringList;
files := TStringList.Create;
try
storage.FileNames('/folder', files);
finally FreeAndNil(files); end;

(To enumerate folders, one would use FolderNames instead of FileNames.)


Additional information on file or folder can be access via FileInfo property:

FileInfo['/full/path/to/file.dat']


Currently, FileInfo only exports file's size (FileInfo[].Size) and file attributes (FileInfo[].Attribute).


Attributes offer you a way to store additional string info for each file and folder. Unlimited number of attributes can be stored and the only limitation is that both attribute name and value must be stringified.


storage.FileInfo['/folder/file.dat'].Attribute['author'] := 'Gp';

At the end, I must mention that it is also possible to Move and Delete files/folders and Compact (defragment) the file system.


If I have convinced you, go and use the stuff. If not, wait for the next episode.


Next in the embedded file system series




  • Internals of a GpStructuredStorage file
  • Total Commander plugin to look into GpStructuredStorage files.


Coming soon to a feed near you.

Specialization

A human being
should be able to change a diaper,
plan an invasion,
butcher a hog,
conn a ship,
design a building,
write a sonnet,
balance accounts,
build a wall,
set a bone,
comfort the dying,
take orders,
give orders,
cooperate,
act alone,
solve equations,
analyze a new problem,
pitch manure,
program a computer,
cook a tasty meal,
fight efficiently,
die gallantly.

Specialization is for insects.

- Robert A. Heinlein

 

UPDATE: Fixed the author (sometimes I'm soooo stupid).

Thursday, September 21, 2006

Sir! Do you need a list? Cheap, just for you!

Actually, it is free and comes with only one string attached.

I have just uploaded new version of my lists unit. At this moment it contains ten useful classes:

  • TGpIntegerList is a TList-compatible class used to store integers.
  • TGpInt64List is a TList-compatible class used to store int64 numbers.
  • TGpIntegerObjectList is a TList-compatible class used to store integers and associated objects.
  • TGpIntegerObjectList is a TList-compatible class used to store int64 numbers and associated objects.
  • TGpCountedStringList is a TStringList descendant that has each string item associated with an integer counter (internally stored in the Objects property).
  • TGpTMethodList is a list of TMethod records.
  • TGpObjectRingBuffer is a fixed-sized ring buffer of TObject references.
  • TGpObjectMap is one-dimensional array, indexed by objects and containing objects.
  • TGpObjectObjectMap is two-dimensional array, indexed by objects and containing objects.
  • TGpDoublyLinkedList is doubly-linked list of TGpDoublyLinkedListObject descendants.

Abuse it at will. Additions welcome.

Wednesday, September 20, 2006

Failed to find standard type 'TObject'? C'mon, DevCo!

I have just installed BDS Hotifx rolloup - the one that everybody else on the planet is blogging about - and now my BDS claims that I don't know how to write a class!

C'mon, DevCo, you can do better than that!

 LATER: OK, third restart of BDS helped. No idea what's going on but I certainly hate it when syntax checker (or whatever they call it) goes amok.

Monday, September 04, 2006

If you'll be visiting Amsterdam this weekend ...

... you can meet me on IBC. Most of the time I'll be on stand 2.111. Look for the guy named Primoz - that's me.

Saturday, September 02, 2006

Can't register already registered component

 

Few days ago an avid Delphi programmer posted this image on the Slovenian Delphi forum:

I've been seeing similar errors from Delphi 5 times. This error usually appears when package in question uses some units that are also used by another (already loaded) package and this second package is not listed in the requires section of the first package.

Following steps helped me to solve such mysteries in the past:

  • Uninstall all custom packages.
  • Restart Delphi.
  • Open each custom package, rebuild and install it. During that process, Delphi usually notices the problem mentioned above, warns about it and updates the requires list.

The other approach that may help is a batch file that rebuilds and reinstalls all custom packages. But that is just a workaround if you can't find the real cause of the problem.

BTW, the colleague has found the problematic package but not by following my procedure (he did try it but without success).

BTW2, he is also an author of interesting open-sourced mail read si.Mail.

Tuesday, August 29, 2006

Do you need a lightweight XML parser/writer?

Then I would recommend OmniXML. It is a MSXML compatible XML parser/writer with full support for Document Object Model 1.1 specification. It is released under the Mozilla Public License 1.1 and comes with a full source code.

Some useful units are also included in the distribution - two different approaches to object persistency (OmniXMLPersistent and OmniXMLProperties), simple database persistency (OmniXMLDatabase), TIniFile-compatible interface (OmniXMLConf). A subset of XPath is also supported.

There are shortcomings, too. The most important is that you cannot verify XML document according to a XML schema. There is also no support for XML canonicalization.

Still, OmniXML is very useful when writing XML documents that stay inside a program (XML-formatted configuration files, data storage etc) or inside a component or pair of components (message exchange). For example, it is used in GExperts for data storage and I'm using it for storage and messaging.

Thursday, August 17, 2006

GpHugeFile 4.0

Today, I have released my GpHugeFile unit version 4.0. It is released under the BSD License.

For the highly impatient:

A little background is maybe in order. I have been publishing Borland Pascal (and later Borland Delphi) units since BBS times. All of them are released either as a free source or under the BSD license. After all, I wanted other programmers to use them. Over the years, some of them grew and some not. Suggestions from other users were incorporated. Some contributors even added useful code. A complete list is available on my Delphi page.

But let's return to a more interesting topic...

TGpHugeFile started as a wrapper around system calls allowing access to really large (over 2 GB) files.

Buffering (prefetching) was added next. TGpHugeFile can optionally buffer all read/written data and significantly speed up sequential (or mostly sequential) access to files. Admittedly, it doesn't work very well when files are access randomly.

In the latest (4.0) reincarnation, it can open files with Unicode names (CreateFileW is called instead of CreateFileA in that case).

There are some other goodies included:
  • It can handle direct access to files (see description for FILE_FLAG_NO_BUFFERING in the MSDN CreateFile documentation. TGpHugeFile will automatically align all buffers and read/write requests.

  • 'Close on EOF' mode is useful for very slow links (i.e. modem access). In this mode, TGpHugeFile will close file handle as soon as last block is read from the file. This will free the file for other programs while your program can still read data from the TGpHugeFile's buffer.

  • File handle is not released during Reset/Rewrite if called when file is already open. Delphi does that wrongly (closes the handle), which is not a very good idea if file is shared between applications as another application may try to open it just at the wrong moment.


There is also a TGpHugeFileStream class - a TStream descendant that wraps TGpHugeFile.

That's about all that I have to say on this topic. If you use TGpHugeFile[Stream] and you like it, leave me a mail (or a comment). I'd like to hear more about your experience with it.

Wednesday, August 16, 2006

Back in business

Just came back from a short vacation (14 days, not nearly enough) in nice city of Lukoran (island of Ugljan, Croatia).

Isn't Google Earth just great!

Tuesday, July 18, 2006

Trouble with services

Multithreaded programming is a pain in the <insert body part>. Whatever you do, sooner or later your code will stop working correctly.

Service writing is not simple, either. There are many limitations that should be taken into account (lack of graphical user interface, for example).

For quite some time now, I was sure I mastered them both, but not so long ago a service/thread combination proved me wrong. There's a catch in service programming that I was not aware of until I had to spend several days tracing down one rarely occurring problem. It was so rare that it happened only at one customer and even there the application crashed only once in several hours.

So what's the catch? It is quite simple. Every Windows service is multithreaded. Even if you don't create threads in the service application, it will be using two threads.

It gets even worse. On the service form, OnCreate and OnDestroy are called from the main application thread while OnStart and OnStop are called from the main service thread.

There is a logic behind this, and it is quite obvious once you stop to think about it. A service application can contain several servicesand as they are independent entities, each must run in a separate thread. On the other hand, OnCreate/OnDestroy are called from the main application which creates those services, and are therefore called from the main application thread. Simple.

But it can bite you.

If you drop some components on the service form, they will be created in destroyed from the main application thread. Of course, they will be used from the appropriate service thread (or from the service thread, if there's only one service in the application, which is usually the case).

Usually, you won't notice this. But things start getting weird if component in question creates hidden message-processing window. Main application thread will process messages for this hidden window, but you'll usually want to process them in the service thread. Even worse, if you implement some kind of message pump in the service thread, two threads will be processing messages for the same component. It won't happen frequently, but it surely will happen and you can imagine the result. Deadlocks. Data corruption. Mayhem.

Yes, that's what happened to me.

BTW, that's also why TTimer doesn't work correctly if you drop it on the service form.

If you don't believe me, try it for yourself. Create a new service application and add some logging to the OnCreate/OnStart/OnStop/OnDestroy:

procedure TService1.ServiceDestroy(Sender: TObject);
begin
Log(Format('Destroy called from process %d thread %d',
[GetCurrentProcessID, GetCurrentThreadID]));
end;

procedure TService1.ServiceCreate(Sender: TObject);
begin
Log(Format('Create called from process %d thread %d',
[GetCurrentProcessID, GetCurrentThreadID]));
end;

procedure TService1.ServiceStart(Sender: TService; var Started: boolean);
begin
Log(Format('Start called from process %d thread %d',
[GetCurrentProcessID, GetCurrentThreadID]));
Started := true;
end;

procedure TService1.ServiceStop(Sender: TService; var Stopped: boolean);
begin
Log(Format('Stop called from process %d thread %d',
[GetCurrentProcessID, GetCurrentThreadID]));
Stopped := true;
end;

I used following trivial logger in the demo:

procedure Log(const msg: string);
var
fLog: textfile;
begin
AssignFile(fLog, 'c:\ServiceDemo.log');
if FileExists('c:\ServiceDemo.log') then
Append(fLog)
else
Rewrite(fLog);
Writeln(fLog, msg);
CloseFile(fLog);
end;

Compile and run:
  • ServiceDemo /install

  • net start ServiceDemo

  • net stop ServiceDemo

  • ServiceDemo /uninstall


Service will get installed, started, stopped and uninstalled. Now check the log file:
Create called from process 6020 thread 4188
Destroy called from process 6020 thread 4188
Create called from process 4840 thread 3184
Start called from process 4840 thread 4588
Stop called from process 4840 thread 4588
Destroy called from process 4840 thread 3184
Create called from process 2968 thread 4640
Destroy called from process 2968 thread 4640

What happened here? First OnCreate/OnDestroy got called (when ServiceDemo /install was run). Next, OnCreate was called from thread 3184, OnStart/OnStop were called from thread 4588 (I told you!) and OnDestroy was called from thread 3184. Finally OnCreate/OnDestroy were called then ServiceDemo was run with /uninstall switch.

So there's another problem with dropping components on the service form - they will also be created/destroy whenever service application is called with /install or /uninstall switch!

OK, you're asking, but what can we do? I suggest creating components in the OnStart event - either manually (in source) or by dropping them into a separate data module, which is not created automatically but in the OnStart event:

procedure TService1.ServiceStart(Sender: TService; var Started: boolean);
begin
Log(Format('Start called from process %d thread %d',
[GetCurrentProcessID, GetCurrentThreadID]));
FDataModule := TDataModule2.Create(nil);
Started := true;
end;

procedure TService1.ServiceStop(Sender: TService; var Stopped: boolean);
begin
Log(Format('Stop called from process %d thread %d',
[GetCurrentProcessID, GetCurrentThreadID]));
FreeAndNil(FDataModule);
Stopped := true;
end;

(BTW, in Delphi 2006 you can only prevent data module from being created automatically in service application by removing the appropriate CreateForm line in the project source.)

It's obvious that data module's OnCreate is now called in the context of the same thread as services OnStart, but just to be sure we can install the service again and check the log:
Create called from process 2472 thread 4344
Destroy called from process 2472 thread 4344
Create called from process 4824 thread 5216
Start called from process 4824 thread 3144
DM Create called from process 4824 thread 3144
Stop called from process 4824 thread 3144
DM Destroy called from process 4824 thread 3144
Destroy called from process 4824 thread 5216
Create called from process 3588 thread 332
Destroy called from process 3588 thread 332

The moral for today: Dont drop anything on the service form. Ever!

Complete demo project is available at: http://17slon.com/blogs/gabr/files/svc_demo.zip

Wednesday, July 12, 2006

Each day I learn something new

(or at least I try to)

Today I found the Delphi Wiki. Not very full - yet - but still contains some useful info. It is waiting for your additions at http://delphi.wikia.com/wiki/Main_Page.

Also, Deepak Shenoy's blog entry on live templates and PInvoke contains many useful links:
And it is a very interesting coverage on extending live templates.

Updated on 2006-07-17: I just found a wonderful Delphi blog aggregator: DelphiFeeds.com. Recommended!

Wednesday, June 28, 2006

Multi-monitor Delphi

As many developers have already noted, developing with Delphi 2005/2006 on a single monitor is a major pain in every part of the body. That's why I hate developing on my notebook. Code fixing is doable, barely, but serious development is out of the question.

Many developers have also noted that 'undocked' layout works better on multiple
monitors. I don't think so - D2006 works just fine in 'docked' mode. The important
thing is to undock some of the windows and keep others docked or everything goes
kaboom.

Something you definitely don't want to do is undock Object Inspector and Messages.
Weirdest things start happening if you do.

In case you're wondering how I have organized my Delphi desktop, here's the picture:

normal work environment



As you can see, my primary monitor is on the right and secondary on the left. This configuration alone makes problems in VCL-based programs. For example, poDesktopCenter form position is calculated incorrectly - but that's another story.

Main monitor is filled with the Editor (of course!), onto which Object Inspector, Tool Palette, and Messages are docked. I said it before and I'll say it again: it is a wise man who keeps Object Inspector and Messages docked to the Editor.

On the left monitor I have Project Manager and ModelMake Code Explorer docked together (also, when needed, Model View appears here), small windows for To-Do List and Structure (to see syntax errors while I'm typing), another window shared between Find References and Refactorings and one hold-them-all container where other useful stuff is bunched together (Clipboard History from GExperts, Breakpoint List, Event Log etc).

I also have a slightly narrower layout for the time when I need more space (to watch TV, for example):

slightly narrower layout



In this layout, Refactorings and Find References are moved into the hold-them-all window and other windows are rearranged to use less space.

Sometimes I need lots of space - for example when comparing current code with an older version, or to look at the project specifications, various standards, internet pages ... To use Delphi on a single monitor, I keep Project Manager and Tool Palete docked and minimized on the right side of the Editor:

tweaked 'default' layout



Rarely used windows (Refactorings, Find References etc) take their share off the Editor space (for the short time they are visible):

tweaked 'default' layout with Find References



When debugging, I'm usually only using one monitor (and in some rare cases even less):

debug layout



That leaves me one full monitor to run the application on. This helps a lot, especially when debugging OnPaint handlers.

Monday, June 26, 2006

Remote access to the fourth power

Just a little list of what I'm using when I need remote access or remote control.

On the top of the list is Microsoft's Remote Deskop. Besides the accessibility (included in standard Windows XP installation), it is also blindingly fast, especially on DSL connections. Even more, it still works fine on dual ISDN connection (128 Kb/s).

What I hate about it is that it messes with my desktop. I'm running dual monitor configuration and everytime I connect remotely to my computer, RD moves all my desktop icons to the primary monitor. Yuck! (A small shell extension named Desktop Icons Save and Restore helps me restore them, but still - Yuck!)

That's why I sometimes use TightVNC for the same purpose. Not so fast, not so good (sometimes the screen is not redrawn when it should be), but it works quite fine on very fast networks; for example on the internal 100 Mb/s Ethernet.

When I'm using notebook and main computer at the same time, I usually use Synergy to control both machines with one keyboard and mouse. Read more about it in Steve Trefethen's blog.

Most rarely used of the great four (and the only one that is not free) is the MaxiVista. It allows you to turn additional computer (a notebook, for example) into an additional screen for your main computer. So when my two displays are not enough, I place the notebook on the table next to them, fire MaxiVista client on the notebook, MaxiVista server on the main computer and configure screen options to use the third monitor. MaxiVista handles image transfer over the network very well and most of the time I don't even notice that the third screen is not physically attached to my main computer.

Saturday, June 24, 2006

Compiler-generated getters and setters

(Sorry, not today. Maybe in a not-too-distant future.)

A very common task when implementing interfaces is writing getter and setter methods. In many cases, getters only access internal fields. In some cases, even setters only access internal fields. Still, we have to code those pesky methods by hand (or in a slightly simpler way using great ModelMaker Code Explorer - oh I like it so much!). I think that Delphi compiler could lend us a hand here and automatically generate trivial accessors when needed.

Let’s take a look at the example:

type
ISimpleInterface = interface
function GetProp1: integer;
procedure SetProp1(value: integer);
property Prop1: integer read GetProp1 write SetProp1;
end;
TSimpleObject = class(TInterfacedObject, ISimpleInterface)
private
FProp1: integer;
public
function GetProp1: integer;
procedure SetProp1(value: integer);
end;
{ TSimpleObject }
function TSimpleObject.GetProp1: integer;
begin
Result := FProp1;
end;
procedure TSimpleObject.SetProp1(value: integer);
begin
FProp1 := value;
end;

Even though we only want to implement the Prop1 property using direct access to a field, we still have to write two very dull methods.

We should be able to tell the compiler to do that for us. It is quite obvious that the compiler has enough data (it already knows the getter/setter signatures from the interface declaration) to generate both methods. The main question is how to instruct it to do so.

At the beginning I intended to abuse the ‘automated’ keyword (as it is already tokenized by the parser) like so:
TSimpleObject = class(TInterfacedObject, ISimpleInterface)
private
FProp1: integer
public
property Prop1: integer read GetProp1 automated FProp1 write SetProp1 automated FProp1;
end;

This was not very optimal as it requires you to declare the Prop1 property in the TSimpleObject, which is not requirement per se when implementing interfaces.

Next idea was:
TSimpleObject = class(TInterfacedObject, ISimpleInterface)
private
FProp1: integer;
public
property Prop1: integer read GetProp1 = FProp1 write SetProp1 = FProp1;
end;

It doesn’t abuse the ‘automated’ keyword but carries same problems.

My current favorite, suggested by a fellow programmer Lee_Nover, uses same syntax as method aliasing:
TSimpleObject = class(TInterfacedObject, ISimpleInterface)
private
FProp1: integer;
public
function ISimpleInterface.GetProp1 = FProp1;
procedure ISimpleInterface.SetProp1 = FProp1;
end;

Although this syntax is very clear and self-describing, I would prefer to see a following simplified form:
TSimpleObject = class(TInterfacedObject, ISimpleInterface)
private
FProp1: integer;
public
function GetProp1 = FProp1;
procedure SetProp1 = FProp1;
end;

So, dear reader, what do you think? If you like the suggestion, open QC #30533 and vote for it.

Friday, June 23, 2006

GpProfile, encore

Once upon a time, there was a nice little profiler called GpProfile. Admittedly, it could be better, faster, and more user friendly, but I wrote it and I liked it. That was then.

But now, now is seven years later and that is in programmer's years, which is 49 normal years. Or something like that. And those years are showing, trust me if I tell you.

So there came a time to rewrite the damn thing from scratch. I'm a great fan of Office 2007 GUI and I like to have my internals modularized and organised (OK, not really my internals ...) so the new version will have a nice clean look and even nicer and cleaner source code.

Just for teaser, here is the old interface:
GpProfile 1.3

and here's the new one:
GpProfile 2.0 alpha

There is always a beginning ...

And it is now. 20060623T201433 UTC.

(Hey, time is important. And time is interesting. I wrote an article on time a long time ago.)

If you want to know more about me, read the profile. Or Google me. Or ask. I'll tell. Or not.

Enough for now.

... and there is always an end. But we have yet to see when that will be.