Using configuration records and operators to reduce number of overloaded methods

When writing libraries you sometimes want to provide users (that is, programmers) with a flexible API. If a specific part of your library can be used in different ways, you may want to provide multiple overloaded methods accepting different combinations of parameters.

For example, IOmniPipeline interface from OmniThreadLibrary implements three overloaded Stage functions.

function  Stage(pipelineStage: TPipelineSimpleStageDelegate; 
  taskConfig: IOmniTaskConfig = nil): IOmniPipeline; overload;

function  Stage(pipelineStage: TPipelineStageDelegate; 
  taskConfig: IOmniTaskConfig = nil): IOmniPipeline; overload;

function  Stage(pipelineStage: TPipelineStageDelegateEx; 
  taskConfig: IOmniTaskConfig = nil): IOmniPipeline; overload;

Delphi’s own System.Threading is even worse. In class TParallel, for example, there are 32 overloads of the &For class function. Thirty two! Not only it is hard to select appropriate function; it is also hard to decode something useful from the code completion tip. Check the image below – can you tell which overloaded version I’m trying to call? Me neither!

Because of all that, it is usually good to minimize number of overloaded methods. We can do some work by adding default parameters, but sometimes this doesn’t help. Today I’d like to present an alternative solution – configuration records and operator overloading. To simplify things, I’ll present a mostly made-up problem. You can download it from github.

Introducing MultiBuilder

When I'm working on OmniThreadLibrary, I have to keep backwards compatibility in mind. And man, is that hard! OmniThreadLibrary supports every Delphi from 2007 onward and that means lots of IFDEFs and some ugly hacks.

Typically I develop new stuff on Berlin or Tokyo and then occasionally start a batch script that tests if everything compiles and runs unit tests for all supported platforms in parallel. (I wrote about that system two years ago in article Setting Up a Parallel Build System.) Dealing with 14 DOS windows, each showing compilation log, is cumbersome, though, and that's why I do this step entirely too infrequently.

For quite some time I wanted to write a simple framework that would put my homebrew build batch into a more formal framework and which would display compilation results in a nicer way. Well, this weekend I had some time and I sat down and put together just that - a MultiBuilder. I can promise that it will be extensively used in development of OmniThreadLibrary v4. (Which, incidentally, will drop support for 2007 to XE. You've been notified.)

The rest of this post represents a short documentation for the project, taken from its GitHub page.

I don't plan to spend much time on this project. If you find it useful and if you would like to make it better, go ahead! Make the changes, create a pull request. I'll be very happy to consider all improvements.

Writing a Simple DSL Compiler with Delphi [7. AST Compiler]

This article provides a description of an AST compiler used for my toy language project. If you are new to this series, I would recommend to start reading with this post. At least you should read the previous post, Intermezzo, as it explains some parts of the compiler that I won't touch here.

Please note that this article describes an initial implementation of the compiler. If you want to browse the code while reading the article, make sure that you have switched to branch dsl_v1.

In my toy compiler framework, a compiler (or codegen as it is called internally), is a piece of code that implements the ISimpleDSLCodegen interface. This interface exposes only one function, Generate, which takes an abstract syntax tree and converts it into an object implementing an ISimpleDSLProgram interface which allows you to call any function in a compiled program by name.

type

  TParameters = TArray;

  TFunctionCall = reference to function (const parameters: TParameters): integer;

  ISimpleDSLProgram = interface ['{2B93BEE7-EF20-41F4-B599-4C28131D6655}']

    function  Call(const functionName: string; const params: TParameters;       var return: integer): boolean;

   end;

  ISimpleDSLCodegen = interface ['{C359C174-E324-4709-86EF-EE61AFE3B1FD}']

    function Generate(const ast: ISimpleDSLAST;      
      var runnable: ISimpleDSLProgram): boolean;

  end;

Writing a Simple DSL Compiler with Delphi [Intermezzo]

When I was preparing an article about the compiler part of my toy language project, I found out that the concept of wrapping a whole program into a bunch of anonymous functions (what the compiler does) is exceedingly hard to explain. I had therefore prepare a simplified version of the compiler, written for a very simplified language ... and then I couldn't stop and I added an AST, parser, tokenizer, interpreter and all shebang.

The result of all that is a program introduction.dpr, a self-contained console program which contains a complete (almost trivial) language together with the full documentation, written in a Literate Programming style. Simply put - you can read it from top to bottom like a story.

As an intermezzo and to simplify my explanation of the compiler, I'm posting the whole program here, reformatted as a blog post.

Writing a Simple DSL Compiler with Delphi [6. AST Dumper]

This article provides a description of a testing tool used for my toy language project. If you are new to this series, I would recommend to start reading with this post. 

Please note that this article describes an initial implementation of the parser. If you want to browse the code while reading the article, make sure that you have switched to branch dsl_v1.

Now that we have a working tokenizer and parser outputting an AST, we can start working on the compiler. Still, it would be great if we can verify whether the parser output (the AST) makes any sense. In other words, we need unit tests.

Writing unit tests for a tree structure is, however, a very tedious operation. In the next post I'll show a test for a tree with only five nodes and it will already be a process one would rather skip. Luckily, we can do something more fun - we can write a code that recreate the original program from an AST.

Writing a Simple DSL Compiler with Delphi [4. Parser]

This article provides a description of a parser used in my toy language project. If you are new to this series, I would recommend to start reading with this post.

Please note that this article describes an initial implementation of the parser. If you want to browse the code while reading the article, make sure that you have switched to branch dsl_v1.

After a short break I'm returning to my "toy compiler" series. This time I'll describe the working of the parser - the part of the code which reads the input (usually in a tokenized form) and generates internal representation of the program (in my case an abstract syntax tree - AST).

The goal of my project was to study the compilation step and parser was just a necessary evil that I had to deal with. That's why it is written in a pretty primitive manner, without using any improvements like a Pratt parser.

Writing a Simple DSL Compiler with Delphi [3. Tokenizer]

This article provides a description of a tokenizer used in my toy language project. If you are new to this series, I would recommend to start reading with this post.

Please note that this article describes an initial implementation of the tokenizer. If you want to browse the code while reading the article, make sure that you have switched to branch dsl_v1.

With this article I'm moving into the gritty part of the project - the code which reads the source code and turns it into a beautiful abstract syntax tree. In other words, I'll be talking about the parser.

I must admit that I spent as little time on the parser as I could. After all, my main goal was to convert AST into an executable code, not to parse input. Still, one cannot write a compiler without writing a parser, so ... here I am.

Writing a Simple DSL Compiler with Delphi [2. Abstract Syntax Tree]

This article provides a description of an abstract syntax tree used to represent "The Language". If you are new to this series, I would recommend to start reading with this post. 

Please note that this article describes an initial implementation of the AST. If you want to browse the code while reading the article, make sure that you have switched to branch dsl_v1.

An abstract syntax tree (AST) is, simply put, a symbolic representation of the program in a form of a tree.

While the textual representation of a program is great for us, humans, computers have hard time dealing with it. Because of that, a special part of any interpreter/compiler, called parser, reads the input and converts it into a computer-readable format - AST. This tree can then be used for multiple purposes. We can, for example, feed to into an interpreter which will then run the program for us, or we can feed it into a compiler to generate an executable program or cross-compiler to generate an equivalent program in a different programming language.

Writing a Simple DSL Compiler with Delphi [1. The Language]

Part 1: The Language

This article provides an informal definition of a simple language (a.k.a. "The Language") I am writing a compiler for. If you are new to this series, I would recommend to start reading with this post.

Let's start with a simple example which calculates an i-th Fibonacci number.

 fib(i) {
   if i < 3 {
     return 1
   } else {
     return fib(i-2) + fib(i-1)
   }
 }

The code is quite simple. First two numbers in a Fibonacci sequence are 1 and every other Fibonacci number is a sum of previous two in the sequence.

Writing a Simple DSL Compiler with Delphi [0. Introduction]

Part 0: Introduction

Some time ago I was listening to the Hanselminutes podcast where a guy described how he wrote a simple interpreter in Go language (YOU should write an interpreter with Thorsten Ball). I was not really interested in writing an interpreter - I did that a long time ago - but a thought crossed my mind. I asked myself whether I can do something better - write a compiler. (Or, rather, a kinda-compiler. You'll see.)

What I wanted to do is to take a small language, parse it, generate an abstract syntax tree, and then convert this into one large anonymous function calling other anonymous functions calling other anonymous functions and so on and so on. It is hard to explain, so let me try using a very simple example.

Setting Up a Parallel Build System

OmniThreadLibrary now supports 11 different Delphi versions (2007, 2009, 2010, XE, XE2, XE3, XE4, XE5, XE6, XE7, XE8), some with very special requirements about the supported pascal syntax (2007 and 2009 clearly standing out in that regard) so it takes quite some time to test the compilation of all demos and run unit tests on all supported editions. (And this time will only increase with the addition of support for mobile platforms and OS X. Not that I’m complaining. Sean is doing a terrific job there!)

It does not help that I don’t have all those Delphis installed on my computer. Most of them are only installed in a VM. And it takes a looooong time to start up a VM, run tests, power it down, start up next VM, and so on. And when I fix something, I have to retest it all ….

This kind of testing hurts. So in the manner of the Continuous Integration mantra, I decided to do it more often.

When Does “Execute on Destroy” Fail

Yesterday I wrote a post about executing some code automatically when a method ends. Today I’ll show you why this approach must be used with care.

Running “Any” Code When a Method Ends

Just like (some) other programmers, I like to abuse the fact that the Delphi compiler only destroys local interfaces when a method ends. If you don’t know what I’m talking about, check out the latest Nick’s post in the Fun Code of the Week series.

As an example, Nick put together a function which changes application cursor and at the end of the method reverts it back to the previous state.

procedure test;
begin
  AutoCursor(crHourGlass);
  // some long operation
  // cursor is automatically reverted when ‘test’ exits
end;

My approach to this pattern is usually slightly different. I like to mark the scope of the local interface with a with statement.

XE3: Internal Error G9413

Today I wanted to check whether the OmniThreadLibrary plays nice with XE3 (yes, I know, I’ve should have done this weeks ago, but there was simply not enough time) and I was completely surprised when OtlSync unit couldn’t be compiled due to an internal error G9413.

One hour later I have reduced the problem to a simple test case (below) and to a simple fix. So the OmniThreadLibrary will be supported in the XE3 (as we all hoped for).

For other poor souls that may have run into this problem, here’s a small test case and a workaround.

implementation

uses
    RTTI;

type
    TTest<T> = class
        procedure Test;
    end;

var
    a: IInterface;

procedure TTest<T>.Test;
var
    aMethCreate: TRttiMethod;
begin
    if Length(aMethCreate.GetParameters) = 0 then
    ;
end;

initialization
    a := nil; // F2084 Internal Error: G9413
end.

The problem here is that the G9413 internal error is not reported on the line which causes the problem. The real culprit here is the ‘if Length(aMethCreate.GetParameters)’ call in the ‘TTest<T>.Test’ method.

To fix the problem it is enough to introduce a temporary variable that stores the result of the GetParameters call.

unit Unit136;

interface

implementation

uses
    RTTI;

type
    TTest<T> = class
        procedure Test;
    end;

var
    a: IInterface;

procedure TTest<T>.Test;
var
    aMethCreate: TRttiMethod;
    params     : TArray<TRttiParameter>;
begin
    params := aMethCreate.GetParameters;
    if Length(params) = 0 then
        ;
end;

initialization
    a := nil;
end.

The problem is reported as QC #108942.

Creating an Object from an Unconstrained Generic Class

As you know if you follow my blog, OmniThreadLibrary now offers a simple way to do optimistic and pessimistic atomic initialization which works for interfaces, objects and (in the case of the pessimistic initialization), anything else. [In case you missed those articles - I also discussed a comparison of both methods and wrote a short post about the third approach to initialization.]

A typical usage of both types of initialization would be:

var
  sl: TStringList;
  ol: Locked<TObjectList>;

Atomic<TStringList>.Initialize(sl,
  function: TStringList
  begin
    Result := TStringList.Create;
  end);

ol.Initialize(
  function: TObjectList
  begin
    Result := TObjectList.Create;
  end);

As you can see, this is pretty long-winded. If you are initializing an interface, then you’ll usually have written a factory method already and the code would be much simpler (example below this paragraph) but in the case of objects this is not very typical.

Atomic<IGpIntegerList>.Initialize(list, TGpIntegerList.CreateInterface);

So I thought – both Atomic and Locked already know what entity type they wrap around so calling a constructor from inside Initialize would be trivial, eh? I could then write a simplified version

Atomic<TStringList>.Initialize(sl);
ol.Initialize;

and use the longer version only when needed, for example to initialize interfaces or to call a non-default object constructor. What could possibly go wrong?

How to crash Delphi compiler in four easy steps

Step one

Start Delphi XE.

Step Two

Create simple project.

Faster CopyRecord required

As we saw yesterday, CopyRecord can be a source of substantial slowdown if records are used extensively. I can see only way to improve the situation – fix the compiler. It should be able to generate custom CopyRecord for each record type (or at least for “simple” records, however that simplicity is defined) and that would speed all record operations immensely.

To push this idea, I’ve created a QC report #83084. If you think this would be a significant improvement to the compiler, make sure to vote on that report.

And while you’re busy voting, I’d just like to state that I also find QC #47559 important (hint, hint).