Writing a Simple DSL Compiler with Delphi [7. AST Compiler]

This article provides a description of an AST compiler used for my toy language project. If you are new to this series, I would recommend to start reading with this post. At least you should read the previous post, Intermezzo, as it explains some parts of the compiler that I won't touch here.

Please note that this article describes an initial implementation of the compiler. If you want to browse the code while reading the article, make sure that you have switched to branch dsl_v1.

In my toy compiler framework, a compiler (or codegen as it is called internally), is a piece of code that implements the ISimpleDSLCodegen interface. This interface exposes only one function, Generate, which takes an abstract syntax tree and converts it into an object implementing an ISimpleDSLProgram interface which allows you to call any function in a compiled program by name.

type

  TParameters = TArray;

  TFunctionCall = reference to function (const parameters: TParameters): integer;

  ISimpleDSLProgram = interface ['{2B93BEE7-EF20-41F4-B599-4C28131D6655}']

    function  Call(const functionName: string; const params: TParameters;       var return: integer): boolean;

   end;

  ISimpleDSLCodegen = interface ['{C359C174-E324-4709-86EF-EE61AFE3B1FD}']

    function Generate(const ast: ISimpleDSLAST;      
      var runnable: ISimpleDSLProgram): boolean;

  end;

OmniThreadLibrary 3.07.4 has been released

Bug fixes:

• TOmniEnumeratorProvider and TOmniValueEnumeratorProvider support dmoPreserveOrder option. Now you can use PreserveOrder modifier on Parallel.ForEach when input is IEnumerable, IEnumerator, TEnumerator, or TEnumerable.
• Fixed 64-bit issues in DSiWin32, GpLists, GpStringHash, and GpStuff.

New features:

• Locked.Value is now both readable and writable property.  
• Moved 'inline' functions around so that they will be inlined more often.

As usual, you can get OmniThreadLibrary from GitHub (3.07.4, HEAD), download the zip, install it with GetIt or with Delphinus.

Writing a Simple DSL Compiler with Delphi [Intermezzo]

When I was preparing an article about the compiler part of my toy language project, I found out that the concept of wrapping a whole program into a bunch of anonymous functions (what the compiler does) is exceedingly hard to explain. I had therefore prepare a simplified version of the compiler, written for a very simplified language ... and then I couldn't stop and I added an AST, parser, tokenizer, interpreter and all shebang.

The result of all that is a program introduction.dpr, a self-contained console program which contains a complete (almost trivial) language together with the full documentation, written in a Literate Programming style. Simply put - you can read it from top to bottom like a story.

As an intermezzo and to simplify my explanation of the compiler, I'm posting the whole program here, reformatted as a blog post.

Writing a Simple DSL Compiler with Delphi [6. AST Dumper]

This article provides a description of a testing tool used for my toy language project. If you are new to this series, I would recommend to start reading with this post. 

Please note that this article describes an initial implementation of the parser. If you want to browse the code while reading the article, make sure that you have switched to branch dsl_v1.

Now that we have a working tokenizer and parser outputting an AST, we can start working on the compiler. Still, it would be great if we can verify whether the parser output (the AST) makes any sense. In other words, we need unit tests.

Writing unit tests for a tree structure is, however, a very tedious operation. In the next post I'll show a test for a tree with only five nodes and it will already be a process one would rather skip. Luckily, we can do something more fun - we can write a code that recreate the original program from an AST.

Writing a Simple DSL Compiler with Delphi [5. Framework]

This article provides a description of a compiler framework used in my toy language project. If you are new to this series, I would recommend to start reading with this post.

We have now a working parser that converts a string of code into an abstract syntax tree - AST. It is, however, not yet time to write about the most interesting piece - the compiler - as we should first do some integration and testing.

Writing a Simple DSL Compiler with Delphi [4. Parser]

This article provides a description of a parser used in my toy language project. If you are new to this series, I would recommend to start reading with this post.

Please note that this article describes an initial implementation of the parser. If you want to browse the code while reading the article, make sure that you have switched to branch dsl_v1.

After a short break I'm returning to my "toy compiler" series. This time I'll describe the working of the parser - the part of the code which reads the input (usually in a tokenized form) and generates internal representation of the program (in my case an abstract syntax tree - AST).

The goal of my project was to study the compilation step and parser was just a necessary evil that I had to deal with. That's why it is written in a pretty primitive manner, without using any improvements like a Pratt parser.

Delphi and Linux

I'd just like to remind my Slovenian readers that on 28th this month I'll be having a presentation about RAD Studio and Linux in Ljubljana.

As the presentation will be given in Slovenian language, the rest of my post containing the description of the presentation is written in that language, too.

Vljudno vabljeni na delavnico "RAD Studio in Linux", na kateri si bomo ogledali:

• Kako namestiti Ubuntu v virtualni računalnik.
• Kaj storiti, ko namestitev nagaja.
• Kako pripraviti virtualni računalnik in RAD Studio za delo.
• Kako napisati konzolno aplikacijo za Windows in Linux, ki streže podatke z uporabo tehnologij FireDAC in DataSnap.
• Kako napisati grafično aplikacijo za Windows, OS X, iOS, Android, ki prikazuje in spreminja podatke na aplikacijskem strežniku iz prejšnjega koraka.
• Z nekaj sreče pa še: 
• Kako to grafično aplikacijo pognati na Linuxu in
• Kako konzolno aplikacijo spremeniti v pravi Linux "service".

Vstop prost, prosimo vas le, da se vnaprej registrirate, da bomo znali pripraviti zadostno količino kave in prigrizkov ;)

Autumn is coming ...

... and with it, a whole bunch of events.

First, I'll be going to IBC 2017 in Amsterdam where we are presenting our products in Hall 2. I'll be there from 15th to 18th, so if anybody wants to meet, contact me.

Immediately after that I'm going to Zegrze (just north of Warsaw) to speak at Zlot Programistów Delphi (Delphi programmers convention) on 21st and 22nd. Never been there before, should be great fun.

A week later (28th) I'll be leading a workshop on Delphi and Linux in Ljubljana. (Slovenian readers - you can sign on now.)

To finish it up, I'll present two all-new topics in ITDevCon in Rome on October 11th and 12th.

All that in four weeks. Should be quite an interesting time ;)

Introducing OmniThreadLibrary Core

Yesterday I wanted to use OmniThreadLibrary as a git submodule in a top secret ;) open source project I'm working on and I was a bit shocked when it turned out that newly cloned OmniThreadLibrary folder is a 83 megs in size. Given that I only need it to support my other project and that I won't do any OTL fixing/development in this submodule, that looked a bit excessive.

So I went ahead and created a core branch which contains only the barebones - the OmniThreadLibrary root folder without any subfolders. If you want to compile such submodule, you'll also need the GpDelphiUnits repo. Together they weight measly 3,5 MB, which is a big improvement over the original 80+ megs.

I'll be keeping core in sync with the latest OTL release, not with the HEAD, so it will also provide a stable platform for all depending repositories.

Writing a Simple DSL Compiler with Delphi [3. Tokenizer]

This article provides a description of a tokenizer used in my toy language project. If you are new to this series, I would recommend to start reading with this post.

Please note that this article describes an initial implementation of the tokenizer. If you want to browse the code while reading the article, make sure that you have switched to branch dsl_v1.

With this article I'm moving into the gritty part of the project - the code which reads the source code and turns it into a beautiful abstract syntax tree. In other words, I'll be talking about the parser.

I must admit that I spent as little time on the parser as I could. After all, my main goal was to convert AST into an executable code, not to parse input. Still, one cannot write a compiler without writing a parser, so ... here I am.

Writing a Simple DSL Compiler with Delphi [2. Abstract Syntax Tree]

This article provides a description of an abstract syntax tree used to represent "The Language". If you are new to this series, I would recommend to start reading with this post. 

Please note that this article describes an initial implementation of the AST. If you want to browse the code while reading the article, make sure that you have switched to branch dsl_v1.

An abstract syntax tree (AST) is, simply put, a symbolic representation of the program in a form of a tree.

While the textual representation of a program is great for us, humans, computers have hard time dealing with it. Because of that, a special part of any interpreter/compiler, called parser, reads the input and converts it into a computer-readable format - AST. This tree can then be used for multiple purposes. We can, for example, feed to into an interpreter which will then run the program for us, or we can feed it into a compiler to generate an executable program or cross-compiler to generate an equivalent program in a different programming language.

Writing a Simple DSL Compiler with Delphi [1. The Language]

Part 1: The Language

This article provides an informal definition of a simple language (a.k.a. "The Language") I am writing a compiler for. If you are new to this series, I would recommend to start reading with this post.

Let's start with a simple example which calculates an i-th Fibonacci number.

 fib(i) {
   if i < 3 {
     return 1
   } else {
     return fib(i-2) + fib(i-1)
   }
 }

The code is quite simple. First two numbers in a Fibonacci sequence are 1 and every other Fibonacci number is a sum of previous two in the sequence.

Writing a Simple DSL Compiler with Delphi [0. Introduction]

Part 0: Introduction

Some time ago I was listening to the Hanselminutes podcast where a guy described how he wrote a simple interpreter in Go language (YOU should write an interpreter with Thorsten Ball). I was not really interested in writing an interpreter - I did that a long time ago - but a thought crossed my mind. I asked myself whether I can do something better - write a compiler. (Or, rather, a kinda-compiler. You'll see.)

What I wanted to do is to take a small language, parse it, generate an abstract syntax tree, and then convert this into one large anonymous function calling other anonymous functions calling other anonymous functions and so on and so on. It is hard to explain, so let me try using a very simple example.

OmniThreadLibrary 3.07.3

TL;DR: Update OmniThreadLibrary now!

A nasty bug was found in the DSiWin32 library. It causes the DSiTimeGetTime64 function to work incorrectly when called from multiple threads at the same time. As this function is central to time measurement in the OmniThreadLibrary, it was essential to release new, fixed version.

This version also contains two small enhancements.

• SetTimer method now accepts TProc and TProc timer methods.
• IOmniTask implements method InvokeOnSelf which can be used to schedule anonymous function execution from a task back to self.  

As usual, you can get OmniThreadLibrary from GitHub (3.07.3, HEAD), download the zip, install it with GetIt or with Delphinus.

Damn, multithreading is hard!

OmniThreadLibrary 3.07.2

This is just a small update which adds few helpful methods. Although the change log mentions a potentially breaking change I don't think this will affect anybody. (And if you are not sure, you can enabled backward-compatible behaviour by adding one line to the code.)

I have also (finally) found some time to work on The book. It is now fully up to date (all OmniThreadLibrary features are documented on its whooping 293 pages) and is only missing one or two introductory chapters.

As usual, you can get OmniThreadLibrary from GitHub (3.07.2, HEAD), download the zip, install it with GetIt or with Delphinus.

Change log follows after the break.

RAD Studio 10.2 links

This post contains just a bunch of links to online resources that I mentioned on the today’s presentation.

10.2 Tokyo comes to Slovenia

In two days I’ll present RAD Studio 10.2 Tokyo in Ljubljana.

Click here for more information and to register.

OmniThreadLibrary 3.07.1

This update brings only few small changes. The biggest of them is support for the Delphi 10.2 Tokyo.

You can get it as a zipped download, git checkout, via Delphinus, and via GetIt.

For more information on OmniThreadLibrary, go to www.omnithreadlibrary.com.

Forward record declaration

Forward declaration is not really a new concept. It was already present in the original Wirth Pascal where it allowed the programmer to do one thing and one thing only – call procedure A from procedure B and procedure B from procedure A. Remember – in ye olde times of good old Pascal we had no interfaces, no classes, no units … just procedures and functions.

Because the code tells more than thousand words, here’s an example (still perfectly valid in modern Object Pascal).

procedure ProcA; forward;

procedure ProcB;
begin
  ProcA;
end;

procedure ProcA;
begin
  ProcB;
end;

OmniThreadLibrary 3.07

New year, new release ;) You can get it as a zipped download, git checkout, via Delphinus, and via GetIt.

For more information on OmniThreadLibrary, go to www.omnithreadlibrary.com.

OmniThreadLibrary 3.07 beta + plans for OmniThreadLibrary 4.0

It is almost time for a new release! If you’d like to see what 3.07 will contain – or if you just want to test it before it is out – click here.

[If you are a serious OTL user - and especially if you are using pre-Seattle Delphi - please test your program(s) with the beta release. Thanks!]

I have started work on OTL 4.0 which will (fingers crossed) fully support cross-platorm work. Yup, it will work on all platforms that Delphi can compile for! Multiplatform support was implemented by Sean B. Durkin so if OTL/Mobile is something you were waiting for a long time, go to OmniThreadLibrary-For-Mobile Google+ community and express your gratitude.

RAD Studio and Version Control + Git – links

This post contains just a bunch of links that I refered to in my two-part (live) presentation on version control management with RAD Studio and Git.

Slides for the presentation: pps, pdf

Embarcadero Akademija: RAD Studio and Version Control

This is an announcement for the next Slovenian “Embarcadero Academy” session. As it is intended for Slovenian Delphi programmers, the rest of this post is in the Slovenian language.

Debug Faster. Debug Smarter.

My CodeRage XI session on debugging in RAD Studio is now available on YouTube.

If you have missed the presentation or want to see it again, now is a good time :)

Additional materials (slides, code) are available on the Presentations page.

Hear me speak at CodeRage XI

This year I’ll be presenting two sessions at CodeRage XI.

On Wednesday, I’ll be exploring the topic of debugging in RAD Studio. I’ll show how advanced options in breakpoint properties work and drop some debugging tips & tricks here and there. Join me in room 1 at 7 am PST / 10 am EST / 16.00 CET.

On Thursday I’ll be demoing new enhancements in FastMM. If you are writing multithreaded code, then this session will be targeted to you! Join me in room 1 at 10 am PST / 1 pm EST / 19.00 CET.

Recording Webinars (repost)

As I’m pretty sure many Delphi programmers will be working on their CodeRage XI videos this weekend ;) , I decided to repost a link to my year old post Recording Webinars – My Workflow. Maybe it will be of some use to you.

OmniThreadLibrary 3.06.2

Another week, another critical update …

There was a logical error – also known as “my own stupidity” – built into OTL 3.06 which prevented programs that used OtlCommon to be started on Windows XP. This is now fixed.

There’s also a tiny addition to OtlSync included in this release.

download / git

OmniThreadLibrary 3.06.1

There was a nasty bug in DSiWin32 unit included with the 3.06 release so I had no other option than to quickly issue a fix. If you have downloaded version 3.06 then please update to 3.06.1 as with 3.06 you could experience weird crashes during program startup. I’m really sorry for that :(

There’s also a tiny addition to TOmniValue included in this release.

download / git

OmniThreadLibrary 3.06

Hi, guys! After a looong time, a new version of OmniThreadLibrary was released! It is available as a zipped download, git checkout, via Delphinus and (really soon) via GetIt.

Following stuff was added/changed/fixed since the 3.05 release:

ITDevCon 2016

ITDevCon was - as always before - a blast! Great topics, great presenters, great organization and great food!

My presentations (slides and code) are now online at the usual place.

OmniThreadLibrary Everywhere

For the last year and some, Sean B. Durkin has been working on the mobile port of the OmniThreadLibrary. There was lots to be done, as OTL was never designed to be a portable library – after all, it was created in Delphi 2007 times.

Recently I merged his latest commits into the mobile OmniThreadLibrary branch. This version is (by his words) functionally complete (although it may be missing some very recent OTL features) but is very much untested. So Sean is asking everybody that is interested in multithreading on OS/X, iOS, and Android to help him find bugs and to make OTL even more awesome!

Please report any findings to his OmniThreadLibrary-For-Mobile-DevelopmentProject on Google+. I am currently working on other aspects of the OTL and didn’t I put much energy into checking his work.

When 64 cores are not enough

Just a little teaser …

Following two branches just sprung to life:
https://github.com/gabr42/OmniThreadLibrary/tree/numa
https://github.com/gabr42/FastMM4-MP/tree/numa

Plus I started planning OTL&FastMM changes:
https://docs.google.com/document/d/1dHiJyHj80TJ2oFAOtFxNtXLMrAhzm7F5SoczBl5ssxM

Feel free to comment on that Google Docs document.

OmniXML Official Repository

In case you are wondering which of the many github (and even bitbucket) omnixml repos is the official one (as www.omnixml.com is still pointing to the old googlecode storage), this is it: https://github.com/mremec/omnixml.

www.omnixml.com should be pointing to the correct repository “real soon now” ;)

RAD Studio Berlin 10.1 Notes and Links

In case you are already using RAD Studio Berlin 10.1 or are merely looking into it, these links and notes may help you.

• What’s New in RAD Studio 10.1 Berlin
• Using C++ lambdas with Parallel Programming Library
• Marco Cantu on [weak] and [unsafe] attributes
• My [weak] vs. [unsafe] demo
• FireUI preview
• Bookmark stack
• Selection expansion (which is my totaly favourite new feature!)
• TBufferedFileStream
• DataSnap Hotfix
• Samsung Galaxy Hotfix
• Samples are installed in c:\Users\Public\Documents\Embarcadero\Studio\18.0\Samples.
• New installer stores files to c:\Users\<user>\Documents\Embarcadero\Studio\18.0\CatalogRepository.
• FireUI Preview source can be found in c:\Program Files (x86)\Embarcadero\Studio\18.0\source\Tools\FireUIAppPreview.

YouTube videos

• Introducing 10.1 Berlin
• FireUI Preview demo starts at 32:20
• New installer
• ListView Item Designer

RAD Studio 10.1 Berlin presentation in Ljubljana

Next week (Wednesday 25th) I’ll be presenting the new and shiny 10.1 release in Ljubljana.

For more info click here.

Defensive Programming and Advanced Debugging

Next Wednesday (9th) I’ll be holding a presentation in Ljubljana. Topic: defensive programming, maintaining a clean code, and debugging tips and tricks. Presentation will be applicable to both Delphi and C++Builder programmers.

As usual, I’ll be presenting in Slovenian language.

For more information and registration click here!

Finding Memory Allocation Bottlenecks with FastMM4

Recently, I had to find bottlenecks in one of our applications that does all sorts of things related to DVB and is handling real-time reading and sending of data streams over IP with bitrates up to 80 Mb/s (in our lab; in real life maybe even more). Our customer created a configuration which essentially brought the app to the crawl and I had to fix it.

It quickly turned out that although the program was not able to handle the load, the CPU was not very busy. The busiest core was only using about 30% CPU. So I suspected the thread contention problems in FastMM, switched it for SapMM and indeed - the problem went away. CPU load went up and application could again handle the load.

Crisis averted, I took time to find the real problem - excessive get/freemem calls in this program. As far as I could tell, there existed no tool to find that so in a true DIY manner I created my own ;)

Connect Your Applications with REST – Links

The materials for my “Connect your applications with REST” workshop are now available online (in Slovenian language).

Below you can find all the interesting DELPHI- and REST-related links I have collected during the preparation of this workshop.

Connect Your Applications with REST

Next Wednesday (18th) I’ll be holding a presentation about all things RESTish in Delphi. We’ll start with basics and then move to DataSnap, REST Client Library, REST Debugger, BaaS and more!
As usual, I’ll be presenting in Slovenian language.
Register here!

Simplify Parallel Programming with Patterns

My CodeRage X session on parallel programming with patterns is now available on YouTube.

You can also download the slides (pps, pdf) and code.

Recording Webinars – My Workflow

Each time I have to record a webinar, I have trouble remembering how exactly did I do it last time, so I decided to document my workflow. Maybe this will help somebody else besides me …

If you have any ideas on how to improve my workflow or if you have a different way of solving same problems, then please post it in the comment section. Thanks!

[v 1.01: Added screenshots of CamStudio and Filmora settings and few words on Audacity settings.]
[v 1.02: Added appendix A with some time&space information.]

Updating a Progress Bar From a Parallel For Loop (Plus Two Bonuses)

During the Q&A part of my Simplify Parallel Programming with Patterns presentation on CodeRage X, I’ve promised the listeners to publish a demo for updating a progress bar from a parallel for loop.

In this article I’ll try to explain few different approaches that all solve this problem. I’ve also put together a demo project which demonstrates all techniques.

Links for 10 Seattle Presentation in Ljubljana

Firstly, thanks to everybody who visited my RAD Studio 10 Seattle presentation! I’m sorry I had to improvise because of bad video drivers, but on the other hand RAD Studio performed great!

Secondly, here is a bunch of links I’ve promised.

OmniThreadLibrary Spotlight now On-line

My short spotlight presentation about high-level OmniThreadLibrary patterns is now available on YouTube.

Conditional Compilation with Features

EDIT: Make sure you read this post to the very end!

Are you developing a code that must work in different Delphi versions? Are you sick of writing code like

{$IF CompilerVersion >= 22}
class function Mapper<T1,T2>: IMapper<T1,T2>;
{$ENDIF}

as you a) never know when some feature was introduced into Delphi and b) constantly wonder which version of Delphi has CompilerVersion equal to 22?

Then maybe you could use this simple include file.

Learn All About RAD Studio 10 “Seattle” in Ljubljana

Next Thursday (24th) I’ll be presenting new RAD Studio 10 Seattle in Ljubljana. This is announcement for Slovenian readers.

Naslednji četrtek (24.) se mi pridružite v kristalni palači na predstavitvi novega RAD Studia 10 Seattle, ki je po mnenju mnogih (in tudi mojem), ena boljših različic tega programskega orodja v zadnjih letih.

Predstavil vam bom izboljšano podporo za Windows 10 (v VCL in FMX), nove vizualne gradnike, izboljšave v IDE, novo podporo za mobilna okolja (razhroščevanje iOS 64-bit programov, pisanje sistemskih storitev za Android), novosti in izboljšave v sistemskih knjižnicah RTL, pa tudi novosti v prevajalnikih za C++ in še marsikaj.

Več podatkov o dogodku najdete tu.

OmniThreadLibrary 3.05

Hear, hear, OmniThreadLibrary 3.05 has just been released!

What’s new? Not much.

• Added Delphi 10 Seattle packages and projects. OmniThreadLibrary now supports Delphis from 2007 to 10 Seattle.
• Setting IOmniThreadPool.MinWorkers property will start up idle worker threads if total number of threads managed by the thread pool is lower than the new value.

There were, however, many quite important bug fixes. You are strongly advised to update to this version!

OmniThreadLibrary on Delphinus

OmniThreadLibrary now supports the Delphinus package installer. If you are using Delphi XE or newer, you can install Delphinus, select OmniThreadLibrary, and click Install. Delphinus will do the same steps GetIt does: download OTL, recompile and install package, and add proper paths to the search and browsing path. Source will be installed to c:\Users\Public\Documents\Embarcadero\Studio\<release>\Comps\.

Spotlight on OmniThreadLibrary

If you are not using OmniThreadLibrary yet, or if you are only using its low-level parts (CreateTask and company), I’m kindly inviting you to a 20-minute presentation “Simple Parallel Programming with Patterns and OmniThreadLibrary”, which will be transmitted as a part of the Embarcadero Technology Partner Spotlights series on September 3rd.

There are three timeslots to chose from so I’m sure everybody can find a suitable time. After the first two presentations I’ll be handling Questions and Answers and Jim McKeeth has kindly offered to do the job for the third presentation as it will be around 02.25 AM here when it is finished.

If you want to see something interesting and new, register here!

[Slides, code, and video will be available on my blog after the presentation.]

OmniThreadLibrary 3.04b on XE8

Just a quick note – OmniThreadLibrary 3.04b is now available on XE8 via GetIt.

Book Update: Miscellaneous

I have updated the Parallel Programming with OmniThreadLibrary book with the Miscellaneous chapter. The book is now almost complete!

Changelog:

• Documented TOmniMessageQueue.
• Documented TOmniTwoWayChannel.
• Documented TOmniValueContainer.
• Documented TOmniCounter.
• Documented TGp4AlignedInt and TGp8AlignedInt.
• Documented TOmniRecordWrapper.
• Documented IOmniAutoDestroyObject.
• Documented IOmniEnvironment.
• Documented examples.

Setting Up a Parallel Build System

OmniThreadLibrary now supports 11 different Delphi versions (2007, 2009, 2010, XE, XE2, XE3, XE4, XE5, XE6, XE7, XE8), some with very special requirements about the supported pascal syntax (2007 and 2009 clearly standing out in that regard) so it takes quite some time to test the compilation of all demos and run unit tests on all supported editions. (And this time will only increase with the addition of support for mobile platforms and OS X. Not that I’m complaining. Sean is doing a terrific job there!)

It does not help that I don’t have all those Delphis installed on my computer. Most of them are only installed in a VM. And it takes a looooong time to start up a VM, run tests, power it down, start up next VM, and so on. And when I fix something, I have to retest it all ….

This kind of testing hurts. So in the manner of the Continuous Integration mantra, I decided to do it more often.

Double Bill: OmniThreadLibrary 3.04b & a Book Update

I have just updated OmniThreadLibrary to version 3.04b. This is kinda important update as I somehow messed the 3.04a release which did not include support for more than 60 concurrent tasks that was added in version 3.04. This important feature is now back and I’ve added few other small improvements.

Changelog for this release:

• Reimplemented support for TWaitFor in the TOmniTaskControl which was removed from the code in release 3.04a by mistake.
• Implemented TOmniSingleThreadUseChecker, a record which checks that the owner is only used from one thread.
• TOmniCommunicationEndpoint.ReceiveWait and .SendWait are now fully thread-safe and support multiple consumers (former) and multiple producers (latter).

Version 3.04b can be downloaded here. Alternatively, you can check out this release or update to the trunk.

I have also released a new version of “the book” which is now 239 pages long! Following sections were added/updated:

• Documented TOmniValue.
• Documented TWaitFor.
• Documented TOmniLockManager.
• Documented TOmniSingleThreadUseChecker.
• Adapted Communication Subsystem documentation.
  

OmniThreadLibrary Book Update

After a (much too) long time, I have (finally) found some time to work on my book “Parallel Programming with OmniThreadLibrary”. For starters, I have updated existing content so that the current version (3.04) is properly documented. The following topics were added/updated:

• Documented Parallel.For.
• Documented Parallel.Map.
• Documented TOmniBlockCollection.ToArray<T>.
• Added longer example for Parallel.For.
• Documented IOmniTaskConfig.SetPriority and .ThreadPool.
• Documented Parallel.Pipeline.PipelineStage[] property.
• Documented new demos.
• Documented other changes and additions.
• Updated hyperlinks.

Update is free (as it always was and as it will always be) for all owners of the book. You probably already received an email with details. If not, you should just go to LeanPub and download the new version.

You can buy the book (and with that support the OmniThreadLibrary development) here.

GpSQLBuilder Improvements

Thanks to [leledumbo], GpSQLBuilder now works with the FPC pascal and supports the INSERT statement.

You can now write statements such as:

  query := CreateGpSQLBuilder
    .Insert
      .Into(DB_TEST)
        .&Set(COL_1, [42])
        .&Set(COL_2, 'abc')
    .AsString;

Which will give you:

INSERT INTO Test (Column1, Column2) VALUES (42, 'abc')

Using OmniThreadLibrary’s Message Queue with a TThread Worker

A reader recently asked about using OmniThreadLibrary’s communication channels with a TThread-based worker thread so I prepared a simple example, now part of the OTL repository (stored in the folder examples\TThread communication).
Two separate topics are covered in this example:

• Sending data from any thread (main or background) to a TThread-based worker.
• Sending data from a TThread-based worker to a form.

OmniThreadLibrary – GetIt!

OmniThreadLibrary is now available on GetIt so it can be installed in XE8 with a single click!

Go, Embarcadero! That’s really great!

XE8 links

In case you’d like to know more about the RAD Studio/Delphi/C++ Builder XE8, following links will give you a good starting point.

Presenting XE8 in Slovenia

Next Thursday (14^th) I’ll be presenting XE8 in Ljubljana. We’ll be doing the standard “launch” combo – official (English) slides and demos + Slovenian presentation.

Register here!

Runtime SQL Query Builder v3

There’s a whole new SQL Query Builder available on GitHub. I have basically rebuilt it from scratch as the previous “architecture” (it does not actually deserve to be called that) was just slapping strings together and that approach had reached its limits.

Query builder is now implemented as three units – GpSQLBuilder.AST implements an abstract syntax tree for SQL queries, GpSQLBuilder.Serialize knows how to serialize this tree to a string and GpSQLBuilder creates the tree and calls the serializer when necessary. Now it will be much simpler to a) add support for new SQL keywords and b) create different serializers, for example a ‘pretty print’ version. (If anybody wants to tackle this last task, please go ahead – I’ll be happy to merge a pretty print serializer into the master branch.)

In addition to that I have added support for different Joins: .InnerJoin, .RightJoin, and .FullJoin are now supported in addition to .LeftJoin.

A .&Case method was redesigned a bit. It now returns an interface which can be sent as a parameter to a .Select, .Column, and .OrderBy methods. You can use it like this:

  query := CreateGpSQLBuilder
    .Select(
      CreateGpSQLBuilder.&Case
        .When([COL_2, '< 0']).&Then('0')
        .When([COL_2, '> 100']).&Then('2')
        .&Else('1')
      .&End)
    .From(DB_TEST);

and

  query := CreateGpSQLBuilder;
  query
    .Select.All
    .From(DB_TEST)
    .OrderBy(
      query.&Case
        .When([COL_2, '< 0']).&Then(COL_3)
        .&Else(COL_4)
      .&End);

New version also supports the .Distinct select flag.

I’ve created a unit test suite (actually I did that before the redesign so that I could refactor my code in piece) which now includes 51 tests, all passing.

Portable OmniThreadLibrary

If you care about running OmniThreadLibrary on mobile devices and on OS X, please join the OmniThreadLibrary-For-Mobile-DevelopmentProject page on Google+ where Sean has started to document his progress on doing exactly that – breaking the chains of Windows and bringing OmniThreadLibrary to all supported platforms!

Go, Sean!

Value Never Used – Or Is It?

Sometimes, dcc32 is just plain stupid …

I have this code in the OmniThreadLibrary:

function TOmniThreadPool.Cancel(taskID: int64): boolean;
var
  res: TOmniWaitableValue;
begin
  {$IF CompilerVersion >= 22}
  Result := false; // not really used
  {$IFEND}
  res := TOmniWaitableValue.Create;
  try
    otpWorkerTask.Invoke(@TOTPWorker.Cancel, [taskID, res]);
    res.WaitFor(INFINITE);
    Result := res.Value;
  finally FreeAndNil(res); end;
end;

The IFDEF part is there because otherwise compiler complains about result being potentially undefined.

[DCC Warning] OtlThreadPool.pas(1315): W1035 Return value of function 'TOmniThreadPool.Cancel' might be undefined

I’ve first observed this with the XE and it was fixed after the XE2 (can’t repeat this with XE8, but I didn’t check every compiler inbetween.)

New Home for OmniThreadLibrary

OmniThreadLibrary has been moved from Google Code to GitHub.

I still have to move existing issues as they were not moved automatically.

New Home for My Units

My collection of open-sourced units has been moved from Google Code to GitHub.

XE8 is out …

… and OmniThreadLibrary works fine with it. (That’s it. Just wanted to let you know.)

Runtime SQL Query Builder

I don’t want to use long SQL strings in the code.

Really.

Firstly, it is a pain to write long multiline strings in Object Pascal. (Embarcadero, are you listening? Can we please get multiline strings in Delphi? Please?)

Secondly, I’m very OCD when it comes to compiler watching my every step. I absolutely hate runtime errors and I want every problem to be detected during the compilation. That includes typos in database field names and changes attributed to the entropy (i.e. changes in the database structure).

On a tangent, that’s why we are creating and managing tables and databases in the code. We have all fields defined as constants, all SQL statements created in runtime and everything stored under version control.Upgrading a database to a version required by the program is therefore a very simple operation. (Downgrading is still a problem. Downgrading is hard.)

Thirdly, concatenating query strings together when you are dynamically building a query string based on the conditions in the program is a mess.

That’s why long years ago I created a simple runtime query builder. It was dormant (in the “good enough for me” state) for a long time but recently I required more functionality from it and I extended it further. It is now in a state which may be useful for somebody else but me so I’m releasing it as an open source under a BSD license.

For impatient: Get GpSQLBuilder on GitHub.

Unit Testing Recap & Downloads

The Unit Testing workshop was pretty much full.

It was good to see that many interested Delphi programmers in one room. Thanks for coming, everybody!

The downloads for the workshop are now online.

What’s New in Delphi XXXX

A great list of “What’s new” for Delphis from 3 to XE7 can be found on StackOverflow.

(just noticed and thought I’d better pass this on)

Unit Testing, Mocking, Inversion of Control

On March 19th (that’s next Thursday), I’ll be leading a workshop dedicated to unit testing Delphi programs. We’ll start with basics and and cover different tools (DUnit, DUnit2, DUnitX, TestInsight) and different programming methodologies.

Register here.

OmniThreadLibrary 3.04

OmniThreadLibrary 3.04 is now released. Get it here or check out the release-3.04 tag (or just follow the trunk, which is the best way to experience the OmniThreadLibrary).

Some other links you may need:

Home page: http://www.omnithreadlibrary.com/
Google+ community: https://plus.google.com/communities/112307748950248514961
Downloads: https://drive.google.com/folderview?id=0BwqVlLNTK4OxVEgzZnZGM1FNMGc&usp=sharing#list
Issue tracker: http://code.google.com/p/omnithreadlibrary/issues/list
SVN checkout instructions: http://code.google.com/p/omnithreadlibrary/source/checkout
Author's blog: http://thedelphigeek.com
Author's home page: http://primoz.gabrijelcic.org
Documentation wiki: http://otl.17slon.com/book/
Documentation book: http://leanpub.com/omnithreadlibrary

Changelog

New features

• Implemented simple and fast Parallel.&For which supports only integer ranges.
• Implemented Parallel.Map.
• Added overloaded Execute methods to IOmniParallelInitializedLoop and IOmniParallelInitializedLoop<T> so that IOmniTask parameter can be passed to the executor.
• Implemented Run overloads that internally call Invoke to start thread worker.
• Implemented TOmniValue.HasArrayItem.
• Implemented TWaitFor which supports waiting any/all and is not limited to 63 objects.
• IOmniTaskControl can wait on any number of comm handles and wait objects. That enables support for >60 tasks in the OtlThreadPool.
• Implemented TOmniBlockingCollection.ToArray<T>.
• NumTasks parameter (OtlParallel) can be negative. In that case, specified number of cores will be reserved for other purposes and all other will be used for processing.
• Example: If NumTasks(-2) is used when process has access to 8 cores, 6 of them (8 - 2) will be used to run the task. 
• Removed project groups.
• Tested with 2007-XE7.

Bug fixes

• Fixed race condition in IOmniPipeline termination code [tnx to Dean Hill].
• Creating parameters with the array overload (IOmniTaskControl.SetParameters(['From', 0, 'To', 99])) was not working.
• Fixed problems in demos 33_BlockingCollection and 47_TaskConfig.

New demos

• 55_ForEachProgress: Demonstrates progress bar updating from a ForEach loop.
• 56_RunInvoke: Simplified 'run & invoke' low-level API.
• 57_For: Simple and fast parallel for.
• 58_ForVsForEach: Speed comparison between Parallel.ForEach, Parallel.For, and TParallel.For (XE7+).
• 59_TWaitFor: Demo for the new TWaitFor class.
• 60_Map: Demonstrates the new Parallel.Map abstraction.
• 61_CollectionToArray: Demonstrates the new TOmniBlockingCollection.ToArray<T> method

OmniThreadLibrary 3.04 Release Candidate

RC for the new OmniThreadLibrary release is available here.

This version should work with Delphis from 2007 to XE7. Win32 and Win64 are supported for Console and VCL applications.

Converting Collection to an Array

Blocking collections (IOmniBlockingCollection) are basic elements for data storage and transfer in many high-level OmniThreadLibrary abstractions. They can, however, be somewhat clumsy when you want to read data from them, as there is no indexed access, just the basic “give me next element” enumeration.

To help a bit, SVN now includes three new class functions in the TOmniBlockingCollection class – ToArray<T>, ToArrayIntf<T>, and ToArrayRec<T>.

class function ToArray<T>(coll: IOmniBlockingCollection):
  TArray<T>;

class function ToArrayIntf<T: IInterface>(
  coll: IOmniBlockingCollection): TArray<T>;

class function ToArrayRec<T: record>(
  coll: IOmniBlockingCollection): TArray<T>;

Parallel Map

At my latest parallel programming presentation a participant suggested that I should extend the OmniThreadLibrary with a parallel mapping abstraction. Dear sir, here is a gift for you.

var
  numbers: TArray<integer>;
  odds   : TArray<string>;
begin
  //initialize the `numbers` array (not shown)
  odds := Parallel.Map<integer,string>(numbers,
    function (const source: integer; var dest: string): boolean
    begin
      Result := Odd(source);
      if Result then
        dest := IntTostr(source);
    end);
  //do something with the `odds` array (not shown)
end;

Implementing Record Assignment Operator [2]

Yesterday I hinted at having a working (and easy to use) solution which allows you to detect a record assignment (or copying, if you want) and to access both records (left and right part of the assignment operation). You can also modify record data when an assignment is detected. I also mentioned that my approach is very ugly, unstable, likely to cause problems in future Delphis and so on. You should read this article for the technical details, not to really use my code in an application. You have been warned!

Before I jump to hardcode Delphi stuff, I’d like to mention few different approaches that various smart people have already implemented. Most of them slipped below my radar in the past – and I’m guessing this may be the case for you too – so I’m listing them here in case you want to do more research on the topic.

Implementing Record Assignment Operator [1]

Following the yesterday’s hunch, I did some research and I came up with a way to detect when a record is copied (simple, supported, working everywhere) and even to access both records (source and target) during that operation (unsafe, unsupported and currently working only for Win32).

Now I can write such code:

Implementing Destructor for a Record

Smart records in Object Pascal are very nice, but they have a stupid limitation – you cannot implement a destructor for a record.

A solution for that is quite simple and can be found all over the internet – add an interface to this record and implement the cleanup in this interface. To make it even simpler, you can use the IGpAutoExecute interface from the completely free GpStuff unit.

Blast from the past: Turbo Pascal raytracer

Paweł’s recent post reminded me of a veeeeeery old raytracer I wrote in years 1985/89. (Yes, young ones, we had computers then!)

Header states:

1985-07-10:
  Computer: VAX-11/750
  Language: Pascal V2
  Graphics: VT100

1988-02-03:
  Computer: IBM-PC
  Language: Turbo Pascal 4.0
  Graphics: EGA 640 x 350 x 16

1989-07-03:
  Computer: IBM-PC/AT
  Language: Turbo Pascal 5.0
  Graphics: VGA 800 x 600 x 256

Quite a blast from the past – the 800 x 600 in 256 colors (out of a great palette of 262144 colors!) was more or less the greatest graphic card which you could get for PC at that time. And the first version was running on a VT100 terminal with an additional graphics board. Sadly, I don’t have that code anymore :(.

Attribute-based Command Line Parsing

Oh, command line parsing, the old enemy of mine!

In ye olden days, when I was learning Pascal, I did some programming on VAX/VMS systems, where you could (and should) leave the job of command line parsing to the DCL – the command line interpreter (something like CMD.EXE on Windows). You just wrote a definition file (see example on the right, found on the web) and DCL did the rest.

Since those days I hated that I have to parse command line programmatically. Sure, there were parsing libraries – and I wrote few of them myself – but I was never happy with them. There was always too much code to write.

Then I was looking for good examples on using attributes in Delphi and suddenly it occurred to me that I could use them for command line parsing, too! And that’s how GpCommandLineParser was born.

Before continuing, I should mention that I have based it (concept-wise, implementation is all my own) on the CommandParser unit, which comes with Delphi and can be found in Samples\Object Pascal\Database\dbExpress\Utils. This is a very nice but mostly unknown command line parser which I can definitely recommend.

OmniThreadLibrary 3.04 beta 1 – testers needed

I have (finally!) implemented support for more than 60 simultaneous tasks in the OTL thread pool and for more than 60 waitable events in the IOmniTaskControl. This also enables support for massively parallel execution in OtlParallel abstractions. IOW, Parallel.ForEach will run correctly on machines with 64 and more cores without needing to call NumTasks(60).

I’ve been testing this change and I think that it’s working correctly, but it is a big (and potentially dangerous) modification and so I’d like to see additional testers trying the new version in real programs before I submit it to the SVN.

So – if you like to live on the edge, if you’d like to help improve OTL and especially (but not required!) if you can test the code on a machine with 64 or more cores, please download the 3.04b1 from Dropbox and try it out. You can post reports here or send them to my email (my address is in the header of every OmniThreadLibrary file). In case of a problem, a short, repeatable, self-contained example will be of an immense help.

Thanks!

Parallel programming with XE7–reminder

A note for Slovenian readers.

Ne pozabite – jutri bom predstavil novo knjižnico System.Threading, ki olajša razvoj hitrih in odzivnih programov za vse platforme, ki jih podpira RAD Studio – Windows, OS X, iOS, Android – ter v obeh programskih jezikih – Delphi in C++. Ogledali si bomo tudi novosti v knjižnici OmniThreadLibrary.

Za več podatkov in prijavo kliknite tu!

Pohitrite programe z RAD Studiom XE7

This is an invitation for the XE7 workshop in Slovenia which is targeted at Slovenian programmers.

RAD Studio XE7 prinaša nov način paralelizacije programov. S kodo, zbrano v enoti System.Threading lahko na enostaven način pohitrite nekatere funkcije v programih, tako da se izvajajo sočasno z drugo kodo. Še lepše - System.Threading deluje na vseh podprtih platformah, torej poleg Windows tudi na OS X, iOS in Androidu. Na delavnici si bomo podrobno ogledali nove načine paralelizacije in pregledali druge knjižnice, ki podpirajo hkratno (paralelno) izvajanje v okolju Windows.

Predavanje bo v četrtek, 27. novembra, 2014.

Kliknite za prijavo

DAPUG 2014 Workshop

Presentations and code samples from DAPUG 2014 workshop are now available online.

Big thanks to organizers (and especially Jens) – we had great time in Denmark! Hope to see you again some time!

DAPUG workshop

Hesselet hotel is very nice and workshop is proceeding smoothly.

Just saying :)

Fixing XE7 + Android SDK 23 ZipAlign Problem

If you install RAD Studio XE7 and then upgrade Android SDK to version 23, you won’t be able to deploy applications to Android devices anymore.

The reason for that is that ZipAlign.exe was moved out of sdk\tools folder in SDK 23 and XE7’s deployment process cannot find it anymore. Now it is located in sdk\build-tools\20.0.0, but only is Android SDK Build-tools rev. 20 are installed, which doesn’t happen automatically.

To fix my Delphi I first had to install Build tools rev. 20. Then I copied sdk\build-tools\20.0.0\zipalign.exe to sdk\tools\zipalign.exe. That fixed the deployment problem.

I’m hoping Embarcadero will fix this in the next update, but for now you can use this workaround.

[Android sdk folder can be found in c:\Users\Public\Documents\Embarcadero\Studio\15.0\PlatformSDKs\adt-bundle-windows-x86-20131030\sdk]

Delphi programmer wanted

This is a job offer for a full time programming job in Ljubljana.

Iščemo Delphi programerja za dolgotrajno razmerje; človeka, ki se ne ustraši stare kode in z veseljem programira s sodobnimi orodji. [XE2, prehod na sodobno različico je že v teku in ga bo pomagal zaključiti prav programer ali programerka, ki ga iščemo.] Delo se opravlja v pisarni v Ljubljani.

Več o podjetju: www.fab-online.com. Kontakt: primoz (pri) fab-online.com.

Users Like FireUI and Dynamic Arrays

A snapshot from the XE7 presentation in Ljubljana.

Highlights of the presentation were FireUI, System.Threading, dynamic arrays, and OmniXML.

‘Twas a very nice crowd, asking LOTS of questions. It was also good to see some young(er) people and some Delphi [insert a prehistoric version here] users who finally decided that they should upgrade.

RAD Studio XE7 Launch in Slovenia

If you want to see all the great new stuff in XE7, join me on Wednesday, 17th in Ljubljana. We’ll cover the new VCL stuff, multithreading support, improvements for multi-device development, business mobile services and more.

As usual, presentation will be in Slovenian language.

More information & registration

Chromecast Randomly Crashing–Solved!

This is one of rare posts on my blog that doesn’t cover pascal programming. Proceed on your own risk ;)

For the last half a year, I’ve been a happy owner of a Chromecast device. Great stuff, especially as it allows me (with a small hack) to watch Netflix on my TV and listen to Pandora. At the beginning, it was a bit unstable, but through regular updates it stabilized quite a bit – until build 17977 got installed.

Since that time I got random crashes while playing any media (video or audio) from any source (in-house or external provider). It would load the player app, start playing, then crash after a few seconds. It would work fine for days, even weeks, but then it would start crashing and I would not be able to use for for half a day, after which it would start working again.

I’ve decided that the hardware is broken, maybe overheating, and I bought another one. (Did I mention it’s cheap?) But no luck – it started behaving the same as soon as it has upgraded to build 17977.

Few days ago I found a network forum thread where someone (sorry, no link, it got lost) mentioned a possibility of a bad power source. (My Chromecast is plugged into TV’s USB port.) And when the device started playing tricks on me today, I plugged it into an external USB power supply. And – surprise, surprise – it started working!

Something weird is still going on. When I start playing in a new player (i.e. when I switch apps), media is played for few seconds, then the TV goes blank and its power light starts blinking, then the picture reappears and media continues to play.

My current guess is that Chromecast does something weird which basically quick-resets the TV. When Chromecast was plugged into TV’s USB port, that also reset the Chromecast, but now that it’s plugged into an external power source, Chromecast continues playing while the TV does its yoyo trick.

I’m not completely satisfied with this solution as I would rather see my Chromecast to turn off together with the TV, but it is a “good enough” workaround. I still love that device, though. It’s the best stuff Google made in a long long time.

I’m posting this blog in hope that it will help other Chromecast users who may run into the same problem.