The Era of Dark Backgrounds

Today i had the idea to migrate old RMCOBOL flat files to SQL server in order to test my new datasnap app with real data. All i had to do was write some programs that read the files and write the data to sequential csv format. COBOL programs of course...

Apart of the fact that i had forgotten many syntax rules and statements of COBOL and needed to find the reference manuals, i also had a 20 years flash back. I just felt the need to write some words about at these days, the "era of dark backgrounds" development was a lot different than today. Building an application was almost a totally hand written task. No mouse, no click, no visual design, no online help, no Internet.


Few of the differencies were captured at my monitor during this flash back. Delphi XE2 IDE stands behind the COBOL "IDE".

The IDE, usually a plain text editor. Some of them had a lot of advanced features like mark blocks of text and move, copy, overwrite like simple copy&paste and if you were lucky a limited and occasional Undo. Syntax highlighting, code completion, syntax & statement help, refactoring, history and compare, code insight and other features of modern IDEs, like Delphi XE2 or even early versions of Delphi, was pure science fiction.

Project Manager, off course we had such a feature, who could not write at DOS command line the most simple command of all "c:\>dir /w *.cbl" ?
Who could not write a simple batch file to compile all or some of the project sources? ... dont ask me what is a batch file!

Compiling was also just a simple command like "c:\>rmc85 myprog". But when it comes to error messages, lets say you not only needed to have the -RTFM- books  in hand ...

I cannot show you any screen shots of debugging a program. When it comes to debugging we simply had ... almost nothing. Run the program and ... pray not to have forgotten a dot (.) inside and "IF" ! If you could not follow the program logic from the source files, there was nothing else to help you inspect the program flow.

"Dark backgrounds development was so hard and difficult?" you may wonder. I do not  have an answer, the only thing i know is that development was always about creation and of course fun!

From COBOL to DataSnap XE2

Many years ago I had built an automation server for my accounting platform using Delphi 6. It was based on TRemoteDatamodule and the Midas components TDatasetProvider & TClientDataset. It was a discovery for me especially speaking for TClientDataset and its unlimited features not only as a Midas client but as a standalone dataset fulfilled with capabilities that a blog post would not be enough to describe.

A couple of weeks ago, after I finished installing and familiarizing my self with the brand new Delphi XE2, I started studying its DataSnap technology. I have to say that I was somehow impressed by the bunch of features I had seen in demos, videos and webinars. Off course Midas components are still at the first line of building database client/server apps with DataSnap, but also a lot of other technologies have been added or heavily enhanced.

Some of them that I noticed are:

• Datasnap server with TCP/IP and HTTP(S) protocols
• Server classes with exposed methods to remote clients
• JSON object marshalling & un-marshalling
• Enhanced DBExpress framework (though I have never used it),
• DataSnap Connectors for Mobile Devices !
• DataSnap Callbacks
• Authorization & Authentication
• And many others that complete a long list to mention.

I was so confused with all these that I decided to give it a real world try by converting a old (from 1989) COBOL application to DataSnap. 
Did I said “Convert” ? Fully rebuild is the right choice of words. 

As database layer I was planning to stick with my well known SQLServer and ADODB in order to focus on Datasnap only. After database schema decided and built into SQL Server, I started the DataSnap server wizard of Delphi XE2. Choose “VCL Forms Application”, enabled all features except HTTPS and selected TDSServerModule as server methods class. 

• VCL Form selected in order to have a live server interface to do things like message logging, start/stop server, monitor its activity etc.
• All features selected in order to be able to test most DataSnap capabilities and be ready for mobile clients.
• TDSServerModule as server methods class provides a TDatamodule surface for VCL non-visual components and IAppServer interface which is essential for Dataset providers and clientdatasets communication (Midas).

The project the wizard  created was a lot different from what I was used using Automation with TypeLib and TRemoteDatamodule, but a few hours later I (think) had a clear understanding of the framework. The major aspects I acknowledge in simple terms are:

• VCLForm: the main form for monitoring server activity
• Container: a TDatamodule that is the main server module containing among other components a TDSServer, the class that manages transports from and to remote clients and server classes, a TDSServerClass which in turn used to specify a server-side class.
• ServerMethods: a TDSServerModule supporting IAppServer interface and contains the published methods that can be called from a remote client using dynamic method invocation.

How these works is pretty simple at its basics. 

Requests are served through the Container’s TDSServer component which coordinates traffic with the help of the other container components. 

As for the TDSServerClass it is the component responsible for creating its associated ServerMethods class. 

ServerMethods class exposes to remote clients its public methods and for the TDSServerModule especially the IAppServer interface for provider/clientdataset communication (Midas).

You can create as many classes as you want, either TComponent, TDataModule, TDSDataModule and expose their published methods, and IAppServer Interface in case of TDSDataModule, by just adding a TDSServerClass component in your server Container and respond to its “OnGetClass” event like this:

procedure TMyServerContainer.DSMyServerMethodsGetClass(DSServerClass
  : TDSServerClass; var PersistentClass: TPersistentClass);
begin
  PersistentClass := TMyClass; //Class of your TComponent, TDatamodule, TDSDataModule
end;

At client side now I had to create a VCL forms application and connect to the server to retrieve and present data for editing to the user. Three components do the base job:

• TSQLConnection: is the connection component that encapsulates a dbExpress connection to a database server. It can connect to various database server but in the case mentioned it connects to my DataSnap server using tcp/ip protocol.
• TDSProviderConnection: a component that provides connectivity to a DataSnap server methods class, the above mentioned ServerMethods (TDSServerModule), using dbExpress.
• TClientDataset: connected to the TDSProviderConnection via RemoteServer property and to the actual provider of the data in ServerMethods via ProviderName property.
  

I won’t describe further the details of setting the properties of those components as there are a lot of video presentations and articles at the official Embarcadero site & blogs. Also having a close look at the DataSnapXE projects in your XE2 samples folder is a must and I found them to be very much helpful.

Back to now, the business logic of my project is already implemented at the server and test clients have succeeded testing and coordinating them.


TClientDataset works as expected and a few extra communication needed between client and server implemented based on streaming TParams (TParamsStorage) via OwnerData. I use a lot this technique to pass info from one module to another and especially for passing user input from UI forms to datamodule methods that need to build runtime queries and return result sets for browsing, reporting etc. See my article series “A Journey to TParams” for more info.


Client Reporting requests are served from the server as entities of ClientDataset XmlPackets and processed at the client and the only things left are UI enhancements. Hope it soon will be completed so i can dig in more interesting aspects like Mobile connectors.

Datasnap & Transparent client lookup fields

Back in year 2000 I faced a problem trying to build my accounting platform to work in 3 tier client/server mode. In classic desktop applications I had separated database access logic and business rules from the user interface forms by using Delphi’s TDatamodule. A small but essential piece of this separation was lookup fields that once defined in the datamodule’s datasets should work transparently on UI forms with no further configuration.

But in the 3-tier model this could not be done, because TClientDataset in the client application receives the datapacket with lookup fields as data (fkData) and readonly. That seems logical as lookupfield info (LookupDataset property mainly) is unknown to the client. So the solution was to add clientdatasets for each server lookupdataset at the client and construct new lookup fields in the dataset that needed such referencies.
Problem! The level of abstraction of data I wanted for UI forms was broken, and even putting them in a client TDatamodule would be a break to the abstraction of data for the framework.

Did I mention “…could not be done”?

Shame of me, I am a Delphi developer.

All I needed was to scratch my head a bit more, and … Eureka! 

I spend some time reading the sources to figure out exactly what TDatasetProvider is doing behind the scenes and concluded that it would be possible to overcome the default behavior. The idea was to gather info about lookup fields, send them to the client where they can be processed and simulate lookup functionality. The process should be transparent to the client so I decided to embed it in a derived TClientDataset. On server side a procedure called from provider’s OnGetDatasetProperties should do the job of collecting and packaging needed info.

The result was “LookCDS”, a small library set of functions, classes and components for Delphi that enables and utilizes at client side the lookup fields defined in a Datasnap remote module. TlkClientDataset, a TClientDataset descendant, contained in LookCDS lib is the main component that supports transparent utilization of server side lookup fields. The library consists of two source files.

• DB_LooksClient.pas: It mainly contains TlkClientDataset, a derived TClientDataset to use at client side.
• UtilLookCDS.pas: Functions and definitions used by DB_LooksClient at client side and by TDatasetProvider at server side.

LookCDS enables transparent support for lookup fields defined in Datasnap server module to the client. It can be used this way:

On server side

• Define lookup fields as usual and just set their provider flags pfInUpdate & pfInWhere to false.
• Add TDatasetProvider component for any lookup dataset your fields refer to and you want to be enabled in the client.
• At the provider of the main dataset (that with lookupfields) add a call to “ProvideDatasetProperties” (in UtilLookCDS.pas) utility procedure.

On client side

• Use TlkClientDataset instead of the usual TClientDataset

That's all.

You can download the “LookCDS” files and a simple Datasnap XE2 demo here:  http://users.hol.gr/~georgev/delphi.htm

and at sourceforge project: http://sourceforge.net/projects/lookcds/