Monday, July 9, 2012

From TParams to TDatasetRecord and beyond

It has been a while since my last blog post. due to many job problems and the economical crisis here at Hellas (Greece). Anyway, for those who have followed my previous posts on "A journey to TParams", i will try to get TDatasetRecord (the class derived from TParams) a bit further, by explaining some aspects of creating derived classes with dataset field mappings.

TDatasetRecord is the class created to to represent a single dataset record. In fact it does not just represents a record, it mainly holds and provides access to record field values even disconnected from the actual dataset. As a descentant of TParams it uses TParam object to fullfil these requirements. Field values can be accessed by using normal TParam(s) methods and properties or TDatasetRecord.Field method, ie:

  aRecord.FindParam('LastName').AsString; 
   //or
  aRecord.ParamByName('LastName').AsString; 
   //or
  aRecord.Field('LastName').AsString;

This is useful, if you agree, but what is so big deal with? As is for now, nothing big, nothing fancy, nothing ... except, if we could have something that really goes into object relational mapping.

First of all, we should be able to access the field as a property of the class and do something like this:

  aContactRecord.LastName.AsString;

So lets get in detail with Delphi and do it. 
Of course we have to define a class derived from TDatasetRecord specifically for our contact dataset entity as sample and then make some cooking deriving just a base class method and defining properties.


The key method to derive is FieldNames. FieldNames class method is intended to return a name from a predefined list of field names for a specific class derived from TDatasetRecord. Base class's implementation just returns an empty string, but derived ones may change it by overriding this method.

FieldNames method can be overriden to return a string -the fieldname- defined in a const array of strings based on it's index. This method is used by internal functionality of TDatasetrecord to build it's fields -TParam- structure  by forcing which fields will be taken in account. This ensures that the record class will always have the same fields structure and at specified positions in the list of fields (actually TParams collection). For more detail you may refer to the implementation of TDatasetRecord.CreateFields method.

Next method of TDatasetRecord to take into consideration is GetParam, which provides access to internal TParam objects based on FieldNames method. Here is it's implementation:

function TDatasetRecord.GetParam(Index: integer): TParam;
begin
  Result := ParamByName(FieldNames(Index));
end;

Now that we have a way to define the internal fields structure and a field object access method by index we can go on and define properties with index specifiers. Index specifiers allow several properties to share the same access method while representing different values and thus GetParam method of TDatasetRecord can do this job for our contact class properties and map fieldnames to TParam objects.

Now let's built it.
Suppose we have to deal with a dataset of contacts with the following fields available:

  • ContactID
  • FirstName
  • LastName
  • DateOfBirth
  • Father
  • Mother
  • Phones
  • Address
  • SocialSecurity
  • CompanyName
  • BankAccount
So, here is the interface section:

  TContactRecord = class(TDatasetRecord)
  private
    function GetFullName: string;
  protected
    class function FieldNames(Index: integer): string; override;
  public
    procedure UpdateDataset;
    procedure AppendDataset;
    property ContactID                : TParam index  0 read GetParam; //ContactID
    property FirstName                : TParam index  1 read GetParam; //FirstName
    property LastName                 : TParam index  2 read GetParam; //LastName
    property CompanyName              : TParam index  3 read GetParam; //CompanyName
    property Phones                   : TParam index  4 read GetParam; //Phones
    property Address                  : TParam index  5 read GetParam; //Address
    property FullName: string read GetFullName;
  end;

And the implementation section:

class function TContactRecord.FieldNames(Index: integer): string;
const Names: array[0..5] of string = (
             'ContactID',                  //0 ContactID
             'FirstName',                  //1 ContactName
             'LastName',                   //2 LastName
             'CompanyName',                //3 CompanyName
             'Phones'.                     //4 CompanyName
             'Address'                     //5 CompanyName
                    );
begin
 if (Index < Low(Names)) or (Index > High(Names)) then Result := ''
 else Result := Names[Index];
end;
 
function TContactRecord.GetFullName: string;
begin
  Result := Lastname + ', ' + FirstName;
end;


Notes:

  • Field position is irrelevant as TDatasetRecord maps it's fields by name, not by position. 
  • We may not want to include all data fields in our class.
  • I also have added a new property that returns the fullname of the contact by concatenating FirstName and LasName. Just a hint of extending the class...

Monday, April 23, 2012

How to: on the fly create runtime data aware controls

There are many situations where you have a dataset with unknown fields structure or you just want to popup a small form and present to user some specific fields from the dataset , ie the fields need to be edited after an unsuccessful validation.

So, here is a simple function i have been using for long time ago to create data aware controls on panels, dialogs, scrollboxes:

uses StdCtrls, DB, DbCtrls, TypInfo;

var
  Label_W: integer = 200; {@ Width of the label controls}
  Control_W: integer = 500; {@ Max width of edit controls}
  Ctrl_Ident: integer = 2; {@ Distance between controls (horz & vert)}

function CreateDatasetEditor(
         COwner: TComponent; {@ The owner of control, it will be responsible for destruction}
         WParent: TWinControl; {@ The parent window where controls will live in}
         DSource: TDataSource; {@ TDataSource to be associated with controls}
         const Names, {@ Array of field names to use
               (optional, empty array will use all fields from TDataSource.Dataset}
               Labels: array of string; {@ Array of labels to use
               (optional, empty cells will use field.DisplayLabel }
         X: integer; Y: integer {X,Y coordinates in WParent to start positioning controls}
         ): TRect; {@ Result TRect used to place controls}

var i, j, iHigh: integer;
    c, ic : TControl;
    s: string;
    fld: TField;
    iL,iT: integer;
    Fields: TFields;
    Canvas: TControlCanvas;

 {@ Create a label control}
 procedure CreateDBLabel(ForField: TField; LabelText: string);
 begin
    with TLabel.Create(COwner) do begin
     Parent := WParent;
     AutoSize := False;
     Left := iL + Ctrl_Ident; Inc(iT,Ctrl_Ident); Top:=iT;
     Width := Label_W;
     WordWrap := False;
     if LabelText<>'' then
        Caption := LabelText
     else
        Caption := ForField.DisplayLabel;
     Alignment := taRightJustify;
     AutoSize := True;
     Transparent := True;
     end;
 end;

 {@ Create editing data aware control}
 function CreateEditField(ForFld: TField; sLabel: string): TControl;
 var w, h: integer;
 begin
  {@ Create edit control's associated label}
  CreateDBLabel(ForFld, sLabel);

  {@ Create actual data aware control based on filed info}
  if (ForFld.DataType in [ftBoolean]) then begin
      Result := TDBCheckBox.Create(nil);
      end
  else
  if (ForFld.DataType in [ftMemo, ftFmtMemo]) then begin
      Result := TDBMemo.Create(nil);
      Result.Width := Control_W;
      end
  else
  if (ForFld.FieldKind = fkLookup) then begin
      Result := TDBLookupComboBox.Create(nil);
      end
  else
      begin
      Result := TDBEdit.Create(nil);
      end;

  {@ Insert created control to COwner component hierarchy (for destruction puproses)}
  COwner.InsertComponent(Result);
  {@ Set control parent, width and other properties}
  Canvas.Control := Result;
  Result.Parent := WParent;
  Result.Enabled := not ForFld.ReadOnly;
  case ForFld.DataType of
    ftWord, ftSmallInt, ftInteger, ftAutoInc, ftLargeint: w := Canvas.TextWidth('###,###,###,###,###')+25;
    ftCurrency, ftFloat: w := Canvas.TextWidth('###,###,###,###,##0.00')+25;
    else
    w := ForFld.DisplayWidth * Canvas.TextWidth('W')+50;
    h := Canvas.TextWidth('Wq')+3;
    end;
  if not (ForFld.DataType in [ftMemo, ftFmtMemo]) then
     if w > Control_W then Result.Width := Control_W else Result.Width := w;
  {@ Connect control to DataSource & Field}
  TypInfo.SetOrdProp(Result,'DataSource',LongInt(DSource));
  TypInfo.SetPropValue(Result,'DataField',ForFld.FieldName);
  {@ Final adjustment of control width}
  if Result.Width > Control_W then Result.Width := Control_W;
 end;

 {@ Position a control in sequence}
 procedure PositControl(c: TControl);
 begin
  c.Left := iL + Ctrl_Ident*2 +Label_W; c.Top:=iT; Inc(iT,c.Height);
  Result.Bottom := iT;
  if Result.Right < c.BoundsRect.Right then
    Result.Right := c.BoundsRect.Right;
 end;

begin
 if not Assigned(DSource.DataSet) then Exit;
 Fields := DSource.DataSet.Fields;
 iL:=X;iT:=Y;
 Result.Left := X;
 Result.Top := Y;
 Canvas := TControlCanvas.Create;
 try
 iHigh := High(Labels);
 if Length(Names) > 0 then
    begin // Create controls from Names array
    j:=High(Names);
    for i:=0 to j do begin
      fld := Fields.FindField(Names[i]);
      if Assigned(Fld) then begin
        s:='';
        if (i<=iHigh) then s := Labels[i];
        c := CreateEditField(Fld,s);
        if Assigned(c) then
           PositControl(c);
        end;
      end;
    end
 else
    begin //Create controls from dataset.fields
    j:=Fields.Count-1;
    for i:=0 to j do
      begin
      s:='';
      if (i<=iHigh) then s := Labels[i];
      c := CreateEditField(Fields[i],s);
      if Assigned(c) then
         PositControl(c);
      end;
    end;
 finally Canvas.Free;
 end;
end;


Have fun developing, because development is fun!


Tuesday, April 10, 2012

How to: Clone TField and TDataset fields structure

This is just a quick tip on how to copy field structure between TDatasets. The interesting part is the "CloneField" function that duplicates the exact class of a TField from one dataset to another.

First the loop that iterates through a source dataset fields collection and clones each item (TField) to  another dataset, the destination. It takes as parameters a source TDataset from where fields structure will be read, a destination TDataset where fields will be created and a boolean that instructs the procedure to add the fields to the existing structure or exactly clone the source structure.

procedure CopyFields(SourceDataset, DestDataset: TDataset; doAdd: Boolean);
var i,p: integer;
    Fld: TField;
    dFld: string;
begin
  if not doAdd then DestDataset.Fields.Clear;
  for i:=0 to SourceDataset.Fields.Count-1 do
    begin
    if Assigned(DestDataset.Fields.FindField(SourceDataset.Fields[i].FieldName)) then
       Continue;
    Fld := CloneField(SourceDataset.Fields[i], DestDataset.Fields.Dataset);
    Fld.DataSet := DestDataset.Fields.Dataset;
    end;
end;

Notice the lines:

Fld := CloneField(SourceDataset.Fields[i], DestDataset.Fields.Dataset);
Fld.DataSet := DestDataset.Fields.Dataset;

The first is the call to "CloneFields" function that creates and returns a new TField object and the second that actually binds the field to the destination dataset. This is required in order to have a functional field in the dataset. Do not rely to the owner of the field that could be any TComponent, ie the form, which is the owner of persistent fields we create with the Delphi form designer.

Now, the function that creates an exact TField descendant class based on another one:

function CloneField(Source: TField; AOwner: TComponent): TField;

  procedure SetProp(Name: string);
  var V: variant;
      PropInfo: PPropInfo;
  begin
   PropInfo := TypInfo.GetPropInfo(Source, Name);
   if PropInfo <> nil then 
     try V := TypInfo.GetPropValue(Source,Name);
      if not VarIsNull(V) then 
         TypInfo.SetPropValue(Result,Name,V); 
     except
      ; //just kill exception
     end;
  end;

begin
  Result := TFieldClass(Source.ClassType).Create(AOwner);

  Result.Alignment              := Source.Alignment;
  Result.AutoGenerateValue      := Source.AutoGenerateValue;
  Result.CustomConstraint       := Source.CustomConstraint;
  Result.ConstraintErrorMessage := Source.ConstraintErrorMessage;
  Result.DefaultExpression      := Source.DefaultExpression;
  Result.DisplayLabel           := Source.DisplayLabel;
  Result.DisplayWidth           := Source.DisplayWidth;
  Result.FieldKind              := Source.FieldKind;
  Result.FieldName              := Source.FieldName;
  Result.ImportedConstraint     := Source.ImportedConstraint;
  Result.LookupDataSet          := Source.LookupDataSet;
  Result.LookupKeyFields        := Source.LookupKeyFields;
  Result.LookupResultField      := Source.LookupResultField;
  Result.KeyFields              := Source.KeyFields;
  Result.LookupCache            := Source.LookupCache;
  Result.ProviderFlags          := Source.ProviderFlags;
  Result.ReadOnly               := Source.ReadOnly;
  Result.Required               := Source.Required;
  Result.Visible                := Source.Visible;

  SetProp('EditMask');
  SetProp('FixedChar');
  SetProp('Size');
  SetProp('Transliterate');
  SetProp('DisplayFormat');
  SetProp('EditFormat');
  SetProp('Currency');
  SetProp('MaxValue');
  SetProp('MinValue');
  SetProp('Precision');
  SetProp('DisplayValues');
  SetProp('BlobType');
  SetProp('ObjectType');
  SetProp('IncludeObjectField');
  SetProp('ReferenceTableName');
  SetProp('Active');
  SetProp('Expression');
  SetProp('GroupingLevel');
  SetProp('IndexName');
end;

The first line of code is the one that creates a new TField descendant from the actual source field class.
Then is the block of base TField common properties assignement, followed by a block of property assignements using runtime  library information (TypInfo) for properties that MAY exist in the actual class. If some of the properties do not exist in the actual class, then they are simply ignored.

Some things to remember:
1.The "doAdd" parameter in "CopyFields" when True results to fields added to the destination fields structure, whilst False forces first to clear the destination fields collection resulting to an exactly same field structure to the destination dataset as the source one.
2.DestDataset has to be inactive in order to call either of the above functions.
3.In "CloneField", if used stand-alone,  "AOwner" represents the TComponent parameter that will be responsible for freeing the field. Usually you will pass the TDataset that the resulting field belongs to, so when the dataset closes it will also be freed.

Have fun developing, cause development is fun!

Saturday, March 31, 2012

TParams, a bit deeper with TDatasetRecord

Well, after two very basic and introductory posts about TParams (post 1, post 2), I think it is time to explain in more detail why I am so excited about. It is not about high-end programming, nor about beauty of coding, nor even about state of the art software development. It is just simplicity!

Now that some basic functionality of TParams has been showed off I can go further and deep into using it in data operations.

As I mentioned in other posts and especially in the first, a typical usage of TParams is to hold the values of the fields from a single database record. I also introduced some utility functions that help communication between TParams and the fields of a dataset.

In fact I rarely use such a procedural approach; I have long time ago crucified COBOL ;) .
Delphi is object oriented and as such we have the option to create classes and encapsulate such functionality. Delphi XE2 introduced a nice new feature; class helpers, for extending existing class functionality, but here we do not want to extend the actual TParams class. We will create a new branch with usages beyond what Borland originally designed for this piece of code. Using plain old inheritance we will create a new class, derived from TParams, which will support the two way communication with datasets.

The class should –in general- be able to:
  • Clone and hold data from a given dataset.
  • Process the data without touching the actual dataset ones.
  • Work in an offline scene with data.
  • Upload data to datasets.

And most importantly, these accomplishments should be almost encapsulated in class and sets of classes that can be extended and adapt specific processing needs by using OOP techniques. Such techniques can be to communicate data between classes, modules, programs and network, create hierarchical lists and associations and even relationships.

In order to accomplish these tasks the class should be able to:
  • Store and access data much the same -pretty- way a dataset access its fields.
  • The class should be able to build its fields based on a given TDataset existing fields structure.
  • The class should be able to transfer field values from a given dataset and back to the same or another dataset corresponding fields.

Some extended functionality could be to add a new record or update an existing one to a dataset, or even delete a record.

Well instead of writing a long blog post, I think it is better to deep into my code. So I have created a project at SourceForge.net where you can find the class and a demo project.

Here is the interface part of the class with descriptive remarks for each member:
{@ TDatasetRecord
   Class to represent a dataset record in a TParams collection with each
   TParam serving as a named field/value corresponding to a dataset field}
TDatasetRecord = Class( TParams )
private
  FDataset: TDataset; {@ Dataset associated with the recordclass}
  FUpdateDataset: TDataset; {@ Dataset to update from recordclass data}
  FRecordAvailable, {@ Record class has it's fields (TParams) created and bound them}
  FRecordExists,    {@ Record has beed loaded from a dataset }
  FIncludeAllFields: Boolean; {@ Create all fields from the dataset}
  function GetParam(Index: integer): TParam;
  function GetDataSet: TDataSet;
  procedure SetDataSet(Value: TDataSet);
  function GetUpdateDataSet: TDataSet;
  procedure SetUpdateDataSet(Value: TDataSet);
  function GetFieldValue(const FieldName: string): Variant;
  procedure SetFieldValue(const FieldName: string; const Value: Variant);
protected
  {@ Initializes recordclass and refreshes current recordfield values from Dataset}
  procedure Initialize(RDataset: TDataset; UDataset: TDataset=nil);
  {@ Create recordclass fields }
  procedure CreateFields; virtual;
  {@ Used by derived classes to set a predefined list of allowed
     recordclass fieldnames}
  class function FieldNames(Index: integer): string; virtual;
public
  {@ Construct recordclass connecting to a Dataset, and optional an update dataset}
  constructor Create(RDataset: TDataset; UDataset: TDataset=nil); reintroduce; virtual;
  destructor Destroy; override;
  function AddField(FldType: TFieldType; const FieldName: string): TParam;
  {@ Add recordfields from an object (TDataset, TFIelds, TFieldList, TFieldDefs) }
  function AddFields(Source: TObject): integer;
  {@ Assign, TParams collection override }
  procedure Assign(Source: TPersistent); override;
  {@ Get a new recordclass object cloning self properties, fields & data}
  function CloneRecord: TDatasetRecord;
  {@ Set/Unset recordfields Bound property, Bound=False clears all TParam values }
  procedure BoundRecord(DoBound: Boolean);
  {@ Get a list of recordfields defined in FieldNames string }
  procedure GetFieldList(List: TList; const FieldNames: string);
  {@ Get a FieldNames string stuffed with all recordfield names }
  function GetFieldNames: string;
  {@ Test Target Fields against recordfields for equal values }
  function EqualsTo(Target: TDatasetRecord; UseFields: string = ''): Boolean;
  {@ Set UpdateDataset field values from recordfields }
  function SetDatasetFields: integer; overload;
  {@ Set Target Dataset field values from recordfields }
  function SetDatasetFields(Target: TDataset): integer; overload;
  {@ Set recordfield values from corresponding From Dataset fields }
  procedure SetRecordFields(From: TDataSet; UnBound: Boolean); overload;
  {@ Set recordfield values from corresponding From recordclass fields }
  procedure SetRecordFields(From: TDatasetRecord; UnBound: Boolean); overload;
  {@ Retrieve recordfield values from current dataset record}
  procedure RefreshRecord(DoOpen: Boolean=True);
  {@ recordField access method by name }
  function Field(Value: string): TParam;
  {@ RecordClass has been filled with recordfield values }
  function IsAvailable: Boolean;
  {@ Associated dataset record existed when recordclass filled with values }
  function IsExisting: Boolean;
  property Dataset: TDataset read GetDataset;
  property UpdateDataset: TDataset read GetUpdateDataset write SetUpdateDataset;
  property IncludeAllFields: Boolean read FIncludeAllFields write FIncludeAllFields default True;
  property FieldValues[const FieldName: string]: Variant read GetFieldValue write SetFieldValue; default;
end;
You can download source code files and a simple Datasnap XE2 demo here:  http://users.hol.gr/~georgev/delphi.htm

and at sourceforge project: http://sourceforge.net/projects/datarecords/

Friday, February 24, 2012

The Era of Dark Backgrounds

Today i had the idea to migrate old RMCOBOL flat files to SQL server in order to test my new datasnap app with real data. All i had to do was write some programs that read the files and write the data to sequential csv format. COBOL programs of course...

Apart of the fact that i had forgotten many syntax rules and statements of COBOL and needed to find the reference manuals, i also had a 20 years flash back. I just felt the need to write some words about at these days, the "era of dark backgrounds" development was a lot different than today. Building an application was almost a totally hand written task. No mouse, no click, no visual design, no online help, no Internet.


Few of the differencies were captured at my monitor during this flash back. Delphi XE2 IDE stands behind the COBOL "IDE".

The IDE, usually a plain text editor. Some of them had a lot of advanced features like mark blocks of text and move, copy, overwrite like simple copy&paste and if you were lucky a limited and occasional Undo. Syntax highlighting, code completion, syntax & statement help, refactoring, history and compare, code insight and other features of modern IDEs, like Delphi XE2 or even early versions of Delphi, was pure science fiction.




Project Manager, off course we had such a feature, who could not write at DOS command line the most simple command of all "c:\>dir /w *.cbl" ?
Who could not write a simple batch file to compile all or some of the project sources? ... dont ask me what is a batch file!




Compiling was also just a simple command like "c:\>rmc85 myprog". But when it comes to error messages, lets say you not only needed to have the -RTFM- books  in hand ...



I cannot show you any screen shots of debugging a program. When it comes to debugging we simply had ... almost nothing. Run the program and ... pray not to have forgotten a dot (.) inside and "IF" ! If you could not follow the program logic from the source files, there was nothing else to help you inspect the program flow.

"Dark backgrounds development was so hard and difficult?" you may wonder. I do not  have an answer, the only thing i know is that development was always about creation and of course fun!

Thursday, February 23, 2012

From COBOL to DataSnap XE2

Many years ago I had built an automation server for my accounting platform using Delphi 6. It was based on TRemoteDatamodule and the Midas components TDatasetProvider & TClientDataset. It was a discovery for me especially speaking for TClientDataset and its unlimited features not only as a Midas client but as a standalone dataset fulfilled with capabilities that a blog post would not be enough to describe.

A couple of weeks ago, after I finished installing and familiarizing my self with the brand new Delphi XE2, I started studying its DataSnap technology. I have to say that I was somehow impressed by the bunch of features I had seen in demos, videos and webinars. Off course Midas components are still at the first line of building database client/server apps with DataSnap, but also a lot of other technologies have been added or heavily enhanced.

Some of them that I noticed are:
  • Datasnap server with TCP/IP and HTTP(S) protocols
  • Server classes with exposed methods to remote clients
  • JSON object marshalling & un-marshalling
  • Enhanced DBExpress framework (though I have never used it),
  • DataSnap Connectors for Mobile Devices !
  • DataSnap Callbacks
  • Authorization & Authentication
  • And many others that complete a long list to mention.
I was so confused with all these that I decided to give it a real world try by converting a old (from 1989) COBOL application to DataSnap. 
Did I said “Convert” ? Fully rebuild is the right choice of words. 

As database layer I was planning to stick with my well known SQLServer and ADODB in order to focus on Datasnap only. After database schema decided and built into SQL Server, I started the DataSnap server wizard of Delphi XE2. Choose “VCL Forms Application”, enabled all features except HTTPS and selected TDSServerModule as server methods class. 
  • VCL Form selected in order to have a live server interface to do things like message logging, start/stop server, monitor its activity etc.
  • All features selected in order to be able to test most DataSnap capabilities and be ready for mobile clients.
  • TDSServerModule as server methods class provides a TDatamodule surface for VCL non-visual components and IAppServer interface which is essential for Dataset providers and clientdatasets communication (Midas).
The project the wizard  created was a lot different from what I was used using Automation with TypeLib and TRemoteDatamodule, but a few hours later I (think) had a clear understanding of the framework. The major aspects I acknowledge in simple terms are:
  • VCLForm: the main form for monitoring server activity
  • Container: a TDatamodule that is the main server module containing among other components a TDSServer, the class that manages transports from and to remote clients and server classes, a TDSServerClass which in turn used to specify a server-side class.
  • ServerMethods: a TDSServerModule supporting IAppServer interface and contains the published methods that can be called from a remote client using dynamic method invocation.
How these works is pretty simple at its basics. 

Requests are served through the Container’s TDSServer component which coordinates traffic with the help of the other container components. 

As for the TDSServerClass it is the component responsible for creating its associated ServerMethods class. 

ServerMethods class exposes to remote clients its public methods and for the TDSServerModule especially the IAppServer interface for provider/clientdataset communication (Midas).

You can create as many classes as you want, either TComponent, TDataModule, TDSDataModule and expose their published methods, and IAppServer Interface in case of TDSDataModule, by just adding a TDSServerClass component in your server Container and respond to its “OnGetClass” event like this:

procedure TMyServerContainer.DSMyServerMethodsGetClass(DSServerClass
  : TDSServerClass; var PersistentClass: TPersistentClass);
begin
  PersistentClass := TMyClass; //Class of your TComponent, TDatamodule, TDSDataModule
end;

At client side now I had to create a VCL forms application and connect to the server to retrieve and present data for editing to the user. Three components do the base job:
  • TSQLConnection: is the connection component that encapsulates a dbExpress connection to a database server. It can connect to various database server but in the case mentioned it connects to my DataSnap server using tcp/ip protocol.
  • TDSProviderConnection: a component that provides connectivity to a DataSnap server methods class, the above mentioned ServerMethods (TDSServerModule), using dbExpress.
  • TClientDataset: connected to the TDSProviderConnection via RemoteServer property and to the actual provider of the data in ServerMethods via ProviderName property.
I won’t describe further the details of setting the properties of those components as there are a lot of video presentations and articles at the official Embarcadero site & blogs. Also having a close look at the DataSnapXE projects in your XE2 samples folder is a must and I found them to be very much helpful.

Back to now, the business logic of my project is already implemented at the server and test clients have succeeded testing and coordinating them.


TClientDataset works as expected and a few extra communication needed between client and server implemented based on streaming TParams (TParamsStorage) via OwnerData. I use a lot this technique to pass info from one module to another and especially for passing user input from UI forms to datamodule methods that need to build runtime queries and return result sets for browsing, reporting etc. See my article series “A Journey to TParams” for more info.


Client Reporting requests are served from the server as entities of ClientDataset XmlPackets and processed at the client and the only things left are UI enhancements. Hope it soon will be completed so i can dig in more interesting aspects like Mobile connectors.

Saturday, February 18, 2012

Datasnap & Transparent client lookup fields

Back in year 2000 I faced a problem trying to build my accounting platform to work in 3 tier client/server mode. In classic desktop applications I had separated database access logic and business rules from the user interface forms by using Delphi’s TDatamodule. A small but essential piece of this separation was lookup fields that once defined in the datamodule’s datasets should work transparently on UI forms with no further configuration.

But in the 3-tier model this could not be done, because TClientDataset in the client application receives the datapacket with lookup fields as data (fkData) and readonly. That seems logical as lookupfield info (LookupDataset property mainly) is unknown to the client. So the solution was to add clientdatasets for each server lookupdataset at the client and construct new lookup fields in the dataset that needed such referencies.
Problem! The level of abstraction of data I wanted for UI forms was broken, and even putting them in a client TDatamodule would be a break to the abstraction of data for the framework.

Did I mention “…could not be done”?
Shame of me, I am a Delphi developer.
All I needed was to scratch my head a bit more, and … Eureka! 

I spend some time reading the sources to figure out exactly what TDatasetProvider is doing behind the scenes and concluded that it would be possible to overcome the default behavior. The idea was to gather info about lookup fields, send them to the client where they can be processed and simulate lookup functionality. The process should be transparent to the client so I decided to embed it in a derived TClientDataset. On server side a procedure called from provider’s OnGetDatasetProperties should do the job of collecting and packaging needed info.

The result was “LookCDS”, a small library set of functions, classes and components for Delphi that enables and utilizes at client side the lookup fields defined in a Datasnap remote module. TlkClientDataset, a TClientDataset descendant, contained in LookCDS lib is the main component that supports transparent utilization of server side lookup fields. The library consists of two source files.
  • DB_LooksClient.pas: It mainly contains TlkClientDataset, a derived TClientDataset to use at client side.
  • UtilLookCDS.pas: Functions and definitions used by DB_LooksClient at client side and by TDatasetProvider at server side.
LookCDS enables transparent support for lookup fields defined in Datasnap server module to the client. It can be used this way:

On server side
  • Define lookup fields as usual and just set their provider flags pfInUpdate & pfInWhere to false.
  • Add TDatasetProvider component for any lookup dataset your fields refer to and you want to be enabled in the client.
  • At the provider of the main dataset (that with lookupfields) add a call to “ProvideDatasetProperties” (in UtilLookCDS.pas) utility procedure.
On client side
  • Use TlkClientDataset instead of the usual TClientDataset

That's all.



You can download the “LookCDS” files and a simple Datasnap XE2 demo here:  http://users.hol.gr/~georgev/delphi.htm

and at sourceforge project: http://sourceforge.net/projects/lookcds/

Saturday, January 21, 2012

A journey to Delphi TParams - Persistence

In the first article I demonstrated some features of TParams we can use at every day development to hold, process and move data structures around. But what if we want to persist such data and come back to them later?

As you will see, persisting is very easy and is based on Delphi’s component streaming mechanism. It is the same core mechanism used by IDE to save and load a form’s design in .dfm files. TStream class introduces methods that work in conjunction with components and filers for loading and saving components in simple and inherited forms. The two TStream methods needed are ReadComponent & WriteComponent and their definitions are:

function ReadComponent(Instance: TComponent): TComponent;
procedure WriteComponent(Instance: TComponent);
Delphi help also introduces two example functions to show how to use the built-in component streaming support to convert any component into a string and convert that string back into a component. These functions are a long time ago in my utilities library!

function ComponentToString(Component: TComponent): string;
var
  BinStream:TMemoryStream;
  StrStream: TStringStream;
  s: string;
begin
  BinStream := TMemoryStream.Create;
  try
    StrStream := TStringStream.Create(s);
    try
      BinStream.WriteComponent(Component);
      BinStream.Seek(0, soFromBeginning);
      ObjectBinaryToText(BinStream, StrStream);
      StrStream.Seek(0, soFromBeginning);
      Result:= StrStream.DataString;
    finally
      StrStream.Free;
    end;
  finally
    BinStream.Free
  end;
end;

function StringToComponent(Value: string): TComponent;
var
  StrStream:TStringStream;
  BinStream: TMemoryStream;
begin
  StrStream := TStringStream.Create(Value);
  try
    BinStream := TMemoryStream.Create;
    try
      ObjectTextToBinary(StrStream, BinStream);
      BinStream.Seek(0, soFromBeginning);
      Result := BinStream.ReadComponent(nil);
    finally
      BinStream.Free;
    end;
  finally
    StrStream.Free;
  end;
end;
Now that we have all the streaming functionality in our hands we can stream in & out a TParams collection, or we cannot?

No, we cannot. Component streaming works with TComponent & descendants and TParams is not one of these, it actually derives from TPersistent->TCollection.

But this is something easily fixed just by declaring a TComponent with a published TParams property. The streaming mechanism can then deal with this component and “magically” save and load the TParams collection.

Here is such a declaration, very simple and clean:

TParamsStorage = class(TComponent)
protected
 FParams: TParams;
published
 property Params: TParams read FParams write FParams;
end;
Now that we have the ability to write and read a TParams collection to and from a string, we can persist it anywhere we want, a local variable, a file, a stream, a database field etc. I personally have a function and procedure to automate the process of converting TParams to string and vice versa. Here they are:

function ParamsToString(Params: TParams): string;
var ps: TParamsStorage;
begin
  ps := TParamsStorage.Create(nil);
  try ps.Params := Params;
   Result := ComponentToString(ps);
  finally ps.Free;
  end;
end;
procedure StringToParams(Value: string; Params: TParams);
var ps: TParamsStorage;
begin
  ps := TParamsStorage.Create(nil);
  try ps.Params := Params;
   ps.Params.Clear;
   StringToComponent(Value,ps);
  finally ps.Free;
  end;
end;
Another interesting effect of having a TParams collection in a string is that we can store this string in a single TParam object, effectively creating a tree structure of TParam collections!You can investigate the whole idea in the code behind the recursive function I use to create/update such structures:

TParamProps = record
  Name: string;
  DataType: TFieldType;
  Value: variant;
end;
function CreateTParams(const aParams: array of TParamProps): TParams;
var i: integer;
begin
  Result := TParams.Create;
  for i:=0 to High(aParams) do
   Result.CreateParam(aParams[i].DataType,aParams[i].Name,ptUnknown).Value := aParams[i].Value;
end;

function UpdateParam(
Params: TParams; const //The root TParams collection
Path: array of string; //TParam names hierarchy path
             FldType: TFieldType; //DataType of TParam to create
Value: Variant //Value of TParam to create/update
): Boolean; //True always 
var WP: TParams;
    P: TParam;
    A: array of string;
    i: integer;
begin
  if Length(Path) = 1 then
     begin
     P := Params.FindParam(Path[0]);
     if not Assigned(P) then
        P := Params.CreateParam(FldType,Path[0],ptUnKnown);
     P.Value := Value;
     Result := True;
     end
  else
     begin
     P := Params.FindParam(Path[0]);
     if not Assigned(P) then
        P := Params.CreateParam(ftString,Path[0],ptUnKnown);
     WP := TParams.Create;
     try if P.AsString <> '' then
           StringToParams(P.AsString, WP);
      SetLength(A,Length(Path)-1);
      for i:=0 to Length(A)-1 do A[i] := Path[i+1];
      Result := UpdateParam(WP,A,FldType,Value);
      P.AsString := UtilDB.ParamsToString(WP);
      finally WP.Free;
      end;
     end;
end;
Have fun developing, because development is fun!
Feel free to modify the above code as per your needs and if you make any enhancements please contact me.

Monday, January 2, 2012

Data Centric Development Framework

An application framework consists of software used by developers to implement the standard structure and base functionality of an application for a specific development environment and/or application needs. Developers usually use object-oriented programming techniques to implement frameworks such that the unique parts of an application can simply inherit from pre-existing classes in the framework.

Through my experience on facing requirements while building data centric applications for clients, i found out that there are common used constructs and need for utilities. So started gathering the tools and libraries i had built around, created new modules, classes and components and modified them so they could work together. Finally something came out of this; a development framework that could support almost all the base functionality i needed and the consistency i wanted my products to have.

My framework is a software development framework, fully object oriented and data centric, for the Delphi development environment that targets the Win32/64 platform and SQL RDBMS. It is designed for rapid application development and focuses on efficiency, usability and consistent functionality of the final product.

User experience, with intensive all day work on data entry and query forms, is among the targets of this development framework, which provides a well balanced UI that helps user productivity. The framework provides forms and data modules with common functionality for almost every kind of data, utility dialogs for every day user work and common behavior of UI and controls.

Combines together many tools, functions, components and user enhancements in an environment that most of its functionality is almost out-of-the-box, helping the developer to rapidly create the base of consistent data centric applications and then focus to the specific requirements of the user i.e. business rules and further UI enhancements.

The main technical features consist of an MDI project template, base form/report classes and data module classes, all with integrated functionality that can be customized via inheritance, component events, virtual methods and class functions. It also contains application and user configuration modules and a lot of tools, dialogs & components that act at runtime in conjunction with all main classes of the framework.

I spent many hours (days, months…) working on this in parallel with other projects and I still make enhancement and adding new features, but finally seems like it worth the investment!
Know i am in the process of upgrading & enhancing it for the new Embarcadero Delphi XE2 and hope i will come back soon with a full technical description.

So stay tuned and have very good year 2012 full of happiness and joy.


Following is a small non-exhaustive descriptive list of the framework’s main features:

User Authentication

User authentication can be set by groups of users and authorities can be granted or denied for form or application defined procedure level.

User Authentication User Authentication User Authentication


Data features

Supports ADO enabled databases and especially SQL Server. Can install and update database via external scripts. Handles internally connection string and other connection parameters. Automatically marks with create/change user & timestamp info the database records that support this feature. Supports transactions, automatic or manual, on record or batch basis. Provides logical data schemas to forms and reports and handles communication of data between different active forms.


Application Desktop - MDI Environment

The application desktop is an MDI controller form class with user customizable main menu and sidebar.
Each user form works inside and controlled by the main application form. User can activate, deactivate, hide, minimize, arrange and switch between client forms inside the application desktop.

Application Desktop - MDI Environment Application Desktop - MDI Environment Application Desktop - MDI Environment

Common User Access

The framework adopts this IBM’s standard (CUA) which was a detailed specification and set strict rules about how applications should look and function. A major set of rules refers to the standardization of keyboard shortcuts (F keys and combinations) that user can use to access common features. Each keyboard shortcut has an application wide meaning so users are never confused when working with the different application forms. Such common keystrokes are F1 for help, F2 for field editing, F3 for search, F4 for lookup, F5 for refresh and so on. The CUA standard was also the basis for the Windows Consistent User Interface standard (CUI), as well as that for OS/2 applications. Most of the standard keystrokes and basic GUI widgets specified by the CUA remain a feature of Windows, but few users are aware of them.


Lookup & Select

Wherever a reference to a field is required user can activate a helper window that let him search and select a value for this field, i.e. when a customer must be selected for a purchase order the user is presented with a list of available customers in a fully customized data grid. Features of filtering grid data and quick locate column value are available also here as almost in all other data grids.

Lookup & Select Lookup & Select Lookup & Select

Data Grids

Data grids are extensively used in many places of the application framework. Data entry forms, query results, lookup&select are some of them. Data grids can be customized in many ways such as visibility of columns, column grouping and summaries. They support common actions like filtering, locate column value, print as is and can be exported to html, csv, xml or text.


Data Filtering

Filtering is a must have for every serious data centric application. When dealing with large amounts data filtering is an essential way to isolate and easily locate those of interest. The framework has extensive support for filtering with multiple different filters per dataset which can be application and/or user defined.

Data Filtering Data Filtering Data Filtering

Dynamic data panels

Dynamic data panels can present data from a dataset record via dynamically created and/or runtime designed controls. Design specifications of the data panels can be saved for runtime reusability. As long as data logic is separated from the data presentation, a single form class which features dynamic control creation is enough and gives consistent UI experience.

Dynamic data panels Dynamic data panels Dynamic data panels

Context Sensitive Popup Menu

Popup menu is a feature widely used in Windows and the framework utilizes this feature with context sensitive popup menu on its forms and user dialogs. It helps user to select from available actions on a form by just right clicking on it.


Form Menu

Each form has its own menu with all the available functions that are defined by the application. The menu supports floating and dockable bars and is fully customizable.


Edit record template

Base form class that supports editing of a single database record in a data panel which can be dynamic. Can also be used as generic record editing form.


Browse/Edit file template

Browse base form class that supports browsing database tables, queries and dataset results in a data grid. Features filtering and all data grid functionality except editing. Can also be used as generic dataset browsing form.
Edit base form class much like Browse file template that supports editing multiple database records in a data grid. Features data filtering, and all data grid standard functionality. Can also be used as generic dataset editing form.

Browse/Edit file template Browse/Edit file template Browse/Edit file template

Reporting template

Base report class that support consistent reporting features and functions along the application. User can preview reports, export report to various formats, change printer settings and define font sets for specific reports.


Query executor

Utility to build and execute queries against the data base. Results are presented in a data grid with capabilities like browse file and also can be used by Report Generator. Query definitions can be saved for later use and can also be parameterized.

Query executor Query executor Query executor

Report manager

Reporting utility that features runtime report designer, save and load reports, custom layouts and reports based on user defined queries or datasets derived by the application like the ones manipulated by a Browse/Edit file templates or produced by Query executor.

Report manager Report manager Report manager

Chart generator

A chart generating form class that can get input from any application dataset and produce on the fly charts (pies, bars, lines etc) on screen and printer.

Chart generator Chart generator 
Chart generator Chart generator
Chart generator Chart generator Chart generator