2008 May 29 - Thu
Evaluating Inter-Process Communication Frameworks
I'm reposting some comments regarding IPC frameworks that I made to the Boost-Users
listserve today. It is in response to someone making unsubstantiated remarks regarding the
relative merits of ACE and Boost, and another looking for some substatiated remarks. What
follows are some substantiated remarks, based upon my personal experience with it and
several other libraries.
I've started working on a number of distributed system projects. As a consequence, I
started looking for distributed system libraries. References to ACE were most pervasive. I
implemented a number of trial applications with the library. That was after plowing through
relevant sections in the three primary ACE reference books. That was a good learning
experience, if only to find out the various patterns in distributed architecture definition.
I had the inter-process/inter-server communications (which only sent simple stuff) working
well within ACE's Acceptor/Connector framework. ACE has a number of other patterns one can
use. I was really impressed with the fact that the examples I used from the books worked as
advertised, and I was able to bend them to my will.
ACE is based upon an interaction of classes, macros, and templates. One has to spend
some
time with the environment in order to become proficient with it. It has a large API. A
number of lower level API's upon which higher level API's are based. For example the
Acceptor/Connector uses constructs described earlier in the books.
Once I had my basic communications going, I realized I needed to get some form concrete
messaging infrastructure in place. I had an impression that TAO, which is a layer above
ACE, would be quite extravagant to implement, with it being an implementation of the CORBA
specification. I wanted something a little lighter (a whole lot lighter actually).
As I worked through that project, I started hearing about ASIO, indirectly through some
other libraries I was using. ASIO is now a member of Boost. I read a review somewhere that
ASIO is a 'modern' replacement for ACE. If you want to get into real template structures
and Boost oriented philosophy, I'd say that is a valid statement. I'd also say that ASIO is
'more to the point' and straight forward than is ACE, at least for the things I want to
accomplish. But like ACE, ASIO is the basic communications infrastructure, no real
messaging capability, which is what distributed computing is all about. ASIO turned out to
be a little harder to get my head wrapped around as it uses a number of advanced C++ and
Boost related idioms. For a run-of-the-mill C++ programmer, ACE would be better. For
someone steeped in the power and obscurity of advanced C++, and is looking to advance their
skill set, ASIO would be better.
I came across
RCF - Interprocess communication for C++, which is a
messaging framework riding atop of ASIO. Flexible, lightweight, and to the point. I worked
through the examples and things worked as advertised. It has the encryption,
publisher/subscriber, oneway/twoway idioms, and a few other nifty features.
At the same time I was doing that, seemingly coincidently, I learned a few more
interesting
facts. Going into this, I realized that I need a message dispatcher/proxy, some decent
failover techniques, and some additional event handling for non-IPC related activities.
Someone suggested ICE from www.zeroc.com for an RCF-like solution, but working to a
larger
scale. I've heard that the library's originator is someone who spent much time on CORBA
standards and redid the concept without the 'benefit' of committee involvement. I think the
library has all the bases covered in terms of lightweight message handling, dispatching,
resiliency, and higher level distributed processing philosophies. The drawback is that it
will have a steeper learning curve than would an implementation using RCF. I like RCF, but
I think I.m going to have to tilt towards ICE (itself, like RCF, developed and focused
towards C++ in a multiple license environment).
On the non-IPC front, Qt's QCoreApplication looks to be a good substrate on which to
build
event driven daemons.
In the end, I think my solutions are going to involve:
- ZeroC's ICE for primary inter-process
communications
- Qt QCoreApplication as a
base for daemon development (which has built-in stuff for
threads/locks, slots/signals)
- Wt, a C++ based web toolkit for
distributed GUI development
- Boost Libraries to fill in all the
holes
- a little legacy layer 3/4 ACE in one library I'm using, but with some work, I think I
can
convert the ACE stuff to ASIO
2008 May 24 - Sat
A Keyword Matching Algorithm
There are a number of well known algorithms out there for taking in a set of keywords and
matching them against test. Aho and Corasick comes to mind, as does the Wu Manber algorithm (the latter I've
implemented, and the code resides elsewhere on this site).
For another project, I didn't need something quite so fancy. Actually two projects come to mind. One is that I
have a input comma separated value file which includes stock symbols, a description, and the associated exchange. I
wanted to keep statitics on what is read in on an exchange basis. My first kick at the can on this was to implement a
string look up table using
2008 May 23 - Fri
RCF - Interprocess Communications for C++
For a couple of distributed computing projects, I've been trying to come up with a
feasible and easy to use method for making applications talk to each other, whether they be
on the same machine or across a network.
I started off doing some work with Douglas C. Schmidt's
ACE: The
ADAPTIVE
Communication
Environment. I plowed through ACE's three primary programming books to see what
would be
the best bit of the environment I would need. I ended up implementing a demo with the
Acceptor - Connector framework, just to see how things worked.
I then started on thinking on the messaging structure and the event handling structures.
ACE's mixture of macros and classes turned out to be a little overwhelming for what I wanted
to accomplish.
During my stint with ACE, I started to use ASIO, from the
Boost libraries. I was first introduced
to ASIO through working with
WT: WebToolKit. I used Wt as a
frontend to a voip call sign in server.
The next step in the evolution is to present a real time call summary report to
authorized management as the calls are authenticated, authorized, and accounted for from a
Radius server. This means sending call detail messages from the Radius server to a central
dispatch server, and then publish to active web clients (with the clients written with
Wt).
As Wt uses ASIO for its underlying network communications, and I had read a remark
somewhere that ASIO is the new improved ACE, I started to look into it as the mechanism for
my inter-process communications. I even got a good chunk of messaging infrastructure
written as was about to get it testing when I found it was all for nought.
I came across
RCF - Interprocess Communications for C++. It is a library that has been in development
for the last few years by a talented fellow by the name of Jarl Lindrud. The library has
implemented all the stuff that I only dreamed about doing: publish/subscribing, stream
encryption, payload filtering, and any number of other nifty features.
I had a few painful moments in getting the library built. After a couple of messages
back and forth to the author, I realized I was trying to build the whole thing into a static
library rather than using an 'include' technique to get the platform specific files built.
The client and server examples built and ran without a hitch. I must admit that I was
impressed by the examples in the ACE books as well: they compiled and ran with little or no
messing about.
The RCF library is better because it deals with serializing native values back and forth,
something that ACE only accomplishes when you get into the TAO and CORBA levels of the
environment.
So now with Boost (which includes ASIO), RCF (which uses ASIO), and Wt (which also uses
ASIO), I think I have all the interprocess tools I need to make my modules talk to each
other. Now I can get on with the meat of my projects.
2008 Mar 26 - Wed
C++ Custom Containers and Iterators
I'm using the
HDF5 File System for holding
time series information. Rather than writing my own binary search implementation to find
particular elements within a particular saved time series, I thought it would be
clever if I designed the
interface so I could use the Standard Template Library's 'find' iterator. If I can make the
STL's 'find' work, then all the other iterators should work just as well, and thus I'll have
an
easy mechanism to access time series with very little programming involved.
I can find any number of web sites containing information on how to work with C++'s
standard containers and iterators. When it comes to finding information on custom
containers and iterators, the information is not quite so plentiful.
The first article I came across was one from TechRepublic called
Extending the C++ STL with custom containers. It didn't quite have the meat I was
expecting.
Bjarne Stroustrup's book, The C++ Programming Language, does have a section on
iterators and a section on containers. In retrospect, they are quite good introductions
to the concept, but I didn't feel the examples were as informative as I would have liked.
Microsoft's MSDN has an article called
C++ and STL: Take Advantage of STL Algorithms by Implementing a Custom Iterator, but
this article only covers the custom iterator side of things, it doesn't discuss how it would
interact with a custom container.
Dr. Dobbs inherited an article entitled
Custom Containers & Iterators for STL-Friendly Code:
A pair of approaches for creating custom containers from the March 2005 issue of C++
Users Journal. Some code extracts are included but there are some pieces missing, such as
the begin() and end() methods and how they are put together. The link in the article to the
original code no longer works. However, I did find that I have the Dr. Dobbs Developer
Library DVD Release 4. On it resides the full example code. That was much more
informative.
Now that I have a better understanding for what I'm looking, I see that the
STL compliant container example has some useful information. In the same vein,
CodeProject has another example:
An STL compliant sorted vector.
Finally, I came across Ulrich Breymann's book called Desiging Components with the C++
STL. It provided all the necessary background to pull it all together. I always thought
there was more to it, but custom containers and iterators may not be so hard after all.
Once I have the code finished, I'll try to have it posted one way or another.
2008 Mar 16 - Sun
Don't Use Defined Macros, Use Templated Inline Functions Instead
In the olden C days, one would use #define MACRO .... to build an inline macro
for computationally quick evaluation of some calculation. When using that method
of programming, one needed to remember to parenthesize extensively in order to prevent
wierd things from happening when calling the function with an expression.
The modern approach is to use a template for an inline function, which yields
all the efficiency of a macro, plus all the predictable behavior and type safety
of a regular function (item #2 in the book Effective C++. An example declaration follows:
template<typename T>
inline void DoWithMax( const T& a, const T& b ) {
f( a > b ? a : b );
}
2008 Jan 18 - Fri
C++ Tools: Face Detection, Artificial Neural Networks
Today I came across a couple of C++ projects relating to Neural Network usage. One is located
at CodeProject called Face Detection C++ Library with Skin and Motion Analysis. The author has used
a number of interesting statistical and analytical methods for face detection. To quote his
mouth-full: "An understanding of wavelet-analysis, dimensionality reduction methods (PCA, LDA,
ICA), artificial neural networks (ANN), support vector machines (SVM), SSE programming, image
processing, morphological operations (erosion, dilation), histogram equalization, image
filtering, C++ programming and some face detection background would be desirable.".
From a C++ library perspective, here is a
Fast Artificial Neural Network Library
(FANN). I believe the library is written in C, but has API bindings in about 13 different
programming language formats, of course with one being C++. The author's description goes as
follows: "Fast Artificial Neural Network Library is a free open source neural network library,
which implements multilayer artificial neural networks in C with support for both fully connected
and sparsely connected networks. Cross-platform execution in both fixed and floating point are
supported. It includes a framework for easy handling of training data sets. It is easy to use,
versatile, well documented, and fast."
2007 Oct 19 - Fri
C++ Libraries from Scaling Web
Scaling Web has a number of open source libraries that may be of use:
- Template based B+ Tree: Implemented in C++, B+ Tree is template based, so it can be
used with any types of data. Memory based B+ Tree is much faster than STL containers like
map, hash_map.
- Embedded File System: Embedded File System (EFS) is a library that allows you to
create virtual file system in a single file.
- Mork Parser in C++: MorkParser is an efficient C++ implementation of Mork database
format reader. The Mork format is used in most Mozilla-based projects, including the Mozilla
browser suite, SeaMonkey, Firefox and Thunderbird.
- Mime Email Parser in C++: C++ implementation of RFC 2045 - Multipurpose Internet Mail
Extensions (MIME) standard.
- C++ IO Streams Library: IO Streams library provides different Java-like interface
streams in C++.
2007 Oct 06 - Sat
Web 2.0 Site Development with Wt -- Dead Easy
I've written an earlier article about
installing the Wt C++ Web Toolkit. I think it took longer to get all the bits and
pieces properly arranged than it did me to cook up an example to check that it did what I
thought it could do.
I'm quote pleased and impressed with the tool. I was able to get a basic web application
up and running in about two days, which were spent reading the documentation, reviewing the
class libraries, looking at some of the samples, poking through the PostgreSQL C library,
and then hacking stuff into the basic Wt 'Hello World' application.
The authors of Wt have done a great job of hiding all the Javascript go-between code. I can focus
on the higher level parts of my design, rather than the underlying machinery to make an
interface work. About the only thing I've encountered that I'd like to change is to take
the inline Javascript code, and get it put into a .js file.
The test I came up with was to open up a database, select a bunch of records, and present
the records in a table which could be stepped through (forwards and backwards) 10 records at
a time... and all this without having to refresh the whole page, just the table itself. The
following code proves it can be done.
It allows one to turn an internet browser into a true application front end with most of
the usual gui functions provided.
This first file is ouipam.cpp, the file with the 'main' start for each session
connection:
//============================================================================
// Name : ouipam.cpp
// Author : Ray Burkholder
// Version :
// Copyright : (c) 2007 One Unified
// Description : OUIPAM: One Unified IP Address Management
//============================================================================
// server push uses WApplication enableUpdates(), triggerUpdate(),
getUpdateLock();
#include "Responder.h"
using namespace Wt;
WApplication *createApplication(const WEnvironment& env) {
// Instantiate the Wt application.
Responder *appl = new Responder(env);
return appl;
}
int main(int argc, char **argv) {
return WRun(argc, argv, &createApplication);
}
This file is Responder.h, the header file for the responder, which handles the WApplication (or session):
//============================================================================
// Name : ouipam.cpp
// Author : Ray Burkholder
// Version :
// Copyright : (c) 2007 One Unified
// Description : OUIPAM: One Unified IP Address Management
//============================================================================
#ifndef RESPONDER_H_
#define RESPONDER_H_
#include <WApplication>
#include <WContainerWidget>
#include <WText>
#include <WPushButton>
#include <WTable>
#include <WTableCell>
#include "libpq-fe.h"
using namespace Wt;
class Responder : public WApplication {
public:
Responder(const WEnvironment& env);
virtual ~Responder();
protected:
WPushButton *button;
WTable *table;
private:
int cnt;
PGconn *conn1;
PGresult *result;
int nRowsToShow;
int cntRowsFound;
int cntColumnsFound;
int ixFirstRowShowing; // 0 offset
void OnButtonListBegin();
void OnButtonForeward();
void OnButtonBackward();
void OnButtonListEnd();
void ShowRows();
};
#endif /*RESPONDER_H_*/
This final file is Responder.cpp, where all the session work is performed. The constructor opens the database, creates the
webpage basics, and assigns events to each of the forward and backward buttons. When a button is pressed, the appropriate
method is called to use the appropriate records and update the table. Again, with Javascript enabled, the web page is not
refreshed. Instead, the browser's DOM is updated directly, and only the table cells change. I look foreward to seeing what
else I can do with this flexibility. As you can see from the header comment, this is the start of an interactive web
application for managing an organizations IP Addresses.
//============================================================================
// Name : ouipam.cpp
// Author : Ray Burkholder
// Version :
// Copyright : (c) 2007 One Unified
// Description : OUIPAM: One Unified IP Address Management
//============================================================================
#include "Responder.h"
#include "WBreak"
#include <sstream>
#include <ostream>
#include <algorithm>
using namespace std;
Responder::Responder(const WEnvironment& env) :
WApplication(env ) {
// Set application title
setTitle("OUIPAM by One Unified");
ostringstream ss;
// perform database query
conn1
= PQconnectdb("hostaddr=127.0.0.1 port=5432 dbname=oneunified user=oneunified
password=xxx");
ConnStatusType stat = PQstatus(conn1 );
if (CONNECTION_OK != stat) {
//PQfinish(conn1);
ss << "pq result= bad("<< stat << ")"<< endl;
root()->addWidget(new WText(ss.str()));
} else {
result = PQexec(conn1, "select * from ianaiftype;");
ExecStatusType statusExec = PQresultStatus(result );
ss << "pgresult=";
bool bTuplesFound = false;
switch (statusExec ) {
case PGRES_EMPTY_QUERY:
ss << "empty query";
break;
case PGRES_COMMAND_OK:
ss << "command ok";
break;
case PGRES_TUPLES_OK:
ss << "tuples found";
bTuplesFound = true;
break;
case PGRES_COPY_OUT:
ss << "copy out";
break;
case PGRES_COPY_IN:
ss << "copy in";
break;
case PGRES_BAD_RESPONSE:
ss << "bad response";
break;
case PGRES_NONFATAL_ERROR:
ss << "non fatal error";
break;
case PGRES_FATAL_ERROR:
ss << "fatal error";
break;
}
root()->addWidget(new WText(ss.str()));
root()->addWidget(new WBreak());
// present some query statistics
if (bTuplesFound ) {
cntRowsFound = PQntuples(result );
cntColumnsFound = PQnfields(result );
ss.str("");
ss << "rows="<< cntRowsFound << ", columns="<<
cntColumnsFound << endl;
root()->addWidget(new WText(ss.str()));
root()->addWidget(new WBreak());
for (int i = 0; i < cntColumnsFound; i++) {
ss.str("");
ss << PQfname(result, i )<< ", format="<< PQfformat(result, i )
<< ", type="<< PQftype(result, i )<< ", size="
<< PQfsize(result, i )<< endl;
root()->addWidget(new WText(ss.str()));
root()->addWidget(new WBreak());
}
// create table for row results
table = new WTable();
root()->addWidget(table);
// assign methods to the buttons
button = new WPushButton( L"<<" );
root()->addWidget(button );
button->clicked.connect(SLOT(this, Responder::OnButtonListBegin));
button = new WPushButton( L"<" );
root()->addWidget(button );
button->clicked.connect(SLOT(this, Responder::OnButtonBackward));
button = new WPushButton( L">" );
root()->addWidget(button );
button->clicked.connect(SLOT(this, Responder::OnButtonForeward));
button = new WPushButton( L">>" );
root()->addWidget(button );
button->clicked.connect(SLOT(this, Responder::OnButtonListEnd));
// show the query results
nRowsToShow = 10;
ixFirstRowShowing = 0;
ShowRows();
}
}
}
Responder::~Responder() {
PQclear(result ); // result exists even with new command, and even if connection is closed;
PQfinish(conn1 );
}
void Responder::ShowRows() {
// update rows in the already created table
WTableCell *cell;
WText *text;
int ixRowToShow = ixFirstRowShowing;
for (int ixTableRow = 0; ixTableRow < nRowsToShow; ixTableRow++) {
for (int ixColumn = 0; ixColumn < cntColumnsFound; ixColumn++) {
cell = table->elementAt(ixTableRow, ixColumn );
cell->clear();
if (ixRowToShow < cntRowsFound ) {
text = new WText();
text->setFormatting(WText::PlainFormatting );
text->setText(PQgetvalue(result, ixRowToShow, ixColumn ) );
cell->addWidget(text );
} else {
}
}
ixRowToShow++;
}
}
void Responder::OnButtonListBegin() {
ixFirstRowShowing = 0;
ShowRows();
}
void Responder::OnButtonBackward() {
ixFirstRowShowing = max( 0, ixFirstRowShowing - nRowsToShow );
ShowRows();
}
void Responder::OnButtonForeward() {
int i = max( 0, cntRowsFound - nRowsToShow );
ixFirstRowShowing = min(i, ixFirstRowShowing + nRowsToShow );
ShowRows();
}
void Responder::OnButtonListEnd() {
ixFirstRowShowing = max( 0, cntRowsFound - nRowsToShow );
ShowRows();
}
2007 Oct 03 - Wed
GSL - GNU Scientific Library
I'm coming across all sorts of interesting things today. Another C++ library I've
encountered is the GSL - GNU
Scientific Library. From the web-site, it is:
a numerical library for C and C++ programmers. It is free software under the GNU General
Public License.
The library provides a wide range of mathematical routines such as random number
generators,
special functions and least-squares fitting. There are over 1000 functions in total with an
extensive test suite.
Some of the subject areas covered include:
| Complex Numbers |
Roots of Polynomials |
Special Functions |
| Vectors and Matrices |
Permutations |
Sorting |
| BLAS Support |
Linear Algebra |
Eigensystems |
| Fast Fourier Transforms |
Quadrature |
Random Numbers |
| Quasi-Random Sequences |
Random Distributions |
Statistics |
| Histograms |
N-Tuples |
Monte Carlo Integration |
| Simulated Annealing |
Differential Equations |
Interpolation |
| Numerical Differentiation |
Chebyshev Approximation |
Series Acceleration |
| Discrete Hankel Transforms |
Root-Finding |
Minimization |
| Least-Squares Fitting |
Physical Constants |
IEEE Floating-Point |
| Discrete Wavelet Transforms |
|
|
Cryptographic Library for C++
ergo offers up some explanation
and examples of using the CryptoPP - The
cryptographic library for C++. He also refers to something else that may be of
interest, a SSL++: C++
Headers for OpenSSL. But I think most just compile and link directly with the headers
from OpenSSL.
|