Getting Started: Part 2

In the last edition of the blog we walked through the first steps of getting OpenMAMA up and running - downloading the RPM, installing onto a system, and getting data flowing between a basic publisher and subscriber. This next edition focuses on the primary usecase for which OpenMAMA was developed - publishing and subscribing financial market data.

Market Data

In order to do so, we're going to again start with our publisher application. In a typical environment this would most likely be something like a feedhandler (a process which consumes data from an exchange, normalizes it, and republishes it for other clients to consume) or other data distribution applications. However in order to keep things simple we'll be working with a small test application included in the OpenMAMA release, capturereplayc, and the sample market data which comes with the RPM. If you're here directly from the previous blog post in the series, feel free to reuse the terminal sessions you already have open - simply kill the mamapublisher and mamasubscriber applications by selecting the terminal and pressing ctrl-c. If you haven't completed the steps outlined in the previous post, I recommend you go back to it now, as it provides a lot of useful information regarding how to setup an OpenMAMA environment. Don't worry, we'll wait...

capturereplayc

capturereplayc is a simple market data publisher, which reads data from OpenMAMA playback formatted files, and publishes that data across a transport. It also handles dictionary requests, and can be used to publish book data to be read by OpenMAMDA applications. The command line for capturereplayc is fairly straight forward, with the following help text:

[damian@openmama openmama]$ capturereplayc -h
capturereplay - Sample application demonstrates how to publish mama messages, and respond to requests from a client inbox.
It accepts the following command line arguments:
[-S source] The source name to use for publisher. Default: WOMBAT.
[-i interval] The interval between messages. Default: 0.5.
[-f filename] The capture filename.
[-m middleware] The middleware to use for publisher [wmw/lbm/qpid]. Default: wmw.
[-tport name] The transport parameters to be used from mama.properties. Default: pub.
[-dictionary] The dictionary file which is sent in response to client requests. Required option.
[-q] Quiet mode. Suppress output.

For the purposes of this example, we will pass in a source (-S TEST), a middleware (-m qpid), a transport (-tport pub), and the location of both the dictionary file and the OpenMAMA playback data itself (-dictionary data/data.dict -f data/openmama_utpcasheuro_capture.5000.10.qpid.mplay).

The majority of these should be familiar from my previous post, and the playback data file will hopefully be self explanatory. The dictionary is an interesting concept in OpenMAMA, and essentially provides vocabulary used for building and decoding messages. Each item in the vocabulary contains three pieces of information - the field identifier (FID), the field name, and the field data type. We usually don't need to worry about how these dictionaries work in OpenMAMA, simply that our application has access to one. For our publisher we read it from a file, though typical subscribers will usually attempt to retrieve one (over OpenMAMA) from a known dictionary source on the platform.

When we start executing the application, you should see a series of "....." characters displayed. This is a pre-processing step conducted by the application reading through the playback data. When the dots have finished advancing, the application should be ready to accept subscriptions. For a more detailed view of what the application is doing during this stage, you may pass multiple '-v' arguments, which will increase the verbosity of the application logging. The application command line should now be:

capturereplayc -S TEST -m qpid -tport pub -dictionary data/data.dict -f data/openmama_utpcasheuro_capture.5000.10.qpid.mplay

The output from the command should look as follows:

[damian@openmama openmama]$ capturereplayc -S TEST -m qpid -tport pub -dictionary data/data.dict -f data/openmama_utpcasheuro_capture.5000.10.qpid.mplay
2013-12-19 10:41:34: ******************************************************************
Note: This build of the MAMA API is not enforcing entitlement checks. Please see the Licensing file for details
********************************************************************
CONNECTION ERROR connection aborted
...........

Don't worry about the CONNECTION ERROR warning for now - this is something produced by the underlying Qpid Proton libraries when they're searching for an end point to connect to. Because we don't have a subscriber yet, Proton complains a little. ;-)

mamalistenc

Now that we have our publisher ready and active, we can set up our client applications. For most users of OpenMAMA, the first and most common client application utilized is mamalistenc. This is a very full featured client app, with a wide variety of command line options and functionality available. For this example however, we're going to use a fairly simple set of these features. The requisite help text is as follows:

[damian@openmama openmama]$ mamalistenc --help
Example Mama API market data subscriber.
Usage: mamalistenc -S symbolNameSpace [-s symbol | -f symbolFile] [-tport transportName] [-dict_tport transportName] [-d dictSymbolNameSpace] [-vDgE?IL1qWB] [-DF fileName] [-hw highWaterMark] [-lw lowWaterMark] [-threads numThreads] [-r throttleRate] [-t subscriptionTimeout] [-version (requires -m)] [-m middleware]
Options:
...
-m The middleware to use [wmw/lbm/tibrv]. Default is wmw.
...
-S, -source, -SN The symbol name space for the data.
-s An individual symbol to subscribe to. Can be passed multiple times.
...
-tport The name of the transport being used. Corresponds to entries in the mama.properties file.
-v Increase the level of logging verbosity. Pass up to 4 times.
...
-?, --help Print this usage information and exit.
Examples:
1. Subscribe to the MSFT BBO from the NASDAQ UTP feed. Feed running with a symbol namespace of "NASDAQ". Middleware is WombatTCP: mamalistenc -S NASDAQ -s MSFT -tport tport_name -m wmw
...

From the above it should be fairly clear what command line options we're going to specify: a middleware (-m qpid), a transport (-tport sub), a source which matches that of the publisher (-S TEST) and a symbol to subscribe to, in this case one we know exists in the data (-s DE000CM95AU4.EUR.XPAR)

Putting this all together we get the full command line:

mamalistenc -m qpid -tport sub -S TEST -s DE000CM95AU4.EUR.XPAR

When we run the command, we should see output like the following (I've removed some of the output to make it a bit clearer):

[damian@openmama openmama]$ mamalistenc -tport sub -m qpid -S TEST -s DE000CM95AU4.EUR.XPAR
2013-12-19 10:59:20: *****************************************************************
Note: This build of the MAMA API is not enforcing entitlement checks. Please see the Licensing file for details
************************************************************************
Type CTRL-C to exit.
Created subscription: source=[TEST], symbol=[DE000CM95AU4.EUR.XPAR] DE000CM95AU4.EUR.XPAR.TEST.DE000CM95AU4.EUR.XPAR
Type: INITIAL Status OK
MdMsgType | 1 | U8 | 1
MdMsgStatus | 2 | U8 | 0
MdSeqNum | 10 | U32 | 1
MamaAppMsgType | 18 | U8 | 100
MamaSenderId | 20 | U64 | 18446744069414626310
wPubId | 495 | STRING | utpcasheuro_prod
wIssueSymbol | 305 | STRING | DE000CM95AU4.EUR.XPAR
wEntitleCode | 496 | I32 | 30870
wDisplayHint | 1195 | I32 | 0
wInstrumentType | 298 | STRING | Warrant
wOrderBookId | 4022 | STRING | 227424
wExchangeId | 3474 | STRING | 1
wAdjPrevClose | 520 | F64 | 0.000000
wPrevVolume | 438 | I64 | 0
| 3321 | STRING | 25
wInstTypeOrig | 1101 | STRING | 303
wLotSize | 324 | I64 | 1
wMarketCategory | 1168 | CHAR | G
wDescription | 258 | STRING | GOLD 2256TPIOPENZ
wCountry | 1048 | STRING | DEU
wMnemonic | 4527 | STRING | 4084Z
...
MamaSendTime | 16 | STRING | 21:30:12.231
DE000CM95AU4.EUR.XPAR.TEST.DE000CM95AU4.EUR.XPAR Type: UPDATE Status OK
MdMsgType | 1 | U8 | 0
MdMsgStatus | 2 | U8 | 0
MdSeqNum | 10 | U32 | 2
MamaAppMsgType | 18 | U8 | 0
MamaSenderId | 20 | U64 | 18446744069414626310
wPubId | 495 | STRING | utpcasheuro_prod
wIssueSymbol | 305 | STRING | DE000CM95AU4.EUR.XPAR
wEntitleCode | 496 | I32 | 30870
wDisplayHint | 1195 | I32 | 0
wInstrumentType | 298 | STRING | Warrant
wOrderBookId | 4022 | STRING | 227424
wExchangeId | 3474 | STRING | 1
...
MamaSendTime | 16 | STRING | 21:38:36.321
DE000CM95AU4.EUR.XPAR.TEST.DE000CM95AU4.EUR.XPAR Type: UPDATE Status OK
MdMsgType | 1 | U8 | 0
MdMsgStatus | 2 | U8 | 0
MdSeqNum | 10 | U32 | 3
MamaAppMsgType | 18 | U8 | 0
MamaSenderId | 20 | U64 | 18446744069414626310
wIssueSymbol | 305 | STRING | DE000CM95AU4.EUR.XPAR
wEntitleCode | 496 | I32 | 30870
wAdjPrevClose | 520 | F64 | 5.970000
wSeqNum | 498 | I32 | 10761
wLineTime | 1174 | STRING | 07:16:29.193
wSrcTime | 465 | STRING | 07:16:28.943
MamaSendTime | 16 | STRING | 21:38:49.918

Explanation

Hopefully the process here all seems straight forward, but this simplicity masks some complexity performed by OpenMAMA under the hood. When the capturereplayc application starts, it begins by initializing OpenMAMA and loading the required middleware bridge. It then sets up a series of publishers for each symbol found in the playback data, then creates a connection on the transport passed to it (what form this connection takes is managed by the loaded middleware bridge, and should be opaque to the user). This connection then waits for requests for data to be sent to it.

The mamalistenc application then starts up, again initializing MAMA and loading the bridge in the same way as capturereplayc. It then makes a request on it's transport for the dictionary, which the publisher sends back to the client. The client then proceeds to make requests for the symbols it is interested in. If the publisher recognizes the symbol requested, it reads the data from the playback file, constructs an initial (essentially a list of all the fields available for a symbol, and their current state), sends it, then publishes each update for that symbol as it finds them in the file.

When these reach the mamalistenc client application, it unpacks the message from the middleware format, then iterates over the data contained within, writing the full message to the screen. At this point it will match the contents of the message with the contents of the dictionary in order to help it better understand the data available, in particular using the FIDs from the message to determine the type of field to be decoded.

bookticker

As mentioned above, capturereplayc also allows us to make subscriptions to orderbook data using the MAMDA example applications. We're going to make use of one such application, bookticker, to demonstrate the functionality. As above, checking the help text for bookticker results in the following:

[damian@openmama openmama]$ bookticker --help Usage: bookticker [-S source] -s symbol [-s symbol ...] [-threads x] [-deltas] (For showing Deltas) [-e] (For showing Entries) [-k] (process market orders) [-tport] [-dict_tport]

Using the same process as mamalistenc, we decide the command line arguments we're going to use: middleware (-m qpid), transport (-tport sub), source (-S TEST), and symbol (-s bPTBIZJYE0064.EUR.ENXL). Note the lower case 'b' at the start of the symbol, which denotes that this symbol contains orderbook data. Putting this all together we get the following command line:

bookticker -m qpid -S TEST -tport sub -s bPTBIZJYE0064.EUR.ENXL

We'll need to stop the previous mamalistenc application if it's still running (easily achieved by pressing ctrl-c), then execute the command above. The resulting output should look like:

[damian@openmama openmama]$ bookticker -m qpid -S TEST -tport sub -s bPTBIZJYE0064.EUR.ENXL
2013-12-19 11:48:38: ********************************************************************
Note: This build of the MAMA API is not enforcing entitlement checks. Please see the Licensing file for details
********************************************************************
MamdaOrderBook strict checking is on
RECAP!!! (seq# 1)
Book for: bPTBIZJYE0064.EUR.ENXL
Time Num Size Price Act | Act Price Size Num Time
COMPLEX DELTA!!! (seq# 2)
Book for: bPTBIZJYE0064.EUR.ENXL
Time Num Size Price Act | Act Price Size Num Time
06:12:59.379 1 107 12.10 A | A 12.67 100 1 06:12:59.379
06:12:59.379 1 248 12.05 A | A 13.20 502 1 06:12:59.379
COMPLEX DELTA!!! (seq# 3)
Book for: bPTBIZJYE0064.EUR.ENXL
Time Num Size Price Act | Act Price Size Num Time
09:36:39.437 1 50000 12.63 A | A 12.67 100 1 06:12:59.379
06:12:59.379 1 107 12.10 A | A 12.68 50000 1 09:36:39.437
06:12:59.379 1 248 12.05 A | A 13.20 502 1 06:12:59.379
COMPLEX DELTA!!! (seq# 4)
Book for: bPTBIZJYE0064.EUR.ENXL
Time Num Size Price Act | Act Price Size Num Time
09:55:57.726 1 50000 12.62 A | A 12.67 100 1 06:12:59.379
06:12:59.379 1 107 12.10 A | A 12.68 50000 1 09:36:39.437
06:12:59.379 1 248 12.05 A | A 13.20 502 1 06:12:59.379

What you see output is a representation of the current state of the order book as maintained by MAMDA. As updates flow in, the book is updated, and levels will be added, removed and altered. Again, the otherwise simple commands hide a reasonable level of complexity which is managed by OpenMAMA under the hood.

What Next?

So, now you've had OpenMAMA up and running, and skipped through a reasonable range of the example applications, the question is where to go from here. At this stage I would take a look at some of the example application code, and start reading up on some of the other documentation. Checking out the code walk through for mamalistencpp will give you a feel for what it takes to write a relatively complete OpenMAMA application, and checking out the contents of the examples folder should demonstrate the breadth of functionality available.

From there, it really depends what you're looking for - you can try implementing some basic message passing for your own applications, dig into the OpenMAMA source and start fixing bugs, write a new middleware bridge for your favourite middleware, or build yourself a complete trading system using OpenMAMA as a backbone. The choice, as they say, is yours...

 

Damian