So you have perf. degradation right there.

Additionally you have to parse text based HTML header to know binary data are send over.

I agree for the most cases this probably does not matter much but in real-time or large number transaction systems (i.e. GOOGLE like case) even such overheads will magnify total results.

Unit test takes around 2 hours while automated integration tests 20-25 minutes. Target of integration tests in our case is to mainly prove trading system is in good shape to be tested on much “deeper” level.

We are also quit a bit constrained availability of external resources (mainly exchange trading hours on testing environments).

BTW: Integration test results in our case are much better indicator of code readiness for QA than unit tests itself.

How long it takes you to do it if case you run integration tests.

I hope this helps.

I started the construction of a comprehensive IT Services Stack. I make it available to the public domain. The early setup is available here:

http://soa-eda.blogspot.com/2008/03/it-services-stack-collaboration.html

I would appreciate to get input from professionals in order to mature and extend the model. Perhaps you could provide some input on one or more of the subjects and/or give the initiative some attention on your blog.

Thanks,
Jack

I have to say majority of my work so far was focused on front office application running as SaaS business model (i.e. fully managed solution for banks). Just last year I have chance stepped into middle office “water” functionality (i.e. clearing trades for that matter). Therefore majority my experience with integrating our system with “external” banks comes from trading part. From this point of view I have to say we do not have any client who would use FpML or XML as data exchange format with us. What they usually ask is “plain” CSV file (or DB version) based on “variant” of ClearVision “welcome” table. This leads me to think “banks” data import process is somehow accustomed to CSV form and servers as bases for transformation to their internal format. As emerging “standard” here I can see FIX drop copy IN/OUT variants which gets quite big traction recently.

Speaking about parallel execution I’m looking forward to hear your experience with resolving issues on this topic especially when there is ordered execution requirement involved.

Latency time measurement on high throughput system can get quite tricky on any platform especially if you need to do consistent monitoring of all events in precision of several millisecond (say total exec. time up to 5ms on any price update from incoming exchange message to update on user screen or delivered to external system). We are running our application exclusively on MSFT server version which does not support us on this case well. Therefore we have to take special steps to compensate for it. And through this experience I know correlation of time on distributed system might get quite “big” correlation errors (well big in terms of short latency time we are measuring on updates here) especially when system is under high load. From all that steams my interest in latency time measurement. Thanks for putting light on this issue from other system perspective.

-Libor

Good overview of the recent banking track at Qcon 2008 in London – talks about Low Latencym processing, John Davies presentation, Swift ML, Betfair’s architecture and more.

Both examples I gave (again pulled up at the last minutes I might add) were cases where, as you say, perfect parallelisation gives near perfect scalability however order-dependent systems are less well served. Obviously this reflects Amdahl’s law and if I’d had more time I would have loved to have shown you some of the issues with had with parallel threads in distributed systems and how we got around them (most of them). Perhaps a topic for QCon in Denmark later this year.

Finally you ask about latency timing, frequently network monitors will go down to nS timing but most Linux systems will happily measure micro seconds or in some cases better. Since we’re usually talking about a few milliseconds a resolution of micro seconds give us plent of resolution to measure latenct at this level. A useful tool we frequently use for seeing what’s going on on the network is ethereal or tethereal (now replaced by wireshark), I think you can get MS versions of it but don’t try it at work a you’ll probably get fired if anyone catches you using it (i.e. try it but don’t get caught). Distributed systems are normally synchronised by NTP, it takes into account the network latency but of course there is always some level of error, I would expect systems in a subnet to be synchronised to better than 2ms.

Good write up, please say hi at the next show,

-John Davies-