Posterous
Joel is using Posterous to post everything online. Shouldn't you?
Dsc_5799_-_version_2__1__thumb
Joel Reymont
owns this site
5
subscribers
 

Tenerife Skunkworks

Boldly going where few have gone before

AlgoScript

I'm on my third iteration of a translator from EasyLanguage. The first two versions were written in Haskell and OCaml and I'm using Lisp now. My goal is to produce code for a trading engine that runs in a shared library or DLL and can be embedded in other products such as NinjaTrader or TradeStation.

The original translator produced C# code but found this approach untenable. Every trading platform I looked at has a different set of trading functions. Generating C# code would have required me to write a library of supporting functions for every target platform to plug in the holes. 

I would have to write the code and test my libraries over and over again. It would also have required me to become an expert in every trading platform I wanted to translate for and made expanding my market rather tedious. Last but not least, anyone could grok my logic by looking at a translation or two and then write the code themselves using the libraries that I have painstakingly produced.

It struck me that I could translate into an intermediate language and build an embeddable execution engine that could run in every trading product I would target with my translator. All the trading products I looked at support DLLs. So long as I supplied my engine as a DLL and exported a set of functions, I could take in price quotes and return buy or sell instructions. 

Targeting my own trading engine simplifies development and testing and lets me focus on adding value to my own products instead of the products of others. I can focus on producing the best embeddable trading engine ever. I will depend on the host platform for price quotes and sending orders to the exchange, at least initially, but will add market data and execution interfaces over time.

Most of the trading products that I'm aware of run on the Windows desktop and are either written using .NET or and are migrating to .NET as we speak. These trading products use C# as their trading language and the differences between them are becoming less and less pronounced. 

I have no intention of slugging it out in the extremely crowded desktop trading space. The embeddable cross-platform trading space, on the other hand, is a great niche. Think unattended execution of trading strategies, grid-based analysis of massive volumes of market data and other mouth-watering goodies. 

The main issue to consider is the choice of trading language for the embeddable engine. Just as with C# on .NET, it's a choice determined by the implementation language. A Haskell-like DSL would have been nice but I shudder at the thought of Haskell as a DLL. I'm sorry but I could not resist the poke!

The OCaml syntax is quite rigid, although the LexiFI folks have hacked it to suit their needs. I could use Camlp4 but I had a very unpleasant experience with it. I mean do you dig the <:expr<, $lid:tbl$, $lid:x$. I do not!

I would like to present a translation of the EKam Scalper in AlgoScript. A Lisp by any other name would smell as sweet?

Filed under  //   compilers   lisp   trading  
Posted July 31, 2008
// 0 Comments

Who let the dogs out?

I've been quiet for the past couple of months and I can finally explain why.

I've been writing Topdog, a compiler from EasyLanguage to C#.

I realized back in February that I had enough money in the bank to last a few months and went through a brief period of soul-searching. Trading did not turn out so well for me but I still wanted to be involved. I always wanted to have a product of my own and on February 23 I set out to write Topdog.

I found the One-Day compilers presentation quite fascinating and so I started with OCaml. Three months have gone by and I now have two fully developed versions of the translator, one written in OCaml and another in Haskell. After briefly deploying the Haskell version I decided to throw it away and soldier on with OCaml.

Why Ocaml? Why not Haskell? I'll list the pros and contras below, in no particular order. I'll write up a summary first, for those of you not interested in reading the whole post.

Summary:

There's an elephant in the room. It's there, it's huge, it's something that nobody talks about: OCaml is the practical Haskell. It's functional, statically typed and blazingly fast. Performance of OCaml code is well defined. With OCaml you stop asking why your code is spending 70% of its time collecting garbage and start actually trying to polish your code.

I cannot emphasize this enough. GHC takes 3-4 minutes to rebuild my project whenever I touch the parser. Ocaml takes 2-3 seconds. I use a 2Ghz Core Duo MacBook Pro and Topdog is decidedly not a large project which makes the difference all the more glaring. With Haskell I was loath to touch the parser or my syntax tree definition, with OCaml I look forward to tweaking things to my hearts content. Normal recompilation time of Topdog is so far as to be almost unnoticeable.

Then there's F#, the OCaml for .NET.

Transforming Abstract Syntax Trees (ASTs)

Polymorphic variants in OCaml make creating "macros" easy, e.g. the following would work for two different ASTs.

let var = `Var 

You can share chunks of syntax trees with poly variants and type conversions:

http://gist.github.com/266868#file_gistfile8.ml

This is not possible in Haskell without boxing your AST chunks. To my best knowledge the above would look like this

data X = X Int

data Y = Y String

data Z = X1 X | Y1 Y

The above introduces an extra level of indirection that is not necessary in OCaml. It may not look like a big deal until you realize that any OCaml function that works on arguments of type X or Y can also work on Z. This is simply not possible in Haskell and requires extra code to unbox constructors X1 and Y1 in Z before the functions that work on types X and Y can be applied.

I also find that transforming ASTs with monads and type classes clutters up my code:

http://gist.github.com/266868#file_gistfile1.hs

        
Yes, you can get used to the return-s and liftM-s but I find they take away from the original intent of the code. Compare to OCaml code:

http://gist.github.com/266868#file_gistfile2.ml

Scrapping Your Boilerplate Code

On the Haskell side there's Scrap your boilerplate code (SYB). The compiler has a type checking pass and I embed token location in the syntax tree to help with error reporting.

http://gist.github.com/266868#file_gistfile3.hs


I need to strip token locations for testing and the syntax tree is quite large. Traversing the tree to dig down to TokenPos and extract Expr is pages of boilerplate code. This is where SYB comes to the rescue and reduces pages of code to the following

http://gist.github.com/266868#file_gistfile4.hs


Everything is not peachy, though. SYB works perfectly when GHC can automatically derive Data and Typeable for you (see Expr above). My previous approach was to embed the position returned by Parsec and to start processing the AST from the parser result. This required me derive Data and Typeable by hand since Parsec's SourcePos and ParserError are hidden in the Parsec library. This required silly code like this

http://gist.github.com/266868#file_gistfile5.hs

and

http://gist.github.com/266868#file_gistfile6.hs


It's less code than I would have had to write without SYB but annoying nonetheless.

What about OCaml? I'm glad you asked!

The OCaml version uses the Camlp4 preprocessor which I vastly prefer over Template Haskell.

http://gist.github.com/266868#file_gistfile7.ml


Other bits

OCaml has built-in marshalling which lets you easily dump data to disk and load it back. There's a big push behind binary serialization with GHC nowadays and it's not hard to marshall your data with Haskell, except it has to be done manually. Libraries like Data.Derive or DrIFT can help you with this task but marshalling with OCaml is still easier.

The Haskell version of Topdog uses Parsec for parsing and PPrint for pretty printing. The OCaml version uses Emmanuel Orzon's dypgen GLR parser generator and Christian Lindig's pretty printer, part of his Quest Tester project.

Filed under  //   compilers   haskell   ocaml   trading  
Posted May 26, 2007
// 0 Comments

The 90 minute Scheme to C compiler

I started writing the EasyLanguage translator in Erlang today but failed to find comprehensive documentation for the lexer and the grammar parser so I switched to Scheme. To be fair, I was also inspired by Bill Clementson's update on Termite. I have a soft spot for Lisp and I can't fight it. Hell, I want to be the next Naughty Dog!

I'm fond of Lisp but cannot afford neither LispWorks nor /God forbid!/ Allegro Common Lisp at the moment. LispWorks has excellent Mac OSX support and comes with an Objective C bridge. It does not come with a shared library runtime, though, which means that I cannot embed it into my applications. 

Yes, some people take the opposite view and of course you can make a shared library out of your application logic and load that into LispWorks but I'm not convinced. Then there's the issue of buying support every year. This and the prohibitive Franz licensing has been discussed ad nauseum. The end result is that there's no Lisp that can be acquired on the cheap and used cross-platform.

This malaise does not apply to Scheme. There are plenty of excellent Scheme implementations but Gambit and Chicken are my favorite. Termite runs on Gambit but pretty much everything I have seen can be easily ported between the two. Gambit is the speed demon but Chicken has the best library support by far. 

I decided to go with Chicken and headed straight to the parsing tools section of the site. I was rewarded with LALR and Silex. An email exchange with Dominique Boucher brought me insight into making LALR and Silex cooperate... and the news that Dominique and Guillaume Germain of Termite work together. Now, if this is not a good sign then I don't know what is. 

I'm a big believer in signs and find that they all point in the same direction when that direction is the right one. The 90 minute C compiler presentation by Mark Feeley was yet another sign. 

I have reservations about applying Erlang to financial analysis and some readers have pointed out that my focus on fault-tolerance might be overrated. I could write my translator in Erlang but I would have no way to give my prospects or customers a binary for a quick demo, I would need to ask them to install the whole Erlang/OTP distribution. 

Scheme compiles to C, is usually free and definitely cross-platform. Building Scheme binaries is a breeze and Scheme can easily be embedded. Scheme seems like the way to go for my translator and possibly even for the trading platform. I will keep you posted!

Filed under  //   compilers   lisp  
Posted January 24, 2006
// 0 Comments