Posterous
Joel is using Posterous to post everything online. Shouldn't you?
Dsc_5799_-_version_2__1__thumb
Joel Reymont
owns this site
5
subscribers
 

Tenerife Skunkworks

Boldly going where few have gone before

AlgoScript

I'm on my third iteration of a translator from EasyLanguage. The first two versions were written in Haskell and OCaml and I'm using Lisp now. My goal is to produce code for a trading engine that runs in a shared library or DLL and can be embedded in other products such as NinjaTrader or TradeStation.

The original translator produced C# code but found this approach untenable. Every trading platform I looked at has a different set of trading functions. Generating C# code would have required me to write a library of supporting functions for every target platform to plug in the holes. 

I would have to write the code and test my libraries over and over again. It would also have required me to become an expert in every trading platform I wanted to translate for and made expanding my market rather tedious. Last but not least, anyone could grok my logic by looking at a translation or two and then write the code themselves using the libraries that I have painstakingly produced.

It struck me that I could translate into an intermediate language and build an embeddable execution engine that could run in every trading product I would target with my translator. All the trading products I looked at support DLLs. So long as I supplied my engine as a DLL and exported a set of functions, I could take in price quotes and return buy or sell instructions. 

Targeting my own trading engine simplifies development and testing and lets me focus on adding value to my own products instead of the products of others. I can focus on producing the best embeddable trading engine ever. I will depend on the host platform for price quotes and sending orders to the exchange, at least initially, but will add market data and execution interfaces over time.

Most of the trading products that I'm aware of run on the Windows desktop and are either written using .NET or and are migrating to .NET as we speak. These trading products use C# as their trading language and the differences between them are becoming less and less pronounced. 

I have no intention of slugging it out in the extremely crowded desktop trading space. The embeddable cross-platform trading space, on the other hand, is a great niche. Think unattended execution of trading strategies, grid-based analysis of massive volumes of market data and other mouth-watering goodies. 

The main issue to consider is the choice of trading language for the embeddable engine. Just as with C# on .NET, it's a choice determined by the implementation language. A Haskell-like DSL would have been nice but I shudder at the thought of Haskell as a DLL. I'm sorry but I could not resist the poke!

The OCaml syntax is quite rigid, although the LexiFI folks have hacked it to suit their needs. I could use Camlp4 but I had a very unpleasant experience with it. I mean do you dig the <:expr<, $lid:tbl$, $lid:x$. I do not!

I would like to present a translation of the EKam Scalper in AlgoScript. A Lisp by any other name would smell as sweet?

Filed under  //   compilers   lisp   trading  
Posted July 31, 2008
// 0 Comments

Transparency and masking

CAPI is a good cross-platform GUI toolkit and has been used to write apps like Prime Trader. Your apps will have a native look on each platform and you won't have to do anything special to make it happen. Still, bits of platform speific code are still required and one example is alpha-blending.

There's no support for alpha-blending in CAPI. You can do masking but specifying a transparent color in your images but this is way too basic to render a poker room so I have to resort to platform-specific code to make it happen.

The poker room is a composite image where most of the elements, save for the carpet, have a greyscale mask in a separate file. A greyscale (0-255) mask lets you specify translucency and simulate shading. This is how the shade cast by the table and chairs is done. The carpet is drawn first, two to ten chairs go next, the table is sandwiched on top and finally the cards, chips, buttons, etc. are drawn.

I should be able to make the poker room a pinboard layout, set the composite poker room image as the background and make dynamic objects such as cards, chips and buttons into pinboard objects.

Alpha-blending is key to this project and is surprisingly easy to simulate even for images formats that do not support an alpha channel. I'm storing most of the images as JPEGs with some BMPs mixed in and the format for each picture is chosen to save space while preserving image quality. A carpet is just 360K as a JPEG, for example, while over 1.1Mb when stored as a PNG without the alpha channel.

The formula to use for alpha-compositing is

    displayColor = sourceColor * alpha / 255
    + backgroundColor * (255 - alpha) / 255

which requires you to retrieve the values of each pixel in 3 images and store the target pixel back after multiplication. LispWorks color components are float values from 0 to 1 so the formula will look slightly different.

The kosher way to accomplish the blending with LispWorks is to load the images, create image access handles, load the pixel data into the image access structures and then retrieve pixels in a loop. Pixel values need to be converted into a color spec before individual color components can be retrieved. This is the naive way of doing it:

This is also a very slow way of doing it as it takes 11-12 seconds to blend three 794x538 JPEG images (carpet, table, mask) on my Powerbook G4 1.25Gz.

I did some digging around at the beginning of the year to be able to load image data into OpenGL textures. I'm not using OpenGL this time but I need to assign the bitmap data back to the image after modifying it.

Getting the Cocoa image handle is done by calling (image-ns-image image), assuming that image was the result of (gp:load-image ...). Bitmap data can be retrieved like this:

and blending is just an optimized array loop:

There's one last bit that needs to be done before the image data is updated. It took me a whole evening of poking around and googling before I finally found my answer.

TIFFRepresentation returns a copy of the bitmap data and the easiest way to update the image is to remove the returned representation and to add it right back with code like this:

Voila! It takes less than a second to blend three large images together. Problem solved. I will still need to poke around to get appropriate code under Windows and Linux.

Filed under  //   lisp  
Posted January 21, 2008
// 0 Comments

Overloading in Lisp

This is a quick summary of findings related to vector processing in Lisp...

There's a comp.lang.lisp thread that started with my asking about overloading but has now taken an interesting turn towards additions to the CLOS MOP. A big thank you to everyone who replied.

I found "Increasing Readability and Efficiency in Common Lisp" to be very useful and there's also a cool example posted by Nikodemus to the #lisp IRC channel (thanks to John for pointing it out). I'm including the example below as it does exactly what I need. It even works in LispWorks!

Last but not least there are the proceedings of the April 2004 Lisp User meeting in Amsterdam.

Filed under  //   lisp  
Posted December 31, 2006
// 0 Comments

memcpy in Lisp

The title is not correct since I'm trying to copy an array of pixels x 3 into an array of pixels x 4 while setting every 4th byte from another array of pixels. I tried different forms of optimizing things in LispWorks and always used these optimization settings:

(eval-when (compile)
  (hcl:toggle-source-debugging nil))

(eval-when (compile load eval)
  (defvar *optimize* '(optimize (safety 0)
                       (space 0)
                       (debug 0)
                       (float 0)
                       (hcl:fixnum-safety 0)
                       (speed 3))))

opengl:gl-vector-aref is actually a macro that expands to fli:dereference ... :index ...

Fastest code:

It positively beats the other two approaches and for some reason has 0 memory allocations while the two examples below have about 150 each. The code is specific to LispWorks.

Slow approach #1:

Slow approach #2 (mixes up the bytes someplace):

Filed under  //   lisp   lispworks   performance  
Posted March 5, 2006
// 0 Comments

The 90 minute Scheme to C compiler

I started writing the EasyLanguage translator in Erlang today but failed to find comprehensive documentation for the lexer and the grammar parser so I switched to Scheme. To be fair, I was also inspired by Bill Clementson's update on Termite. I have a soft spot for Lisp and I can't fight it. Hell, I want to be the next Naughty Dog!

I'm fond of Lisp but cannot afford neither LispWorks nor /God forbid!/ Allegro Common Lisp at the moment. LispWorks has excellent Mac OSX support and comes with an Objective C bridge. It does not come with a shared library runtime, though, which means that I cannot embed it into my applications. 

Yes, some people take the opposite view and of course you can make a shared library out of your application logic and load that into LispWorks but I'm not convinced. Then there's the issue of buying support every year. This and the prohibitive Franz licensing has been discussed ad nauseum. The end result is that there's no Lisp that can be acquired on the cheap and used cross-platform.

This malaise does not apply to Scheme. There are plenty of excellent Scheme implementations but Gambit and Chicken are my favorite. Termite runs on Gambit but pretty much everything I have seen can be easily ported between the two. Gambit is the speed demon but Chicken has the best library support by far. 

I decided to go with Chicken and headed straight to the parsing tools section of the site. I was rewarded with LALR and Silex. An email exchange with Dominique Boucher brought me insight into making LALR and Silex cooperate... and the news that Dominique and Guillaume Germain of Termite work together. Now, if this is not a good sign then I don't know what is. 

I'm a big believer in signs and find that they all point in the same direction when that direction is the right one. The 90 minute C compiler presentation by Mark Feeley was yet another sign. 

I have reservations about applying Erlang to financial analysis and some readers have pointed out that my focus on fault-tolerance might be overrated. I could write my translator in Erlang but I would have no way to give my prospects or customers a binary for a quick demo, I would need to ask them to install the whole Erlang/OTP distribution. 

Scheme compiles to C, is usually free and definitely cross-platform. Building Scheme binaries is a breeze and Scheme can easily be embedded. Scheme seems like the way to go for my translator and possibly even for the trading platform. I will keep you posted!

Filed under  //   compilers   lisp  
Posted January 24, 2006
// 0 Comments

Naughty Dog and the fundamental turn towards concurrency

Jak & Daxter by Naughty Dog is the best example of Lisp used in game development. There is a good postmortem on the project at Gamasutra.

People have wondered why Naughty Dog is not using GOAL (Game-oriented Assembly Lisp) for their PS3 games and yesterday I stumbled upon a detailed answer. It insightful to read through the whole thread but you might want to skip directly to the example of what GOAL looks like.

There might be a quiet resurgence of Lisp use in game development but is Lisp the answer, specially now that the free lunch is over? Lisp does not force you to code for concurrency whereas Erlang does.

I have time to spare in the next couple of weeks and I’m thinking of rewriting Frag from Haskell to either Lisp or Erlang. I’m pretty good at both but which should I pick on my quest for interesting conclusions and amazing discoveries?

Filed under  //   concurrency   games   lisp  
Posted January 17, 2006
// 0 Comments

My long road to Lisp

I was asked in a comp.lang.lisp thread to elaborate on my choice of Lisp for writing poker room software, whether I had to convince my backers, etc. Poker room software, if you are wondering, is the client that you download at Pokeroom and the server that the client connects to.

The short answer is in this article and no, because I have no backers. The longer answer is quite long because I'm driven not only by business rationale but also by lots of emotional reasons.

I started programming at about 14 while in Cuba, using Basic. I got my first paid computer job at 17. I was living in Russia and had to break the copy protection of some medical software. At 18 I was at Z-Code software in San Rafael, working with a bunch of great hackers. Anyone remembers Z-Mail for Unix?

This last point really has something to do with Lisp as the guy responsible for the terminal version of Z-Code was Bob Glickstein who wrote a fine book about Emacs extensions. I'm sure Bob would have written Z-Mail in Lisp if the owners of the company would have let him. I don't think they were as fond of Lisp as he was. As for me, those setq's in my .emacs were really foreign to me and I was always begging Bob to fix this or that thing for me. That was 1994.

From 19 to 24 I made a fine career on Wall St finishing as head of prime brokerage technology at Deutsche Bank with a large financial package and a brand new Land Rover. I spent most of that time, writing trade processing systems in Perl. I wonder if this has been the secret of my success?

I did internet consulting for about a year, still hacking Perl, then switched to Java and started running my own offshore development shop. We even delivered a .NET app once /I can't believe I'm saying this!/. Most of the work was for the gaming industry and at some point the client ordered poker room software and I let a Siberian programming team convince me that the only way I could fit into the client's 125K budget was if I let them write the software in Delphi.

They suggested that I would have to charge the client 200K+ if they were to use C++. Of course I did not tell them how much I was charging the client, I was just multiplying their quote by two. I learned afterwards that they were a Delphi-only shop. I would say things went downhill from there. After a bankruptcy (the bank took the Land Rover away), botched poker project, returning to Russia to minimize costs and run my own development team on-site, disbanding the dev team for lack of work and selling software for a year, I decided to resurrect the Delphi poker.

For anyone interested, the original poker source code is still available. Going open source was a marketing move on my part, that was the niche I was gonna occupy. The Linux of Poker Room Software! That did not work out, btw. I had better success selling the 90% done source code previously /did it twice/.

If I learned one thing during my year in software sales that was that it's better to sell a product that you own than to sell your services. I want to get rich! Yes, I do. I have not had a corporate job in 6 years and I want it to continue this way. I want the freedom to work on whatever I want! I have this freedom now except that I'm quite poor. Oh, well, at least I live in Tenerife and the temperature during the day never falls below +20C.

We are getting close to Lisp, I promise...

I consider myself to be a pragmatic and logical type. Maybe too logical since my psychologist suggested was that I pay closer attention to my feelings when making my decisions.

We are even closer to Lisp now...

That Delphi poker code is crap. I cannot bear the thought of looking at it again. I earnestly spent a couple of months trying to debug it and generally fix it up. No way, this is an app that by design cannot be verified! And I don't even like playing poker. Nonetheless, I had an opportunity on my hands to gain my creative freedom and I decided to excercise that opportunity. So back last summer I dedicated myself to finishing that poker software.

I'm a Unix guy and I hate Windows. I have been lusting after the PowerBook G4 ever after it came out. Software sales paid good and I bought one a year ago and had to take this into account when choosing my poker development tools. I considered Java too slow, Python too wordy, C# was not ready for cross-platform prime-time. So I chose C++.

Game development is something that I wanted to learn for a while and so I saved myself a bunch of Gamasutra articles before going on vacation last September. This one was a revelation. I checked out Scheme and liked it a lot, I'm still not sure why. I don't think it was Scheme in particular, I think it was just the idea of using Lisp. Still, I came back and stuck to my C++ guns.

I consider myself handy with C++. I don't shy from learning complex things if they save me time. I started using the C++ Boost libraries. I wanted elegante ways to serialize my classes and Boost seemed to provide that. I scrapped my own hairy serialization code and dived into Boost. For one reason or another things did not turn out as elegant as I expected. I found that I'm not enjoying coding and that time that I could have better spent on application functionality is going to very low-level stuff that I should have been done with a while ago.

I forgot what straw broke the camel's back but I switched with Scheme a couple of months ago. Yes, all this swinging from one language to another was delaying my plans of eventual richess but if I was gonna spend most of the day at the keyboard I thought I should enjoy the process.

I read someone's response to the "My road to Lisp" survey and how they started with Scheme and then took up Common Lisp. I looked at CL and I don't think I want to go back. I came to think of it as a purely emotional decision. I'm lazy and I much prefer (incf foo) to (set! foo (+ foo 1)).

As for writing poker in Lisp, my card code (including hand ranking) in C++ is 450 lines vs. 111 lines of Lisp. The unit tests are 2019 lines of C++ and just 118 lines of Lisp. And most important of all, I'm thoroughly enjoing myself. Coding in Lisp just feels deep down good so why would I code in anything else?

I think my poker software will be much easier to maintain and enhance in Lisp. Adding new card games will be a snap. Lisp is made for writing poker-playing bots. Last but not least, I can offer clients the source code when they buy a license. I doubt they will go to anyone else for enhancements.

Filed under  //   joelr   lisp  
Posted December 20, 2005
// 0 Comments