Also look at: http://sourceforge.net/tracker/index.php?unc=detail&aid=1509758&group_id=103281&atid=738747

I think this has the basis of everything that is needed. I've not tested this, but having launched the editor (i.e. equationeditor.php) in the appropriate way from TinyMCE, you will possibly just need to replace the FCKEditor code in the javascript at the front with:

var text=getEquationStr();
  text = '<img src="<?php echo(EQUATIONENGINE); ?>?'+escape(text)+'" />';
 
  tinyMCE.execCommand('mceInsertContent', false, text);

That should at least get the equation image into the editor. Ideally what we then need to do is enable TinyMCE editor to somehow allow images of this form to be double clicked and edited, by relaunching the equation editor with the old equation. Probably a lot tricker.