You right on this regard if it is the conversion from [tex]x+y+z[/tex[ to an image tag that is the challenge?

I've not used PHPBB for many years, so I'm not sure how much control there is within plugins. Is the problem that PHPBB doesn't find the closing [/tex] tag when you add new lines, or that the image tag it creates has newlines characters within the source attribute.

I'd be surprised if the later caused a problem. However I would have though there must be many other plugins that hit similar problems. There might be a simple hack that can be implemented - thats what we used to do with PHPBB v1 when I used it. Eventually we decided it was easier to write our own forum (that you see today).