We are developing an online collaborative whiteboard, and we are using Canvas to display all graphics and text on boards. There is no out-of-the-box solution for displaying text on Canvas as you would in regular HTML. Hence, over the past several years, we’ve been iterating on this; and we finally think we’ve reached the best solution.
In this article, we will walk you through this transition from Flash to Canvas and why we gave up on using SVG foreignObject in the process.
In 2015, we were making a product using Flash. Flash has a built-in text editor that works well, so we didn’t have to do anything extra with that. But at that time, Flash was already dying, and eventually, we decided to move to HTML5 Canvas. The task we were faced with was the following: we needed to draw text on Canvas exactly as in regular HTML while also keeping all text that was previously made in Flash.
We wanted to allow the user to edit text directly in our product, without noticing the transition between editing and viewing modes. We were thinking about a text editor that appears when the user clicks on a text area, and that can be closed by moving the cursor away from it. The text displayed in that editor, however, should look identical to the text drawn on Canvas.
We used an open source library for the editor. Still, we were not comfortable with the existing solutions for rendering HTML elements on Canvas due to performance issues and the lack of functionality.
We considered several solutions:
The SVG foreignObject workflow is as follows: we have the HTML code from the editor -> we put code into foreignObject -> magic happens -> we get an image -> we put that image on Canvas.
About that magic part. While most browsers support foreignObject, each browser has its own set of quirks when it comes to passing the data to Canvas. Firefox uses Blob object; in Edge, you need to encode the image as a Base64 string and use the data URI scheme; and in IE11, foreignObject doesn’t work at all.
After that work, we found some interesting bugs we hadn’t seen in Flash. Text with the same font and size was displayed differently in different browsers. For example, the last word on a line could wrap and overlap the text on the next line. It was important to us that users get widgets displayed the same, regardless of the browser they use. There was no problem with that in Flash since it is the same everywhere.
We did solve this problem. First, we started calculating the width of each text line independently of browser and server data. The difference in height remained, but it doesn’t bother the users in our case.
Second, experimentally we found that we needed to add some unusual CSS properties to the editor and the SVG element to reduce the display difference between browsers:
What we got from SVG foreignObject in the end:
Everything was going well, but one day our designers came to us and asked us to add font support for their mockups.
We wondered if this could be done using foreignObject. It turned out that foreignObject has a particular feature that becomes a fatal flaw in this case. It can display HTML, but it cannot access external resources, so they need to be encoded as a Base64 string and added inside the SVG.
This means that if you have four text objects that use OpenSans as a font, that font needs to be downloaded four times by the user. This approach did not suit us.
We decided to implement our own Canvas Text, with good performance, vector image, and IE11 support.
Why do we need vector images? Each object in our product can be zoomed, and with vector graphics, we don’t have to create additional images for different levels of zoom. In the case of Canvas.fillText(), which draws a raster image, we have to redraw it each time the zoom level is changed, which we thought had a significant impact on performance.
The first thing we did was a simple prototype to test the performance.
How this prototype worked:
The prototype had two goals: to check that there is no delay when Canvas redraws the objects on zoom and that the time it takes to turn HTML into an object does not exceed the time it takes to make an image out of SVG.
The first goal was met successfully. Zooming hardly had any impact on the performance. But there were problems with reaching the second goal: processing of a large enough text took a significant amount of time, and the first performance measures yielded poor results. The new approach took almost twice as long as SVG to draw a 1000-character text.
We decided to use the most reliable method of code optimization: replacing the testing method with one we like.
We went to our analysts and asked them about the most common text length that our users use. It turned out that the average text length was 14 characters. For short texts like this, our prototype showed significantly better results because it has a linear slowdown as the text length is increased while the SVG wrapping took almost always the same amount of time, regardless of the text length used. We were okay with that: we can safely lose large-text performance if, on average, this approach would be faster than using SVG.
After several variations of updating Canvas Text, we arrived at the following algorithm:
Step 1. Break the text into logical chunks
Step 2. Join the chunks into a single object with the correct coordinates and styles
One advantage of this approach is that we can cover all our code, from HTML to making a text object, with unit tests. Thanks to this, we can check the parsing and the drawing steps separately, which helped us to significantly speed up the development process.
In the end, we added font support, IE11 support, and covered everything with unit tests, and the drawing speed has, in most cases, increased compared to foreignObject. We tested this version on our beta users and released it. It seems like a success!
Until our tech support received messages from the users who use a right-to-left (RTL) writing system. It turned out that we forgot about the existence of such languages.
Fortunately, it wasn’t hard to add support for RTL, since the standard Canvas.fillText() already supports it.
But while we were dealing with that, we found even more interesting cases that fillText() didn’t support. We came across bidirectional texts, where part of the text is written from right to left, then from left to right, and then from right to left again.
The only solution we knew was to go to the W3C specification for browsers and try to replicate it inside Canvas Text. It was tough and painful, but we managed to add basic support. More info about the bidirectional algorithm here and here.
If you have experience in solving such problems, please share it in the comments.
P.S.: This article was first published on Medium.
In this article, we will walk you through this transition from Flash to Canvas and why we gave up on using SVG foreignObject in the process.
Transition from Flash
In 2015, we were making a product using Flash. Flash has a built-in text editor that works well, so we didn’t have to do anything extra with that. But at that time, Flash was already dying, and eventually, we decided to move to HTML5 Canvas. The task we were faced with was the following: we needed to draw text on Canvas exactly as in regular HTML while also keeping all text that was previously made in Flash.
We wanted to allow the user to edit text directly in our product, without noticing the transition between editing and viewing modes. We were thinking about a text editor that appears when the user clicks on a text area, and that can be closed by moving the cursor away from it. The text displayed in that editor, however, should look identical to the text drawn on Canvas.
We used an open source library for the editor. Still, we were not comfortable with the existing solutions for rendering HTML elements on Canvas due to performance issues and the lack of functionality.
We considered several solutions:
- Use Canvas.fillText(). It draws text the same as regular HTML, can be stylized and works in all browsers. Yet it doesn’t support multiline text with variable formatting. This problem can be solved, but it requires a significant time investment.
- Draw DOM over Canvas. This option did not work for us, because in our product, each object on Canvas has a z-index, which conflicts with z-index in DOM.
- Convert HTML to SVG. HTML can be converted to an image using a foreignObject element. It allows us to bake HTML inside SVG and work with it as an image. This is the option we chose.
Working with SVG foreignObject
The SVG foreignObject workflow is as follows: we have the HTML code from the editor -> we put code into foreignObject -> magic happens -> we get an image -> we put that image on Canvas.
About that magic part. While most browsers support foreignObject, each browser has its own set of quirks when it comes to passing the data to Canvas. Firefox uses Blob object; in Edge, you need to encode the image as a Base64 string and use the data URI scheme; and in IE11, foreignObject doesn’t work at all.
getImageUrl(svg: string, browser: string): string {
let dataUrl = ''
switch (browser) {
case browsers.FIREFOX:
let domUrl = window.URL || window.webkitURL || window
let blob = new Blob([svg], {type: 'image/svg+xml;charset=utf-8'})
dataUrl = domUrl.createObjectURL(blob)
break
case browsers.EDGE:
let encodedSvg = encodeURIComponent(svg)
dataUrl = 'data:image/svg+xml;base64,' + btoa(window.unescape(encodedSvg))
break
default:
dataUrl = 'data:image/svg+xml,' + encodeURIComponent(svg)
return dataUrl
}
After that work, we found some interesting bugs we hadn’t seen in Flash. Text with the same font and size was displayed differently in different browsers. For example, the last word on a line could wrap and overlap the text on the next line. It was important to us that users get widgets displayed the same, regardless of the browser they use. There was no problem with that in Flash since it is the same everywhere.
We did solve this problem. First, we started calculating the width of each text line independently of browser and server data. The difference in height remained, but it doesn’t bother the users in our case.
Second, experimentally we found that we needed to add some unusual CSS properties to the editor and the SVG element to reduce the display difference between browsers:
- font-kerning: auto; which controls the kerning of the font. More information
- webkit-font-smoothing: antialiased; which controls the antialiasing of the font. More information.
What we got from SVG foreignObject in the end:
- We can use any content: text, tables, charts
- foreignObject returns an image in vector format
- It works in all modern browsers except IE11.
Why we moved away from foreignObject
Everything was going well, but one day our designers came to us and asked us to add font support for their mockups.
We wondered if this could be done using foreignObject. It turned out that foreignObject has a particular feature that becomes a fatal flaw in this case. It can display HTML, but it cannot access external resources, so they need to be encoded as a Base64 string and added inside the SVG.
This means that if you have four text objects that use OpenSans as a font, that font needs to be downloaded four times by the user. This approach did not suit us.
We decided to implement our own Canvas Text, with good performance, vector image, and IE11 support.
Why do we need vector images? Each object in our product can be zoomed, and with vector graphics, we don’t have to create additional images for different levels of zoom. In the case of Canvas.fillText(), which draws a raster image, we have to redraw it each time the zoom level is changed, which we thought had a significant impact on performance.
Creating a prototype
The first thing we did was a simple prototype to test the performance.
How this prototype worked:
- We send a text to a function
- The function returns an object, which contains every word from the received text with coordinates and styles for drawing
- We send this object to Canvas
- Canvas draws the text.
The prototype had two goals: to check that there is no delay when Canvas redraws the objects on zoom and that the time it takes to turn HTML into an object does not exceed the time it takes to make an image out of SVG.
The first goal was met successfully. Zooming hardly had any impact on the performance. But there were problems with reaching the second goal: processing of a large enough text took a significant amount of time, and the first performance measures yielded poor results. The new approach took almost twice as long as SVG to draw a 1000-character text.
We decided to use the most reliable method of code optimization: replacing the testing method with one we like.
We went to our analysts and asked them about the most common text length that our users use. It turned out that the average text length was 14 characters. For short texts like this, our prototype showed significantly better results because it has a linear slowdown as the text length is increased while the SVG wrapping took almost always the same amount of time, regardless of the text length used. We were okay with that: we can safely lose large-text performance if, on average, this approach would be faster than using SVG.
After several variations of updating Canvas Text, we arrived at the following algorithm:
Step 1. Break the text into logical chunks
- Split the text into blocks: paragraphs, lists, etc.
- Split the blocks into smaller blocks according to the style used
- Split the small blocks into words.
Step 2. Join the chunks into a single object with the correct coordinates and styles
- Calculate the width and height of every word
- Join words that were broken apart after substep 2 of the previous step
- Make a line out of words; if a word doesn’t fit on the line, cut it until it does
- Make paragraphs and lists
- Calculate X and Y for each word
- Get a full object for drawing.
One advantage of this approach is that we can cover all our code, from HTML to making a text object, with unit tests. Thanks to this, we can check the parsing and the drawing steps separately, which helped us to significantly speed up the development process.
In the end, we added font support, IE11 support, and covered everything with unit tests, and the drawing speed has, in most cases, increased compared to foreignObject. We tested this version on our beta users and released it. It seems like a success!
Success only lasted 30 minutes
Until our tech support received messages from the users who use a right-to-left (RTL) writing system. It turned out that we forgot about the existence of such languages.
Fortunately, it wasn’t hard to add support for RTL, since the standard Canvas.fillText() already supports it.
But while we were dealing with that, we found even more interesting cases that fillText() didn’t support. We came across bidirectional texts, where part of the text is written from right to left, then from left to right, and then from right to left again.
The only solution we knew was to go to the W3C specification for browsers and try to replicate it inside Canvas Text. It was tough and painful, but we managed to add basic support. More info about the bidirectional algorithm here and here.
The brief conclusions we have made for ourselves
- Use SVG foreignObject to display HTML as an image.
- Always analyze your product before making a decision.
- Make prototypes. They can show that what seems to be a complicated solution can actually be pretty straightforward.
- Write the code from the very start in a way that it can be covered with unit tests.
- In international products, it is important to remember that there are a lot of different languages, including ones with bidirectional writing systems.
If you have experience in solving such problems, please share it in the comments.
P.S.: This article was first published on Medium.