I wanted to implement a whole canvas native workspace, I started with a html only implementation and "fake canvas" approach but even for moderate number of panels (~50) the responsiveness dropped quite a lot. So I went with a new implementation with "real canvas" and worked out the panels/interaction.
In terms of performance this approach worked out much better (able to easily handle 1000+ panels/nested panels at 60+ fps) but the complexity goes very high in the implementation of each panel. As in I cant just use for eg a markdown rendering js library or any third party ui component. And frequently make a change would result in massive performance degradation.
Currently I have text/markdown/image/audio/video and pdf panel (you can drag and drop files)
I wonder if i should take another go at it and implement using html-in-canvas api. Afraid of the api being not maintained if they dont see benefit or usage?
I would use it to render HTML elements such as inputs or a website in videos. Even though HTML is powerful the control over every pixel is important for high quality videos.
Back in the WebVR days the sentiment was we can never do this because security.
In my experience with standards bodies/Chrome, anything outside of the owning company's product goals is "no, because security", and then then they flip-flop on that when the browser feature becomes a dependency for something an internal team is working on (which happened many times with WebVR, now WebXR).
So I'm curious what is Google working on that has this as a dependency, now that they've 180'd on it?
This had to do with leaking privacy sensitive information such as visited links by reading back pixels from a rendered <a>. They fixed this with a concept they call "read-back-allowed rendering"
"Security" is usually not a binary, but how much money you are willing to spend to figure out a way to do it securely. It shouldn't be surprising that chrome spends more resources on things that have a business case than things that do not.
Since then I've actually met these two and had to conclude that:
- They didn't have any use-cases in mind that weren't cosmetic
- They don't quite understand how being part of an advertising company shapes how they build and conceive of web tech
- They react quite poorly to reactions that aren't cheerleading
The answer for curved elements was handwaving and "just approximate it with flat projective transforms" which tells you this is not a serious attempt to expose mature UI internals, but just another shiny epicycle on the creaky leviathan that is the HTML5 web api.
The list of features that "automatically work" rests on assumptions that are only valid for minor visual adjustments. The moment you try to use HTML-in-Canvas as a more basic building block, a lot of that falls away.
e.g. Suppose I render a piece of text and then use that as a mask for visual effects. If the browser now draws a text selection highlight into that mask, the text becomes unreadable.
I've read your blog enough times to recognize it instantly, yet it's the first time I notice that you blog header is an animated and interactive 3D demo. Very cool!
Honestly, I think the push to constantly add browser features (and thereby expand the already massive web API) is misguided. This feature, in particular, seems like it should be solved with library, rather than by further bloating the browser.
If I put on my conspiracy hat, it looks like Google is pushing to expand the web API as a means to maintain its lead in the browser market by making it harder for other browsers (i.e. Firefox) to maintain feature parity.
TBH, the first time I read about this, I had to check the date to make sure it wasn't an April Fool's joke I had missed. I'm sure there are some things this would be useful for, but the sheer amount of complexity it would add to web renderers doesn't seem commensurate with how useful it would be.
This kind of stuff just puts even more distance between established engines and smaller projects like Servo.
I have the same theory but for different reasons(commented something similar to a recent css functions thread), in this case though I don't thing you can solve it with a library, the canvas api is very limited, text layouting is very hard especially if you want to support different languages and directions. Let alone rendering a text input, if the use case is games, those usually don't have very complex inputs and that would be ok I think to implement a simpler one. IMO we should just have exposed the Skia text layout api in javascript.
You can't solve this with a library without essentially implementing a separate browser engine in JS (or WASM). Doing this directly in the browser engine is far simpler than that.
In terms of performance this approach worked out much better (able to easily handle 1000+ panels/nested panels at 60+ fps) but the complexity goes very high in the implementation of each panel. As in I cant just use for eg a markdown rendering js library or any third party ui component. And frequently make a change would result in massive performance degradation.
https://canaster.in/
Currently I have text/markdown/image/audio/video and pdf panel (you can drag and drop files)
I wonder if i should take another go at it and implement using html-in-canvas api. Afraid of the api being not maintained if they dont see benefit or usage?
Related bunch of comments from that time: https://news.ycombinator.com/item?id=48201222
What you described is the CSS painting api (part of CSS Houdini)
In my experience with standards bodies/Chrome, anything outside of the owning company's product goals is "no, because security", and then then they flip-flop on that when the browser feature becomes a dependency for something an internal team is working on (which happened many times with WebVR, now WebXR).
So I'm curious what is Google working on that has this as a dependency, now that they've 180'd on it?
Since then I've actually met these two and had to conclude that:
- They didn't have any use-cases in mind that weren't cosmetic
- They don't quite understand how being part of an advertising company shapes how they build and conceive of web tech
- They react quite poorly to reactions that aren't cheerleading
The answer for curved elements was handwaving and "just approximate it with flat projective transforms" which tells you this is not a serious attempt to expose mature UI internals, but just another shiny epicycle on the creaky leviathan that is the HTML5 web api.
The list of features that "automatically work" rests on assumptions that are only valid for minor visual adjustments. The moment you try to use HTML-in-Canvas as a more basic building block, a lot of that falls away.
e.g. Suppose I render a piece of text and then use that as a mask for visual effects. If the browser now draws a text selection highlight into that mask, the text becomes unreadable.
I've read your blog enough times to recognize it instantly, yet it's the first time I notice that you blog header is an animated and interactive 3D demo. Very cool!
If I put on my conspiracy hat, it looks like Google is pushing to expand the web API as a means to maintain its lead in the browser market by making it harder for other browsers (i.e. Firefox) to maintain feature parity.
This kind of stuff just puts even more distance between established engines and smaller projects like Servo.
The browser API has become so big because the implementers are quite tenacious, with multiple parties doing “extend” in parallel…
Here's hoping for it being DOM-in-Canvas all they way down.