Back in December of 2007, Simon Willison came up with a very neat Comet demo written with just a few lines of Dojo code. It’s a presentation/slideshow app where a master client page, controlling the slideshow, pushes the URLs of slide images out to subscribed clients over a Comet channel. The idea is that subscribers can view a conference presentation on their laptops, instead of the big screen—or even from another room, or over the internet.
In Simon’s implementation, the slides are simply a series of JPEGs, hosted on the Comet server. I’ve been playing around with HTML Canvas recently, and began to wonder if I could use it for something more sophisticated: is it possible to push arbitrary web content between browsers as a dynamically generated image?
The answer is ‘Yes’. And also ‘No’.
The ‘Yes’ is thanks to two canvas methods that do nearly all of the necessary work. The first, 2dContext.drawWindow() renders an arbitrary part of the browser window into a canvas, which allows you to effectively screengrab the browser from JavaScript. Following that, canvas.toDataURL() outputs a representation of the canvas as a PNG or JPEG using the data: URL scheme, which can be wrapped in JSON and pushed to subscribed clients for display.
However, the ‘No’ is due to drawWindow being a non-standard, Mozilla-only addition to the canvas API. Furthermore, security restrictions—designed to prevent bad guys from stealing the content of your screen—mean that the drawWindow function is only available to the browser itself, or to browser add-ons.
So I made a proof-of-concept Firefox extension called ‘Screenpush’. Here’s a demo:
The Screenpush Extension
The Screenpush UI is just an icon in the Firefox status bar (which reflects connection status), and a context-popup menu. There are two methods of publishing your screen to subscribing clients. You can manually invoke a screenpush by double-clicking the status bar icon (equivalent to “Push Current Screen” in the popup menu), or you can check the “Auto-Push Screen” menu item. In auto-push mode, Screenpush will publish a grab whenever a new page is loaded, or the content of a page changes, whenever you click or type in a page, and whenever you scroll the page, resize the browser or switch tabs. This mode works well for broadcasting an S5 slideshow, or demoing navigation through a website.
When auto-push is active, the “Track Mouse Pointer” option is available. When you enable this setting, Screenpush is triggered by mouse movements, tracking your cursor position with a fake mouse pointer drawn onto the published image (it even has a nice drop shadow in FF3.1). In this mode, Screenpush works like a rudimentary streaming screencast. It will react to mouseover JavaScript menus, CSS hover effects etc., but the downside is the generation of much more Comet traffic, so this mode may not perform too well across the internet.
Extension Settings
The Screenpush extension has a simple options screen (accessed via the pop-up menu, or from the Firefox Add-Ons dialog), where you can configure the Cometd service location and Bayeux channel to use, as well as selecting PNG or JPEG output, and JPEG quality. PNG looks nicest for simple pages, but scales badly for anything detailed or photographic. For instance, Google Maps screens in Satellite view can render to PNGs larger than a megabyte, which makes the Comet push pretty sluggish. Using JPEGs, with quality set around 70, seems to be the best compromise between image quality and push responsiveness.
There’s also one Screenpush setting that isn’t exposed in the UI. It’s accessible through the about:config screen, under the “extensions.screenpush.pushDelay” key. This value is the lag in milliseconds between an event firing and a screengrab being taken and published, and acts as a throttle on the frequency of screen pushes, and hence on Comet traffic. The lag also allows a few moments for a rapid sequence of events to conclude, such as those resulting from a mouse move, window scroll or a DOM modification cascade. You might want to experiment with this setting to make Screenpush a bit snappier.
The Subscriber Client
The Screenpush subscriber is just a static HTML page and a bit of JavaScript. Enter your Cometd service URL and channel (the default is /screenpush/demo, same as in the Extension), and hit Connect. Once you get the “Connected” message, the page will auto-update as the Screenpush extension publishes images.
The subscriber works in most modern browsers, with some minor caveats (see below). Since Cometd can use JSONP for cross-domain communication, the subscriber page can be hosted anywhere, even loaded straight from the filesystem. The page just needs to have its Dojo libs in the same directory.
As with Simon’s demo, the code for the subscriber is mostly trivially simple: a few lines of JavaScript to receive a data: URL via Comet and set it as an image source.
var receive = function(comet) {
var img = dojo.byId("screenpush-image");
img.src = comet.data.dataUrl;
}
dojox.cometd.subscribe(cometd.c,receive);
A Note on Browser Compatibility
The Screenpush subscriber works very well in Firefox and Chrome, and mostly works in Safari, Camino and Opera. IE7 is a non-starter as there it has no support for the data: URL scheme. I haven’t bothered to try IE8.
Safari and Opera appear to have problems maintaining a long-lived cross-domain Comet connection, eventually leading to disconnection. You can avoid this issue by hosting your Screenpush subscriber page on the same host and port as the Cometd service.
Camino just has a minor issue where it refuses to display certain data: images, meaning that the Screenpush subscriber will intermittently display empty images.
The Cometd Server
There’s no need for any Screenpush-specific server side code, you just need a compliant Bayeux/Cometd server. I downloaded Jetty7, which runs a Cometd service out of the box at http://<yourhost>:8080/cometd/cometd. You may need to tweak the maximum form post size allowed by your server, especially if using PNG compression (see Extension Settings). If you start to see the Screenpush error icon when you push a large or detailed screen, this will be the problem. I raised the default maximum form size on Jetty from 200Kb to 10Mb by adding this to etc/jetty.xml:
<Call class="java.lang.System" name="setProperty"> <Arg>org.mortbay.jetty.Request.maxFormContentSize</Arg> <Arg>10000000</Arg> </Call>
Miscellaneous Technical Detail
The core of the Screenpush extension is little more than the drawWindow/toDataURL calls I mentioned earlier, along with code to push out the generated data URL over Comet. The window content gets rendered to a hidden canvas element embedded in the statusbar next to the Screenpush icon. Here is a simplified version of the actual push code:
// Get a reference to the window object
var win = gBrowser.selectedBrowser.contentWindow;
// Get the element that knows window dimensions and scroll info
var clientInfo = win.document.compatMode == "CSS1Compat" ?
win.document.documentElement :
win.document.body;
var region = { x: win.scrollX,
y: win.scrollY,
width: clientInfo.clientWidth,
height: clientInfo.clientHeight };
// Size the canvas
canvas.style.maxwidth = region.width + "px";
canvas.style.maxheight = region.height + "px";
canvas.width = region.width;
canvas.height = region.height;
// Render to the canvas
var ctx = canvas.getContext("2d");
ctx.clearRect(region.x, region.y, region.width, region.height);
ctx.drawWindow(window, region.x, region.y,
region.width, region.height, "#000");
// Push out the rendered image
var dataUrl = canvas.toDataURL();
jQuery.comet.publish(cometdChannel, { dataUrl: dataUrl });
I’m using a hacked version of the jQuery Comet plugin to connect to the Comet service and push out image data (I don’t know if using jQuery within an extension is a great idea, but I’m not aware of any standalone Cometd client libraries at present). I had to prevent jQuery interpreting the Ajax requests made from browser chrome as cross-domain request, and switching to JSONP mode. JSONP is a non-starter inside an extension, as you can’t load external scripts from chrome—it would be a huge security hole.
Tracking Cometd connection state is also slightly tough. The Screenpush subscriber client actually does a better job of this than the extension itself, as the Dojo Comet library it uses supports client-side meta events that you can subscribe to for information about connection attempts. I had to hack my own rough interpretation of the client-side /cometd/meta/ channel into jQuery Comet. This feeds back information about connection attempts to the UI, where problems are reflected by a change to the Screenpush statusbar icon. The main outstanding problem is that since Cometd connections are long-lived, they don’t timeout easily, so it can take a long time to learn that there’s a connection issue.
The other significant part of the extension code is event subscription wrangling. Screenpush listens on the DOMAttrModified, DOMNodeInserted and DOMNodeDeleted events, and the problem is that dynamic pages tend to generate great cascades of these events when updating the UI. This also happens with scroll, mousemove, and resize events. Obviously, it would be very inefficient to push out a screengrab every time one of these events fires, so my low-tech solution is the pushDelay timer I described in Extension Settings. To make things more efficient, I also cache the last-published data URL, and use it to see if the screen has actually changed since the last push (with Track Pointer switched on, the composited mouse pointer is itself part of the rendered image, so moving the mouse in this mode will correctly push out a new image with the pointer position updated).
Try It Out
If you want to try out Screenpush (and report bugs to me), then first of all grab the Screenpush extension XPI . You can download Jetty7 from Codehaus—I used the latest pre-release. The Screenpush subscriber webpage and files are zipped up in screenpush-subscriber.zip. Or you can just browse to the subscriber page hosted on this site, but take note of the cross-domain Comet issue with certain browsers.
And let me know if you find any practical use for Screenpush—I’ll tidy it up and release it properly if anybody wants to use it.
