Before settling on the project to implement a full accessibility cache in Firefox, I investigated several other alternatives. In my last post, I discussed asynchronous accessibility APIs. Another alternative I considered is switching to UI Automation and having the UIA tree accessed directly in web content processes, rather than communicating via the main UI process. While this is not possible with IAccessible2 because of browser sandboxes, it could theoretically be possible with UIA because it has an intermediary Windows component called UIAutomationCore which sits between the client and the server. As an operating system component, there are less concerns with regard to allowing it in the browser sandbox. Unfortunately, this proved to be infeasible then and I think it is still infeasible now: First and foremost, UIA would need to be enhanced to support this. UIA calls would need to arrive on a background worker thread. Providers can specify ProviderOptions_UseComThreading to ensure t

Accessibility API queries are generally synchronous: each query blocks both the client and the server until the query completes and returns its result. This causes significant challenges for modern, multi-process web browsers, requiring them to cache the accessibility trees from all other processes in the main UI process. This raises the question: why not make accessibility APIs asynchronous? As usual, there is a theoretical/principle answer and a pragmatic answer. Let’s start with theory. For a browser, synchronous accessibility APIs are a real problem. So much of the complexity in multi-process browser accessibility architecture comes back to the need to support synchronous APIs. If they were async, we might not need to maintain a cache in the main process at all, which would save us a lot of pain, complexity and performance problems. From that perspective, async APIs would be great. Async APIs would likely also help assistive technology products to avoid hangs due to queries taking

UI Automation (UIA) is Microsoft’s recommended accessibility framework for Windows, replacing the earlier Microsoft Active Accessibility (MSAA) framework. Despite this, to access web content, screen readers such as NVDA and JAWS continue to use IAccessible2, an open source API based on MSAA. This is not just because of the significant effort involved in switching to a different API, although that is certainly a factor. More importantly, it is because UIA is currently insufficient as an accessibility API for the web. Here are some of the reasons: UIA relies on LocalizedControlType to distinguish many controls, rather than adding more proper semantic ControlTypes. This means that aside from using a different message to report the control, clients can’t reliably identify the types of these controls in order to choose different behaviour for them. This also extends to landmark types that UIA doesn’t support. For these, LandmarkType is specified as custom and the client can only use Localiz

Sometimes, it is necessary to move a row up or down in a Google Sheet. For example, I might be maintaining a list of tasks ordered from highest to lowest priority and realise that a task later in the sheet is actually higher priority than earlier tasks. This Google support article says you can do this by selecting the row and then choosing Move row up (or down) from the Edit menu. However, when I selected the cells in the row, these Move row items still didn’t appear in the Edit menu. The solution is to press shift+space twice to select the row, after which the Move sub-menu will appear in the Edit menu, from which you can choose Row up or Row down. Shift+space is the keyboard shortcut to select a row, but the first press only selects the cells in the row. To select the row itself, which you must do in order to move it, you need to press it again. Putting it all together, the entire keyboard sequence to move a row up in Firefox on Windows is shift+space, shift+space, alt+shift+e, m, k.

Everyone loves paying bills, right? One of the best parts about paying bills is surely that they usually provide all the amounts, billing codes, reference numbers, etc. with punctuation and spaces; e.g. amount $1,234, reference code 123 456-789. But banking and payment portals - sometimes even the payment portal for the organisation issuing the bill! - won’t accept spaces and punctuation! This means you can’t simply copy and paste the numbers, which is really annoying. Wouldn’t it be awesom if there were some easy, automatic way to strip out everything except numbers and the decimal point before pasting? I recently started using Ditto, an absolutely superb clipboard manager for Windows. It turns out you can do this with Ditto using a paste script. I originally thought that paste scripts always ran, which I wouldn’t want in this case, but it turns out that they actually appear as items in the Special Paste menu. To set this up, go to Ditto -> Options -> General -> Advanced -&gt

The Firefox accessibility engine is responsible for providing assistive technologies like screen readers with the information they need to access web page content. For the past couple of years, the Firefox accessibility team have been working on a major re-architecture of the accessibility engine to significantly improve its speed, reliability and maintainability. We call this project “Cache the World”. In this post, I explain the reasons for such a massive undertaking and describe how the new architecture solves these issues. The need for speed The biggest motivation for this project is to make Firefox faster when used with screen readers and other assistive technologies, particularly on Windows. Let’s start by taking a look at some numbers. The table below provides the approximate time taken to perform various tasks with Firefox and the NVDA screen reader, both before and after this re-architecture. Before (no cache) After (with cache) Load nsCSSFrameConstructor.cpp on Searchfox, whi

I recently got a new laptop: Dell XPS15 9510. While this is a pretty nice machine overall, its audio drivers are an abomination. Among other things, the Waves MaxxAudio software it ships with eventually leaks all of your system memory if you use audio constantly for hours, which is the case for screen reader users. I eventually got fed up and disabled the Waves crap, but this makes it impossible for me to use the headset mic on my Earpods. To work around that, I bought an Apple USB-C to 3.5-mm Headphone Jack Adapter. As well as supporting the mic on the Earpods, this adapter also supports the volume and play/pause buttons! However, play/pause only plays or pauses. In contrast, on the iPhone, pressing it twice skips to the next track and pressing it thrice skips to the previous track. I discovered that these buttons simply get sent as media key presses. So, I wrote a little AutoHotkey script to intercept the play/pause button and translate double and triple presses into next track and p

I’m finally learning to cook some decent food, so I need to be able to read recipes. For a while, I was reading them out of Simplenote on my iPhone. However, I encountered several frustrations with this approach (and this applies to any notes or text app really): When you’re not editing, Simplenote shows the note such that each line is an item for VoiceOver; i.e. you flick right to read the next line. However, if you bump the screen or perform the wrong gesture accidentally, you can easily lose your position. If the screen locks or you have to switch apps, you lose your position completely, since VoiceOver doesn’t restore focus to the last focused item in apps. When editing, you can review the note line by line using the rotor. The advantage here is that the cursor doesn’t get lost when you switch apps or the screen locks. However, lines can be smaller than is ideal due to the screen size, so one recipe instruction might get split across multiple lines. Also, moving the editing cursor

Introduction An awesome feature in Firefox that has existed forever is the ability to assign keywords to bookmarks. For example, I could assign the word “bank” to take me directly to the login page for my bank. Then, all I have to do is type “bank” into the address bar and press enter, and I’m there. Another awesome feature in Firefox is the ability to use the address bar to switch to open tabs. For example, if I want to switch to my Twitter tab, I can type “% twitter” into the address bar, then press down arrow and enter, and I’m there. Inspired by these two features, I started to wonder: what if you could have tab keywords to quickly switch to tabs you use a lot? If you only have 8 tabs you use a lot, you can switch to the first 8 tabs with control+1 through control+8. If you have more than that, you can search them with the address bar, but that gets messy if you have multiple pages with similar titles or a page title doesn’t contain keywords that are quick to search. For example, i

At CSUN this year, I attended the open source math accessibility sprint co-hosted by the Shuttleworth Foundation and Benetech, where major players in the field gathered to discuss and hack on various aspects of open source math accessibility. My team, which also included Kathi Fletcher, Volker Sorge and Derek Riemer, tackled reading of mainstream math with open source tools.Last year, NVDA introduced support for reading and interactive navigation of math content in web browsers and in Microsoft Word and PowerPoint. To facilitate this, NVDA uses MathPlayer 4 from Design Science. While MathPlayer is a great, free solution that is already helping many users, it is closed source, proprietary software, which severely limits its future potential. Thus, there is a great need for a fully open source alternative.Some time ago, Volker Sorge implemented support for math in ChromeVox and later forked this into a separate project called Speech Rule Engine (SRE). There were two major pieces to our t