How about taking another step back, and having a browser that functions as a scriptable user-customizable editable window manager like HyperCard?
Run the web browser directly on the hardware, and implement the desktop window manager with that, in a way that the whole system is a scriptable integrated development environment. Then it's easy to implement tabbed windows, pie menus, and user-editable HyperCard-like interfaces.
Here's a big step in the right direction of supporting pie menus on the desktop:
Simon Schneegans's "Kando" project aims to implement cross-platform pie menus on the desktop. Simon implemented the beautiful "Pie Fly" Gnome shell extension and WYSIWYG pie menu editor, so I am really looking forward to what he does with other desktop interfaces!
>Kando will be a pie menu for the desktop. It will be highly customizable and will allow you to create your own menus and actions. For instance, you can use it to control your music player, to open your favorite websites or to simulate shortcuts. It will be available for Windows, Linux and maybe macOS.
>Fly-Pie is an extension for GNOME Shell which lets you open marking menus via keyboard shortcuts. And — to the best of my knowledge — it is the first GNOME Shell extension with achievements!
>You can use it to launch applications, simulate hotkeys, open URLs and much more. It features a continuous learning curve which lets you gradually lift-off from a grumpie menu rookie to a snappie menu pielot. (You got it? Like pilot, but with a ). Once you opened a marking menu, you can seamlessly transition between three alternative selection modes:
>Point-and-Click: Select items by clicking on them or anywhere in the corresponding wedges.
>Marking-Mode: Select items by drawing gestures. To do this, click anywhere and drag your mouse. Pausing or making a turn selects the currently dragged item.
>Turbo-Mode: You can also "draw" gestures while holding Ctrl, Shift, or Alt without having to press your mouse button! This is especially useful when you opened the menu with a shortcut involving such a modifier.
Fly-Pie 7: GNOME Shell 40+ and a new WYSIWYG Menu Editor!:
>At Sun we experimented with implementing an X11 window manager in NeWS. We didn't have transparency at the time (1992), but we did support shaped windows!
>The NeWS window manager supported cool stuff (for both X11 and NeWS windows!) like rooms, virtual scrolling desktops, tabbed windows, pie menus, was easily extensible and deeply customisable in PostScript, and ran locally in the window server so it could respond instantly to input events, lock the input queue and provide feedback and manipulate windows immediately without causing any context switches or dealing with asynchronous locking, unlocking and event handling. You'd never lose a keystroke or click when switching between applications, for example.
[...]
And here's how I think you should design a programmable "window manager" these days -- but it would be much more than just a window manager! It would be great for integrating legacy desktop and mobile applications into VR, for example!
>Don asks Peter Korn: Hey I would love to bounce an idea off of you! I didn't realize how much work you've done in accessibility.
>There is a window manager for the Mac called Slate, that is extensible in JavaScript -- it makes a hidden WebView and uses its JS interpreter by extending it with some interfaces to the app to do window management, using the Mac Accessibility API.
>So I wanted to make pie menus for it, and thought of a good approach: make the hidden WebView not so hidden, but in the topmost layer of windows, covering all the screens, with a transparent background, that shows the desktop through anywhere you don't draw html.
>Then just make pie menus with JavaScript, which I've done. Works like a charm!
>THEN the next step I would like to do is this:
>aQuery -- like jQuery, but for selecting, querying and manipulating Mac app user interfaces via the Accessibility framework and protocols.
>So you can write jQuery-like selectors that search for and select Accessibility objects, and then it provides a convenient high level API for doing all kinds of stuff with them. So you can write higher level plugin widgets with aQuery that use HTML with jQuery, or even other types of user interfaces like voice recognition/synthesis, video tracking, augmented reality, web services, etc!
>For example, I want to click on a window and it will dynamically configure jQuery Pie Menus with the commands in the menu of a live Mac app. Or make a hypercard-like user interface builder that lets people drag buttons or commands out of Mac apps into their own stacks, and make special purpose simplified guis for controlling and integrating Mac apps.
>Does that sound crazy? I think it just might work! Implement the aQuery "selector engine" and heavy lifting in Objective C so that it runs really fast, and presents a nice high level useful interface to JavaScript.
>Soon after the invention of the movie camera, there was a "genera" of films that consisted of nothing but pointing a movie camera at a stage, and filming a play in one shot.
>That's the classic example of using a new technology to emulate an old technology, without taking advantage of the unique advantages of the new technology, before the grammar and language of film had been invented.
>Simply projecting desktop user interfaces designed for flat 2D screens and mice into VR is still in the "novelty show" age, like filming staged plays written for a theater, without any editing, shots, or film grammar.
>VR window managers are just a stop-gap backwards-compatibility bridge, while people work on inventing a grammar and language of interactive VR and AR user interfaces, and re-implement all the desktop and mobile applications from the ground up so they're not merely usable but actually enjoyable and aesthetically pleasing to use in VR.
>The current definition of "window manager," especially as it applies to X-Windows desktops, tightly constrains how we think and what we expect of user interface and application design. We need something much more flexible and extensible. Unfortunately X-Windows decades ago rejected the crucially important ideas behind NeWS and AJAX, that the window manager should be open-ended and dynamically extensible with downloadable code, which is the key to making efficient, deeply integrated user interfaces.
>For example, the "Dragon Naturally Speaking" speech synthesis and recognition system has "dragonfly", a Python-based "speech manager" that is capable of hooking into existing unmodified desktop applications, and scripting custom speech based user interfaces.
>Another more ambitious example is Morgan Dixon's work on Prefab, that screen-scrapes the pixels of desktop apps, and uses pattern recognition and composition to remix and modify them. This is like cinematographers finally discovering they can edit films, cut and splice shots together, overlay text and graphics and pictures-in-pictures and adjacent frames. But Prefab isn't built around a scripting language like dragonfly, NeWS or AJAX.
>Here's some stuff I've written about the direction that user interfaces should take to move beyond the antique notion of "window managers", and enables much deeper integration and accessibility and alternative input and output methods.
>Glad to see people are still making better window managers! [...] I think extensibility and accessibility are extremely important for window managers. [...] I'd like to take that idea a lot further, so I wrote up some ideas about programming window management, accessibility, screen scraping, pattern recognition and automation in JavaScript. [...] Check out Morgan Dixon's and James Fogarty's amazing work on user interface customization with Prefab, about which they've published several excellent CHI papers: [...]
>Imagine if every interface was open source. Any of us could modify the software we use every day. Unfortunately, we don't have the source.
>Prefab realizes this vision using only the pixels of everyday interfaces. This video shows the use of Prefab to add new functionality to Adobe Photoshop, Apple iTunes, and Microsoft Windows Media Player. Prefab represents a new approach to deploying HCI research in everyday software, and is also the first step toward a future where anybody can modify any interface.
>Here are some other interesting things related to scriptable window management and accessibility to check out: aQuery -- Like jQuery for Accessibility
>It would also be great to flesh out the accessibility and speech recognition APIs, and make it possible to write all kinds of intelligent application automation and integration scripts, bots, with nice HTML user interfaces in JavaScript. Take a look at what Dragon Naturally Speaking has done with Python:
>I would like to discuss how we could integrate Prefab with a Javascriptable, extensible API like aQuery, so you could write "selectors" that used prefab's pattern recognition techniques, bind those to JavaScript event handlers, and write high level widgets on top of that in JavaScript, and implement the graphical overlays and gui enhancements in HTML/Canvas/etc like I've done with Slate and the WebView overlay.
Run the web browser directly on the hardware, and implement the desktop window manager with that, in a way that the whole system is a scriptable integrated development environment. Then it's easy to implement tabbed windows, pie menus, and user-editable HyperCard-like interfaces.
Here's a big step in the right direction of supporting pie menus on the desktop:
Simon Schneegans's "Kando" project aims to implement cross-platform pie menus on the desktop. Simon implemented the beautiful "Pie Fly" Gnome shell extension and WYSIWYG pie menu editor, so I am really looking forward to what he does with other desktop interfaces!
Introducing: Kando:
https://ko-fi.com/post/Introducing-Ken-Do-L3L7L0FQ2
https://github.com/kando-menu/kando
>Kando will be a pie menu for the desktop. It will be highly customizable and will allow you to create your own menus and actions. For instance, you can use it to control your music player, to open your favorite websites or to simulate shortcuts. It will be available for Windows, Linux and maybe macOS.
Fly-Pie:
https://github.com/Schneegans/Fly-Pie
>Fly-Pie is an extension for GNOME Shell which lets you open marking menus via keyboard shortcuts. And — to the best of my knowledge — it is the first GNOME Shell extension with achievements!
>You can use it to launch applications, simulate hotkeys, open URLs and much more. It features a continuous learning curve which lets you gradually lift-off from a grumpie menu rookie to a snappie menu pielot. (You got it? Like pilot, but with a ). Once you opened a marking menu, you can seamlessly transition between three alternative selection modes:
>Point-and-Click: Select items by clicking on them or anywhere in the corresponding wedges.
>Marking-Mode: Select items by drawing gestures. To do this, click anywhere and drag your mouse. Pausing or making a turn selects the currently dragged item.
>Turbo-Mode: You can also "draw" gestures while holding Ctrl, Shift, or Alt without having to press your mouse button! This is especially useful when you opened the menu with a shortcut involving such a modifier.
Fly-Pie 7: GNOME Shell 40+ and a new WYSIWYG Menu Editor!:
https://www.youtube.com/watch?v=sRT3O9-H5Xs
Fly-Pie 10: A new Clipboard Menu, proper touch support & much more!:
https://www.youtube.com/watch?v=BGXtckqhEIk
I've written about reinventing scriptable HyperCard-like window managers with pie menus and tabbed windows before:
SimCity, Cellular Automata, and Happy Tool for HyperLook (nee HyperNeWS (nee GoodNeWS)):
https://donhopkins.medium.com/hyperlook-nee-hypernews-nee-go...
https://news.ycombinator.com/item?id=13817649
>At Sun we experimented with implementing an X11 window manager in NeWS. We didn't have transparency at the time (1992), but we did support shaped windows!
>The NeWS window manager supported cool stuff (for both X11 and NeWS windows!) like rooms, virtual scrolling desktops, tabbed windows, pie menus, was easily extensible and deeply customisable in PostScript, and ran locally in the window server so it could respond instantly to input events, lock the input queue and provide feedback and manipulate windows immediately without causing any context switches or dealing with asynchronous locking, unlocking and event handling. You'd never lose a keystroke or click when switching between applications, for example.
[...]
And here's how I think you should design a programmable "window manager" these days -- but it would be much more than just a window manager! It would be great for integrating legacy desktop and mobile applications into VR, for example!
>aQuery -- Like jQuery for Accessibility
http://donhopkins.com/mediawiki/index.php/AQuery
>Don asks Peter Korn: Hey I would love to bounce an idea off of you! I didn't realize how much work you've done in accessibility.
>There is a window manager for the Mac called Slate, that is extensible in JavaScript -- it makes a hidden WebView and uses its JS interpreter by extending it with some interfaces to the app to do window management, using the Mac Accessibility API.
>So I wanted to make pie menus for it, and thought of a good approach: make the hidden WebView not so hidden, but in the topmost layer of windows, covering all the screens, with a transparent background, that shows the desktop through anywhere you don't draw html.
>Then just make pie menus with JavaScript, which I've done. Works like a charm!
>THEN the next step I would like to do is this:
>aQuery -- like jQuery, but for selecting, querying and manipulating Mac app user interfaces via the Accessibility framework and protocols.
>So you can write jQuery-like selectors that search for and select Accessibility objects, and then it provides a convenient high level API for doing all kinds of stuff with them. So you can write higher level plugin widgets with aQuery that use HTML with jQuery, or even other types of user interfaces like voice recognition/synthesis, video tracking, augmented reality, web services, etc!
>For example, I want to click on a window and it will dynamically configure jQuery Pie Menus with the commands in the menu of a live Mac app. Or make a hypercard-like user interface builder that lets people drag buttons or commands out of Mac apps into their own stacks, and make special purpose simplified guis for controlling and integrating Mac apps.
>Does that sound crazy? I think it just might work! Implement the aQuery "selector engine" and heavy lifting in Objective C so that it runs really fast, and presents a nice high level useful interface to JavaScript.
[...]
https://news.ycombinator.com/item?id=29105919
>>Would you mind to share your opinion on Wayland with us?
>Thanks for asking! Hold my bong. ;) [lots of opinions omitted, see link above...]
https://news.ycombinator.com/item?id=22829690
>>Simula: A VR window manager for Linux [...]
>Soon after the invention of the movie camera, there was a "genera" of films that consisted of nothing but pointing a movie camera at a stage, and filming a play in one shot.
>That's the classic example of using a new technology to emulate an old technology, without taking advantage of the unique advantages of the new technology, before the grammar and language of film had been invented.
>Simply projecting desktop user interfaces designed for flat 2D screens and mice into VR is still in the "novelty show" age, like filming staged plays written for a theater, without any editing, shots, or film grammar.
>VR window managers are just a stop-gap backwards-compatibility bridge, while people work on inventing a grammar and language of interactive VR and AR user interfaces, and re-implement all the desktop and mobile applications from the ground up so they're not merely usable but actually enjoyable and aesthetically pleasing to use in VR.
>The current definition of "window manager," especially as it applies to X-Windows desktops, tightly constrains how we think and what we expect of user interface and application design. We need something much more flexible and extensible. Unfortunately X-Windows decades ago rejected the crucially important ideas behind NeWS and AJAX, that the window manager should be open-ended and dynamically extensible with downloadable code, which is the key to making efficient, deeply integrated user interfaces.
>For example, the "Dragon Naturally Speaking" speech synthesis and recognition system has "dragonfly", a Python-based "speech manager" that is capable of hooking into existing unmodified desktop applications, and scripting custom speech based user interfaces.
>Another more ambitious example is Morgan Dixon's work on Prefab, that screen-scrapes the pixels of desktop apps, and uses pattern recognition and composition to remix and modify them. This is like cinematographers finally discovering they can edit films, cut and splice shots together, overlay text and graphics and pictures-in-pictures and adjacent frames. But Prefab isn't built around a scripting language like dragonfly, NeWS or AJAX.
>Here's some stuff I've written about the direction that user interfaces should take to move beyond the antique notion of "window managers", and enables much deeper integration and accessibility and alternative input and output methods.
https://news.ycombinator.com/item?id=14182061
>Glad to see people are still making better window managers! [...] I think extensibility and accessibility are extremely important for window managers. [...] I'd like to take that idea a lot further, so I wrote up some ideas about programming window management, accessibility, screen scraping, pattern recognition and automation in JavaScript. [...] Check out Morgan Dixon's and James Fogarty's amazing work on user interface customization with Prefab, about which they've published several excellent CHI papers: [...]
>Imagine if every interface was open source. Any of us could modify the software we use every day. Unfortunately, we don't have the source.
>Prefab realizes this vision using only the pixels of everyday interfaces. This video shows the use of Prefab to add new functionality to Adobe Photoshop, Apple iTunes, and Microsoft Windows Media Player. Prefab represents a new approach to deploying HCI research in everyday software, and is also the first step toward a future where anybody can modify any interface.
https://news.ycombinator.com/item?id=18797818
>Here are some other interesting things related to scriptable window management and accessibility to check out: aQuery -- Like jQuery for Accessibility
https://web.archive.org/web/20180826132551/http://donhopkins...
>It would also be great to flesh out the accessibility and speech recognition APIs, and make it possible to write all kinds of intelligent application automation and integration scripts, bots, with nice HTML user interfaces in JavaScript. Take a look at what Dragon Naturally Speaking has done with Python:
https://github.com/t4ngo/dragonfly
>Morgan Dixon's work with Prefab is brilliant.
>I would like to discuss how we could integrate Prefab with a Javascriptable, extensible API like aQuery, so you could write "selectors" that used prefab's pattern recognition techniques, bind those to JavaScript event handlers, and write high level widgets on top of that in JavaScript, and implement the graphical overlays and gui enhancements in HTML/Canvas/etc like I've done with Slate and the WebView overlay.