Nyxt Browser Development - A Technical Perspective

By John Mercouris

Recently we had an enlightening discussion on one of our chat servers where we talked about the status of the macOS port, how things are progressing with Nyxt, why we've made some fundamental decisions, and more.

After the discussion, we realized that the information could be valuable for others, so we've summarized it here in a Q/A format style document.

Q: What is the current status of Nyxt for macOS?

Nyxt is currently in development for macOS with efforts focused on making it more stable and reliable.

Q: I thought Nyxt was supposed to be renderer agnostic, why is the macOS port taking so long?

jmercouris: it is in fact surprising it has taken so long, and it is because of our lack of fastidiousness
jmercouris: we spent a lot of time ensuring that it would be cross platform compatible, but what happened was we had only really ONE platform that we were working on
jmercouris: and we ended up making a lot of changes that didn't consider whether they would work on /all/ platforms
jmercouris: so when we had a macOS/Cocoa port AND WebKitGTK+ things with regards to portability were great, but then we sunsetted the original macOS port due to limitations in its library/implementation
jmercouris: that was the decline of the foreign (cross platform) interface I had worked very hard to design and preserve
jmercouris: regardless, it is not as if the design has borne no fruits, it would be infinitely harder, if not impossible to port Nyxt if we didn't have this interface

Q: Why not use CEF or Chrome directly? Why Electron?

jmercouris: Another quite complicated question :-D
jmercouris: Well, the chrome renderer is used, wrapped by electron
jmercouris: we aren't using chrome directly because it is very difficult to build this toolchain
jmercouris: we were going to use cef https://cef-builds.spotifycdn.com/docs/122.1/index.html
jmercouris: we again experienced a lot of challenges writing bindings for it
jmercouris: that is not to say it is impossible, in fact it is going to be significantly easier to port CEF after this new port
user0: interesting that it's so difficult. I feel like in expectation a lot of things are much simpler than they end up being... considering there are so many chromium based browsers out there, I would've thought it would be easy, but then again, I imagine they leave it without an easy quickstart
jmercouris: yeah, I wish it were simpler, but it isn't
jmercouris: so that's the thing
jmercouris: it is EASY to make a chromium browser
jmercouris: you just take chromium, fork it, make some flag changes and compile it
user0: ah, but not to modify the browser itself with CL?
jmercouris: yes

Q: Will extensions be available in the new port?

Yes, regular web extensions will be loadable in the Electron port of Nyxt.

Q: Is the FFI you made nyxt specific or will it be a standalone project?

jmercouris: it is a standalone project
jmercouris: that is why I linked to it: https://github.com/atlas-engineer/cl-electron/
user0: that's cool
jmercouris: you can use it to make arbitrary programs in Lisp that connect to electron
user0: ah, yeah, I didn't realize that's what you were talking about
user0: I've seen it

Q: What challenges are faced in using WebKitGTK+ on macOS?

WebKitGTK+ is notably slow on macOS and lacks full support for macOS APIs, leading to performance issues.

Q: Why is there a need for both WebKitGTK+ and Electron ports of Nyxt?

Nyxt aims to be renderer agnostic to adapt to evolving web engine technologies and maintain flexibility.

user0: Is webkitgtk+ not portable as well?
user0: Why do we need a specific electron port?
user0: or more generally what UI library is being used in linux that is not portable to mac?
user0: it sounds like everything is depending on electron, but I would've thought the current UI library was also meant to be portable
jmercouris: WebKitGTK+ is portable on macOS
jmercouris: it is just that the WebKitGTK+ port is very slow on macOS, and we need another renderer anyways
jmercouris: also the WebKitGTK+ team does not prioritize support for macOS
jmercouris: ostensibly we could add the support, but that would be a full-time job in itself
jmercouris: so the UI library is GTK
user0: ah WebKitGTK+ in general is very slow on macOS or because of they way Nyxt is coded?
jmercouris: also not really portable on Windows
jmercouris: WebKitGTK+ is generally very slow on macOS
jmercouris: it doesn't use any of the macOS APIs

Q: Will you deprecate WebKitGTK+ when you release the Electron port?

user0: Why is there a WebKitGTK+ team at all then? Sorry if this is beating around the bush. It sounds like all their work will be replaced by the electron app
user0: Meaning once the electron based app is out, it should be portable accross all OSs, so then what's the point of having another UI port with WebKitGTK+?
user0: Is it because that's the current working version so new features are added there, but really, once electron is out, it will be deprecated?
jmercouris: it will NOT be replaced
jmercouris: we will support both WebkitGTK+ and Electron versions of Nyxt
user0: ah ok. What's the benefit of a WebKitGTK+ version? In case Electron drops support for linux?
user0: Meaning, it will essentially be the same exact application. correct?
jmercouris: that is not the point
jmercouris: the point is to maintain Nyxt as being renderer agnostic
jmercouris: web engines always change, technologies change
jmercouris: we want to be resistant to breakages
user0: ah ok, so that Nyxt will not be dependent on a specific renderer, to decouple the code, and allow for flexibility in the future
jmercouris: yes, that is exactly it

Q: I thought Safari used WebKit and Safari is fast. Why isn't Nyxt fast on macOS?

jmercouris: in fact Safari does use WebKit
jmercouris: but it does not use WebKitGTK+, that is a specific WebKit Port
jmercouris: the port that Safari uses is not available with all of the APIs to regular developers
user0: Ah interesting, it's not open source is what I'm understanding from what you are saying
jmercouris: that is correct
jmercouris: so when you write Swift/Objective-C on a Mac you have access to a "WKWebview" however this is a VERY limited API
jmercouris: you can't manage things like cookies or make a proper browser out of it
jmercouris: they reserve these private APIs for their own usage
jmercouris: they don't want you making browsers that will compete with Safari, presumably
user0: So basically the source of the problem is the compatibility between the original WebKit and the GTK UI Library, since WebKitGTK+ uses a port, and not the original WebKit, hence all the issues, since the port is made specifically for Gnome and not macOS.
jmercouris: Close enough
jmercouris: WebKitGTK+ is a port of WebKit
jmercouris: WebKit is just an engine that calculates the rendering of a page given some HTML
jmercouris: GTK is responsible for actually drawing that page
jmercouris: GTK has poor performance on macOS
user0: Ah ok, I think I now finally got it. Thanks, sorry if that took a lot to explain
jmercouris: No problem, it is a convoluted topic

Q: How does Nyxt ensure compatibility across different platforms?

Nyxt's foreign interface API allows easy porting to different rendering engines like WebKitGTK+ and Electron.

Q: Is the Gecko engine used in Firefox portable?

The Gecko engine is tightly coupled with Firefox, making it challenging to use in other applications outside the Firefox context.

Q: What about Quantum?

jmercouris: Quantum is basically Gecko evolved
jmercouris: they developed another project called "Servo"
jmercouris: which was a pure Rust based, parallelized engine
jmercouris: it effectively functioned as a testbed for new technologies/methods
jmercouris: they then took these ideas and incorporated them into Gecko, making Quantum
jmercouris: they didn't take all of the ideas, just the ones that worked
user1: And Servo itself is just a tiny little bit above "completely unusable".
jmercouris: that's exactly correct
jmercouris: I did try using it some years ago, and it couldn't quite work
user1: Same experience. I forked that Qutebrowser and tried to slap Servo on and it worked really well except for the part where Servo (at that time) couldn't even render the acid test. 
user0: Ah, so Gecko, even though it's a rendering engine and officially a separate project, is really tightly coupled to firefox to a point where it's hard to actually use it in any other application
jmercouris: that's correct
jmercouris: I wouldn't even call it "officially" a separate project
jmercouris: that is just what they call their engine
user1: Gecko lives in the Firefox monorepo and is completely unuseable outside of the Firefox context, thus you get FF spinoffs but no one has a FF version of Electron and never will.
jmercouris: this is precisely what we are trying to avoid with Nyxt
user0: yeah, even their docs on embedding it are empty: https://wiki.mozilla.org/Gecko:Overview#Embedding

Q: What about writing a universal API for Nyxt?

user1: Had you considered writing some sort of universal API for Nyxt and then writing three mappings for that API to communicate with the OS native WebKit variants?
jmercouris: that is exactly what we did
jmercouris: it is what we call the "foreign-interface"
jmercouris: it is our API for creating a new port of Nyxt
jmercouris: https://github.com/atlas-engineer/nyxt/blob/master/source/foreign-interface.lisp
jmercouris: If you implement those methods, you can port Nyxt to whatever engine you want

Conclusion

Nyxt's development journey highlights the importance of maintaining flexibility in adapting to changing web engine technologies and the challenges in ensuring cross-platform compatibility.