Overhaul HTML syntax to specify backends

[LeaVerou]

Problem

It just occurred to me that we have a bit of an API smell: we have three different attributes doing similar but slightly different things:

• mv-storage handles saving, and also loading iff there is no mv-source
• mv-source overrides mv-storage for reading data
• mv-init defines a fallback data source to load, if mv-storage/mv-source is empty
• mv-uploads overrides mv-storage for uploads

Because each of these can take metadata of its own, we also have a ton of mv-storage-*, mv-source-*, mv-init-* attributes as well. And yet, not all use cases are covered anyway, e.g.

• it's not possible to use another Mavo app as source/storage (without using an intermediate element)
• it is not possible to save to multiple places
• It is not possible to combine data from multiple sources
• You cannot have multiple fallbacks
• You cannot route uploads of different types to different backends (e.g. images to imgur, videos to YouTuve, PDFs to your server)

Plus there is a lot of ad-hoc-ness:

• Which login UI to show? Currently we only show login UI for the "primary backend", which is in order of priority, storage OR source OR init. This also introduces a UI question of how to display multiple login UIs and how to make it clear what each is for.
• mv-autosave currently applies to mv-storage, but it's not mv-storage-autosave

Ideation

Decomposing mv-storage/mv-source/mv-init/mv-uploads and the use cases above:

• We sometimes need different backends for different purposes (read, write, uploads)
• Uploads may need to use a separate backend by type
• Reading data may need to use fallback(s). This is far more common when using the same backend for storage, and the file simply doesn't exist yet. Writing data doesn't require fallbacks.
• It would be nice to support writes to multiple backends, but not essential

Solution

Custom element for backends

In Madata, we introduced a <ma-data> component and a <madata-auth> component. While these would probably not be appropriate to adopt as-is (unless we evolve them significantly), I do think custom elements are the way to go here: using an element per backend, and having the Mavo app reference them to declare what they’re for (the association would happen implicitly if only one backend is specified).

Custom elements allow a lot of flexibility, and encapsulate a lot of complexity. For example, we could even do things like:

• If a backend doesn't require expressions, it could even be defined outside the Mavo app, and have multiple apps refer to it. Or even define a backend within a Mavo app that uses expressions from it, and have another Mavo app refer to that backend!
• We could load secondary data from multiple data sources and reference them via dot notation (today this requires a whole separate Mavo app)

It may even be easier to introduce this and migrate to Madata in one go as it abstracts a lot of stuff that now lives scattered around the Mavo codebase. It also means we don't need to use an IIFE build of Madata, since we'll be using the component, which can use ESM just fine.

I’m hoping the Mavo version could simply be a subclass of a generic component, rather than a whole parallel implementation.

• Element name ideas (in order of preference): <mv-backend>, <mv-data>, <mv-storage>
• src attribute for the location
• Unprefixed attributes for the different parameters.
  • Implementation: This will be tricky, since observedAttributes is designed to take a static array, but perhaps we could simply add all backend properties there. This will require backends declaring their settings declaratively, but that's probably a good thing anyway.

Syntax for linking to backends

Possibly the most challenging part of this is what is the best way for the Mavo app to reference these.
For now, we can continue using the same set of attributes, but instead of URLs they will use ids (unsure whether to use plain ids or #-prefixed).

To keep things simple, the MVP would only support one id, but the plan is to support multiple in the future.

Migration plan

The mv-storage/mv-source/mv-init/mv-uploads attributes would be shortcuts to this and would generate the appropriate attribute and element behind the scenes. Same with mv-storage-*, mv-source-*, mv-init-* but these would also print a deprecation warning (users would be advised to use the element to specify metadata on their backends).

mv-autosave would be moved to the respective backend element. If the element was specified by the user (and not automatically generated), we should also print a deprecation warning.

[joyously]

Writing data doesn't require fallbacks.

Does this mean there would be no way to have a fallback, or that all errors are handled in the component?

• authentication failed
• no write permission (whether initially or changed since login)
• cannot reach server
• malformed request for that server
• other, such as Terms of service: over quota or too much too fast

Without a fallback, their data is just lost?

[LeaVerou]

@joyously Not sure what you're referring to? That’s describing how Mavo currently works, that data reading has fallbacks (if the source backend has no data) but writing doesn't (because it doesn't matter if the write backend has no data). It has nothing to do with these error conditions, which will continue to be handled the same. If anything, with the new syntax, the idea is that eventually you can provide multiple backends for writing, so that you can have redundancy.

[joyously]

That’s describing how Mavo currently works

Thank you. I couldn't determine that it described the current functionality. When I was trying to write my storage backend, I really couldn't tell what it was supposed to handle and what the base class handled. (perhaps lack of docs)
Since the storage backend doesn't have to deal with the mv-* attributes, it doesn't make sense to me for the custom element to have them, but without URLs. I guess I don't quite grok where the code is that takes a URL and makes an ID (and why).

[karger]

Before diving into engineering, I'd prefer to back up and think about what kind of functionality, and what kind of conceptual model to support it, is the right one for mavo applications. In particular, I think it's worth thinking about whether a model that has worked well for decades---a file system---is the right one for these web applications, or whether web applications in general need an entirely different model.

For starters let's look at the argument for a filesystem approach. People have a very strong understanding of its conceptual model, and the fact that our "files" would usually live on the web instead of the device is not a big deal in this era of google drive and dropbox (perhaps webdav just came too early). People would have no difficulty with ideas like "to edit my own data with this app, I open this file, but to look at my friend's data that they are sharing with me, I open that file." If I launch an app for the first time with no configuration, there's a customary workflow of opening a new blank "file" in such circumstances. We can continue to use our current heuristics for specifying the default "filename" for that new file. This would also simplify things for a user who for example wanted to have two distinct data files and switch between them.

If we pursued this approach we would probably want to develop a "file manager" component that would allow the user to list, open, delete, and move files. But all of this maps quite naturally to many of our back ends. We would also, for convenience, want to remember which was the last opened file so we could open it on the next visit, but this could easily be recorded in localstorage.

Some complication comes from the fact that certain functionalities that an app requires might only be available on certain back-ends. For example if I am relying on firebase realtime updates I can't switch my back end to github and expect that to keep working (though I could make it work with dropbox https://developers.dropbox.com/detecting-changes-guide ). I think this points towards an author specifying which capabilities are required of their app, which would cause the set of available back-ends to be filtered to meet that specification.

[LeaVerou]

@karger I don't understand how your message relates to the first post at all, did you mean to post it somewhere else?
This is about changing the HTML syntax to match the current conceptual model better, it has nothing to do with whether the conceptual model for storage will be filesystem based or not.

[LeaVerou]

That’s describing how Mavo currently works

Thank you. I couldn't determine that it described the current functionality. When I was trying to write my storage backend, I really couldn't tell what it was supposed to handle and what the base class handled. (perhaps lack of docs) Since the storage backend doesn't have to deal with the mv-* attributes, it doesn't make sense to me for the custom element to have them, but without URLs. I guess I don't quite grok where the code is that takes a URL and makes an ID (and why).

Yes, lack of docs, especially for plugin development, is sadly a very pervasive problem :( Hopefully once we move to ESM we can just use typedoc.

[DmitrySharabin]

For now, we can continue using the same set of attributes, but instead of URLs they will use ids (unsure whether to use plain ids or #-prefixed).

Do I understand correctly that we can go either the “HTML way” or “CSS way“? By “HTML way” I mean “plain ids” similar to how ids are used to link to datalists via the list attribute or associate form elements with their labels via the for attribute? And by “CSS way“ I mean #-prefixed ids like in ID selectors.

The HTML way feels more natural and more aligned with the rest of the native elements we have in HTML. However, the CSS way feels more powerful, especially if we plan to support linking to multiple backends—the author can simply provide a (valid) CSS selector, and they are done.

I wonder, is there an attribute in HTML whose value is a CSS selector? 🤔

[karger]

@karger I don't understand how your message relates to the first post at all, did you mean to post it somewhere else? This is about changing the HTML syntax to match the current conceptual model better, it has nothing to do with whether the conceptual model for storage will be filesystem based or not.

My concern was about investing substantial effort in updating our HTML syntax (which also probably means refining the current conceptual model of storage) if it turns out that the current conceptual model is not a good one.

[karger]

Reading your initial post, I wonder just how much of what you are doing needs to be specialized to "back ends". From the perspective of a mavo app that wants to access the data of the custom element, all mavo needs is that custom element to have a standard api for (i) getting the data blob from that component and (ii) giving the data blob to the component. So long as the custom element has that contract, it could be (i) something whose data is implicit, computed when queried, (ii) an interface to some smart-home hardware, (iii) a chart or other data visualization, etc.

[LeaVerou]

@karger

My concern was about investing substantial effort in updating our HTML syntax (which also probably means refining the current conceptual model of storage) if it turns out that the current conceptual model is not a good one.

Please open a separate issue to brainstorm changes to the conceptual model. It is off-topic here.

Reading your initial post, I wonder just how much of what you are doing needs to be specialized to "back ends". From the perspective of a mavo app that wants to access the data of the custom element, all mavo needs is that custom element to have a standard api for (i) getting the data blob from that component and (ii) giving the data blob to the component. So long as the custom element has that contract, it could be (i) something whose data is implicit, computed when queried, (ii) an interface to some smart-home hardware, (iii) a chart or other data visualization, etc.

"Backend" is the general term we use for any data provider. Elements like what you are describing are already a type of backend we support. If you have better ideas about what the concept could be called, feel free to suggest them (that would actually be on-topic, since we're also discussing naming of that element).

[DmitrySharabin]

Thinking of this a bit more, I try to imagine how the new syntax would work.

Suppose we decided to go with <mv-backend> (I like this general name and find it rather widely spread and well-known even among people who don't write any backend code). And suppose we work with the GitHub backend. With all the supported storage attributes we might have something like this:

<mv-backend
	id="github"
	src="https://github.com"
	username="mavoweb"
	repo="test"
	branch="js-first-tests"
	filepath="data"
	filename="countries.json">
</mv-backend>

Considering that custom elements must have the closing tag, we might end up with an empty element with a bunch of attributes inside its opening tag. In simple cases, it's acceptable from my perspective. However, what if we go with an alternative approach for more complex cases like the one above? (Can we have both? Can we mix them?)

Since all storage attributes are the backend options, what if we re-use the existing <option> element to define those options, like so?

<mv-backend id="github" src="https://github.com">
	<option id="username" value="mavoweb"></option>
	<option id="repo" value="test"></option>
	<option id="branch" value="js-first-tests"></option>
	<option id="filepath" value="data"></option>
	<option id="filename" value="countries.json"></option>
</mv-backend>

Or simply (If the value attribute is omitted, the value is taken from the text content of the <option> element)

<mv-backend id="github" src="https://github.com">
	<option id="username">mavoweb</option>
	<option id="repo">test</option>
	<option id="branch">js-first-tests</option>
	<option id="filepath">data</option>
	<option id="filename">countries.json</option>
</mv-backend>

Or we can follow the approach we already use when localizing UI and text in Mavo: we use the value attribute for the option name, and the <option> text content as the option value.

<mv-backend id="github" src="https://github.com">
	<option value="username">mavoweb</option>
	<option value="repo">test</option>
	<option value="branch">js-first-tests</option>
	<option value="filepath">data</option>
	<option value="filename">countries.json</option>
</mv-backend>

It would also be nice if <option>s could have a name attribute so we could have <option name="..." value="..."></option> or <option name="...">value</option>. Unfortunately, it's not valid HTML (please correct me if I'm wrong) since <option> doesn't include this attribute.

With the proposed syntax, we might allow authors to:

• generate the backend options with expressions

<mv-backend id="github" src="https://github.com" mv-list mv-value="options">
	<option mv-list-item value="[id]" mv-group>[value]</option>
</mv-backend>

Where

{
	...
	"options": [
		{
			"id": "username",
			"value": "mavoweb"
		},
		{
			"id": "repo",
			"value": "test"
		},
		{
			"id": "branch",
			"value": "js-first-tests"
		},
		{
			"id": "filepath",
			"value": "data"
		},
		{
			"id": "filename",
			"value": "countries.json"
		}
	]
	...
}

• provide fallbacks (initial values) for options whose values are set with expressions

<option id="...:" value="[expression]">fallback/initial value</option>

It will also allow the opening tag not to be polluted with many attributes but to contain only the essential (and global) ones.

However, this approach might have another downside (amongst the others, I can't see yet 😅). What if we decide to provide the backend initial/fallback data as <mv-backend> text content? It won't be possible with <option>s inside it, right?

By providing initial/fallback data, I imagine something like this:

<mv-backend id="github" src="https://github.com/...">
	<!-- Initial (fallback) data -->
	{
		"country": [
			{
				"name": "Online"
			},
			{
				"code": "us",
				"name": "United States"
			},
			{
				"code": "gb",
				"name": "United Kingdom"
			}
		]
	}
</mv-backend>

[LeaVerou]

<option> already has a different meaning in HTML, of providing a list with a machine-readable value and a human-readable value, which is not needed here, we only need one value. If anything, the most suitable separate element would have been the old <param> element. 😁

Furthermore, it's unclear what benefit the additional structure provides: I proposed making backends an element because they already have their own metadata, but backend properties don't have their own metadata, they are just key-value pairs. Another reason to make something an element is to provide formatting, which you cannot do in an attribute, but that also doesn't apply here. Authors can already provide expressions for these, even if they are attributes.

In terms of implementation, making the params elements currently makes the API more technically challenging to implement, since web components have observedAttributes but nothing for monitoring children, you just fall back to mutation observers. This by itself would drive API decisions, but it's yet another reason.

And not to forget that more verbose syntax can degrade DX if the verbosity is not justified, so it needs to be a balance.

[DmitrySharabin]

Fair! Thank you for your explanation. 🙏 Let’s ignore my proposal, then. 😄