Hyper-schema: Operation description objects

[handrews]

The current LDO conflates link existence with operations. This is often misleading (especially because"method": "get" doesn't mean quite what most people would think, and "method": "post" can mean things other than POST even with HTTP.

This behavior mostly comes from HTML forms. However, discussions here have proven that HTML is neither a good role model for detailed linking, nor do analogies with HTML reliably clarify things for folks (although that might be just me).

A better approach would be to have an array of Operation Description Objects (ODOs) within each Link Description Object (LDO). Here's a partial meta-schema for ODOs

{
    "type": "object",
    "properties": {
        "title": {"type": "string"},
        "description": {"type": "string"},
        "submission": {"type": "object", "properties": ...},
        "target": {"type": "object", "properties": ...},
    }
}

submission and target could alternately be called request and response. They gather all keywords relevant to submissions/requests and targets/responses respectively. I will file separate issues for discussing exactly what goes in there- the minimal thing would just be to move the current submission and target keywords into these objects.

title and description do what they usually do.

See also issues #94 (replace method with behavioral keywords), #73 (for an allow hint), and #92 (referring to JSON Home's hint mechanism). I will also file a separate issue to look at how all of these proposals, as well as the forthcoming submission and response proposals, might fit together.

[awwright]

HTML has some very well known problems, but they're well described by various experts; and overwhelmingly HTML is the primere hypermedia format. HTML, I think, is overwhelmingly a good role model.

What do you mean by "link existence" and "operations"? What's the difference?

Ignore for a moment what the choice of terms is, so "get" and "post" are just opaque labels that could be anything. What's the actual problem with the meanings defined in the latest published draft?

Can you describe a little bit how you see the functionality mapping onto RFC 5988 Web Linking?

[handrews]

but they're well described by various experts

What does this mean, exactly? Why is this a mitigating factor?

Here's how I think of things. You can, of course, disregard this on the grounds that maybe I just suck at my job and it's certainly not hyper-schema's role to compensate for that :-) But if you'll assume for a moment that I'm vaguely competent, let me walk you through the practical concerns. I'm probably telling you a lot of things that you know, but I can't think of any other way to highlight the rationale behind my concerns.

This got more than a little melodramatic, but I'm posting it anyway. Just keep in mind that I'm not as totally overwrought here as it might sound, and still think this is all great and productive discussion :-)

---

I need to set up the architecture for an open-ended suite of APIs that are being worked on by multiple engineering teams scattered across the world. Some of them have a lot of expertise, some of them have a vague notion that APIs exist and have something to do with HTTP. Some of them have senior people readily available, some of them do not. Almost none of them are truly comfortable with actual REST (because really, the percentage of engineers I've met that can coherently and accurately describe REST is very small). There are also three generations of legacy HTTP APIs that have to fit in there somewhere. Some people are really into REST, others are looking for any excuse to use RPC.

This describes both my last project and the one I am starting to work on now (the last project only had one legacy generation). Part of my job (both times) is/was to sell the engineering teams on a set of tools and standards to follow. For all of the reasons that came up when other folks spoke up win the "Who uses Hyper-Schema?", I want to use Hyper-Schema. JSON Schema is/was already accepted for validation, but I need/needed to figure out a hypermedia solution.

Whether or not I can sell a particular solution has a lot to do with the UX of that solution. You may find "UX" a surprising term here, but there is user experience for APIs, standards, etc. etc. The users here are the engineers and engineering managers. The experience is determined by how easy the system is to learn and work with.

How approachable is it? Is it intuitive, confusing, or actively misleading? How many counter-intuitive things do developers have to hold in their heads while working with the system? How many complaints will managers have to deal with if the engineers don't consider the system fun to work with? How will it impact recruiting? Will we lose any good employees over this thing?

That last one may seem far-fetched but it really is not- it had nothing to do with REST APIs but I had to correct something in a major new architecture years ago because, whether it worked or not, the people who had to deal with it hated it. I could come up with all sorts of explanations why it was really fine, and it didn't mean the horrible things they thought, but the whole project was a major change and I had already pushed them through many adjustments in how they conceptualized their work. That was just one thing too many (it was an educational experience).

Now, back to REST and Hyper-Schema. At my previous project, I was still new to all of this stuff, and there was a certain amount of architectural inertia already. I convinced myself that Hyper-Schema could do what we wanted (which was mostly correct). I couldn't sell the other two architects on it, much less the managers or engineers. The other architects gave it a good solid go, and they were some of the smartest people I've worked with. But too many things just didn't line up with expectations or map cleanly. I had already spent months convincing them that we needed and would benefit from HATEOAS and implementing a generic API client, and hyper-schema was just that one thing too many.

I would prefer not to repeat that experience. The spec is now a lot more clear. I understand REST and hyper-schema a lot better now than I did three years ago, and discussions here have further clarified my understanding of both. It's more clear what roles hyper-schema does and does not fill, and how to fill those other roles. People are more receptive to HATEOAS and other RESTful concepts than they were before. But I can just see how this will go:

Me: "so here's how you say what you can do, but get and post don't actually mean get and post, and get doesn't necessarily mean you can fetch the resource at all"

Them: "wait, what? why are we restricted to only two things that don't even do what they say?"

Me: "It's based on HTML forms because HTML is the original success story of hypermedia"

Them: "But this is JSON and APIs, not HTML and browser forms..."

Me: "Just think of 'get' and 'post' as opaque tokens"

Junior engineers: "What even?"

Engineering managers: "So every time we onboard a new person we have to explain this to them? How likely is this to increase our bug rate?"

Product managers: "We can hide all this from client developers, but how easy will it be for partners who want to extend our API ecosystem to learn this?"

Me: "No seriously, it's fine, some experts worked out a way to describe the problems and they're well understood."

"Everyone": (not printable in a public forum)

At this point, I get told to find a better system, or teams just all go their own ways.

---

While this is a bit of a worst case scenario, it's really not that far-fetched. If everything else is perfect, I doubt the method get/post thing would be fatal. At worst I might have to allow folks to add an extension field to work around it. But even the notion of "these are abstract operations, not CRUD actions" is going to be a major education effort, and putting a confusing format on top of that will make it an order of magnitude more difficult. At most 50% of adopting a major engineering approach is technical- the remainder has more to do with social factors than anything else.

There are a lot of other systems for HTTP APIs out there. Some of them are at least kinda close to supporting REST, and could do so with minimal patching over. If they are viewed as much easier / much less costly to use than hyper-schema, we won't end up using hyper-schema. No sane head of engineering is going to adopt a system that is viewed as more difficult and more costly (in terms of time, morale, or whatever) than readily available alternatives.

How interested are we in seeing JSON Hyper-Schema being adopted widely? I am very committed to hyper-schema, but it needs to be enjoyable enough to work with and think about that I can get real buy-in across a broad range of experience levels and attitudes.

[handrews]

Can you describe a little bit how you see the functionality mapping onto RFC 5988 Web Linking?

The target section would correlate with the target attributes in the link element/header. Although that's the part I've given the least thought so far- I stuffed the existing targetScema and mediaType keywords into a separate target section mostly for symmetry. A "post" operation and other operations on the same URI template with the same resolution may produce very different target representations, so I put target in the ODO instead of leaving it up in the LDO.

However, RFC 5988 doesn't cover the request side of operations, so I don't think it maps at all, nor should it. Remember, there's still an LDO containing the list of ODOs. The LDO is mainly what correlates to the stuff in RFC 5988.

[handrews]

This was part of the same attempt to come up with a more abstract and flexible option than just importing "allow" from http as noted in #94. It's clearly too far off-base to pursue further.

All of my points about "get" and "post" being extremely problematic and likely straws-that-broke-the-camel's-back still hold.

This perhaps would have had a better reaction if I'd called them "Form Description Objects". Nothing would change about the proposal, but the term "operations" seems to be too rpc-ish or service-definition-ish or something.

[awwright]

Keep in mind the way we're describing things here is going to be different than how you describe usage to coworkers. We're designing a specification whose audience is developers of validation libraries. So when I say that "get" and "post" are opaque tokens, that's because we can make the vocabulary more intuitive later, right now we're more concerned with how is this going to work.

For regular, end-developers who are merely designing hypermedia APIs (instead of entire validators and media types), I would strongly suggest RESTful Web APIs by Leonard Richardson and Mike Amundsen. This is going to get you off on the right foot and set things in the right frame for people who haven't thought about hypermedia as a machine-interactive format before.

[handrews]

Keep in mind the way we're describing things here is going to be different than how you describe usage to coworkers. We're designing a specification whose audience is developers of validation libraries.

Who do you think writes the schemas? Specification implementors are highly motivated to work through all of the issue and adapt to whatever- that's what I'm doing here. Schema authors are more likely to say "this system is confusing, let's use Swagger." I can explain the difference between hypermedia and service definitions, but at the end of the day, this has to be usable by the people who write the schemas, not just the people who build tools.

So when I say that "get" and "post" are opaque tokens, that's because we can make the vocabulary more intuitive later, right now we're more concerned with how is this going to work.

Right now I'm concerned with that, but I'm at least as concerned with demonstrating that this is a system that people want to use even if we're still working through the rough edges. Why such a strong pushback against picking better terminology now? The longer we leave it as-is the more people write schemas and the more inertia builds for preserving compatibility.

[handrews]

To wrap this up in the unlikely event that anyone comes through and reads this (because I want it to be clear why I closed it instead of it just looking like I got frustrated and rant-y. Which I did, but that's not why I closed it).

• I came up with ODOs because I (incorrectly) thought you wanted something more like that than the HTML terminology, and wanted me to file a proposal on it based on a comment in the "Does anyone use hyper-schema?" thread. You may have wanted a proposal, but this was not it.
• Whether they are called forms or operations, it's possible to have multiple ways to send input to a resource using the same URI with the same relation, and the target hints are likely different as well. It feels more intuitive to me to describe those different ways of supplying input within the same LDO instead of repeating parts of the LDO each time. I still feel like there's an issue here, but I'll have to think on it more to decide whether I want to pursue it further, and if so how.
• I prefer using a bit more structure to make everything more readable, so I grouped the submission vs target names (I would no longer suggest calling them request and response, even though I put that in the original description). I'm honestly not sure why I thought anyone would be receptive to that as JSON Schema strongly prefers grouping keywords with prefixes over using additional levels of structure. I dislike the lack of symmetry in current submission vs target keywords, but that's a different issue, really.

The rest of the comments here are about schema author UX, which I still think is a major concern for this project. While I appreciate the book rec and will check that out, the book does not appear to address my primary designer/developer UX concerns. I'm troubled by how readily this has been dismissed, but this issue is not the correct forum for further discussion.

[jdesrosiers]

@handrews, first of all, I wanted to say that I believe these discussions have been very valuable even if we aren't all always on the same page. I hope you don't get discouraged and stop contributing. Every time we all disagree, we get one step closer to consensus on what Hyper-Schema should be.

I share you concerns about "get" and "post" and have some of my own (separate issues filed). I don't think we ended up with something better this time around.

[handrews]

@jdesrosiers thanks for the encouragement! I do not feel that anyone has been overly or unfairly critical, and I've also found these discussions valuable. I'm concerned over the UX thing but not in the sense of thinking the discussion has been totally shut down.

I'm feeling better about things today (took the day off work for an interactive performing gig- switching to art mode for a day helped clear my head). My main concern is that I don't want to contribute more noise than signal, and it felt like I was tilting that way yesterday.