Add serial protocol #59

ed7coyne · 2016-02-03T00:14:31Z

Add serial protocol for Firebase "modem"

Related #35.

Initial copy from google doc

ed7coyne · 2016-02-03T00:40:02Z

serial_protocol.md

+	>>GET /user/aturing
+	<<39 { "first" : "Alan", "last" : "Turing" }
+
+##GET_VALUE


@proppy
First place that might be controversial. I think it is important to not force the user to parse json. I would almost be inclined to not include the primary "get" at all. But I could see it as useful for some users.

I feel that a goal of this project is to provide cloud backends for non-cloud users who have sofar stayed local or offline due to a reluctance to face the learning curve of cloud tech. json isn't a common protocol in the embedded world and is relatively hard/expensive to decode.

proppy · 2016-02-03T20:36:23Z

serial_protocol.md

+##INIT
+Must be called after creating a Serial connection, it can take either just a host for accessing public variables or you may also provide a secret for accessing protected variables in the database.
+###Usage
+	INIT $Host


maybe we could have that be FIREBASE $HOST $SECRET

Hmm not sure about that one, INIT implies that it needs to be called everytime to initialize the connection.

Maybe CONNECT $HOST $SECRET?

Just It seems these should all be commands to the system and phrased as such where FIREBASE isn't really a command.

I was thinking that if the CHIPLET was programmed with multiple APIs, it might expose similar commands derived from REST verbs over different APIs (hence the explicit 'FIREBASE').

But that may be a premature discussion, what about 'BEGIN' to make it Arduinoish?

Ah got ya. Hadn't considered the other APIs. BEGIN works too, we can do BEGIN_FIREBASE if we want to leave the option open for other APIs.

Let's do only begin for now ;) I agree we should worry about multiple API when we have more than 1 binding ;) sorry for the disgression.

Then what about:

API firebase version BEGIN someurl

I am not really sold on adding an extra call. I think the fewer calls you need to make to get the env setup to start communicating the better.

I guess for versions I would just change the begin call. i.e. create a BEGIN_FIREBASE2 if we ever need to implement a breaking change.

What I liked about the API prelude is that it makes it feels declarative. Like an #include or an import. Later we might add some extra calls that allow you to manage the lifecycle of the APIs loaded in your chiplet: list version, OTA, etc.

Alright I am on board with something like that. We can have a concept of "default api" that will load at startup.

So for now we can just leave this out all together and address it later on when we introduce other APIs or versions.

Alright I am on board with something like that. We can have a concept of "default api" that will load at startup.

Yes, so for now I think it's fine to just have BEGIN url

proppy · 2016-02-03T20:41:29Z

/cc @aliafshar

remove GET_VALUE

proppy · 2016-02-03T23:35:02Z

serial_protocol.md

+	$DATA_BYTE_COUNT $DATA
+###Examples
+	>>GET_VALUE /user/aturing/first
+	<<4 Alan


another issue with having the size as TEXT is that is also a variable length unless we do some padding.

true but you can just read until whitespace. there is also Serial.parseInt()

Then what's the improvement over reading until \n? That you're guaranteed the size will never cross a given threshold (a few digits).

Maybe this could come in the same kind of stateful MODEs than type prefix and forced json strings?

If you know on the outset the size of the value coming back you can prepare for it. For example you can allocate a buffer large enough to hold it without having to allocate a smaller buffer and grow it. You can also quickly decide if something is too large to process and close the connection.

We could also return text without the json escaping which might be nice so the user doesn't need to unescape it themselves. For example if they wanted to send it directly to a display.

proppy · 2016-02-04T01:33:09Z

serial_protocol.md

+
+##GET_BULK
+Same as GET but returns value with size prefix. useful when value is expected to be large so you can know the size before accepting value.
+Also only returns values at leaf nodes, if called on internal node returns error.


I was thinking that the bulk MODE would return the data always as it is coming from fbase (json escaped), otherwise it's a strict subset of GET behavior.

Maybe we need separate flag/mode for json/size/type and keep the default very simple and readable (leaf value, un escaped, no prefix), wdyt?

I am ok with just dropping the bulk mode for now. We can add it back if we see a need for it in practice.
I do think the return from GET needs a prefix though, to allow reliable testing to see if it is an error or valid response before you try to parse it.

I'd love if we added all of those as optional stateful flags you could turn on and off so that we can experiment with the wire protocol and see how it feels without adding all the different combination as separate commands.

Then we can debate which one is the default, but everything you listed sounds nice to have:

type prefix

size prefix

decoded leaf value

disabling non leaf value

proppy · 2016-02-04T06:31:41Z

After thinking more about it I think you can disregard most of my comments wrt to optional type prefix.

It would introduce ambiguities in the protocol to make type prefix optional if we unescape leaf values: client won't be able to distinguish an error from a string value that start with !, or a number value from a string with a number.

proppy · 2016-02-04T06:37:32Z

I also really like the Redis protocol doc you linked http://redis.io/topics/protocol.

I think most of it is directly applicable here and we could easily derive from it with a few caveats:

we don't really have arrays
number can be float
we also have booleans
leaf simple strings and numbers are not size prefixed only bulk json values are.

proppy · 2016-02-04T06:42:06Z

serial_protocol.md

+##Response Format Byte
+All responses will be prefixed with one of the following bytes signifying the response type.
+```
+  + If response is ok and a raw value


Should we also have something for raw number and booleans ( maybe : and ? ?

proppy · 2016-02-05T14:43:34Z

serial_protocol.md

+  + If response is ok and a raw string value.
+  * If response is ok and a raw string value prefixed by count of bytes in response then new line.
+  # If response is ok and a integer value.
+  . If response is ok and a float value.


json doesn't have a difference between float and int: so maybe just :number

json doesn't but I think there is a lot of value in providing the distinction for c++ parsing. So you can do a atoi() call with confidence knowing that the string will parse. I think we can determine pretty easily in the library (check for a '.').

I would be down for always return float, so that we can always use atofon the receiving size.

We never know if something is intended to be stored as an int, or as a float on the firebase side: it might just be that the current intermediate value is a round number. It seems easy enough for the client to cast it back to an int when needed (or maybe we could expose a typed GET that always return int).

I see where that could be a problem if the value changes from float to int due to chance. Alright I am onboard with leaving the ambiguity up to the user to resolve. Just label them a "number", the user will probably know with domain context what to expect.

yes, that's why I think we might just need : for number and just return what we get from firebase.

…into Serial

ed7coyne · 2016-02-10T19:03:07Z

I think we have reached consensus here. Are there any more outstanding issues or does this protocol look like a solid starting point?

proppy · 2016-02-12T01:34:20Z

serial_protocol.md

+###Usage
+	GET $PATH
+###Response
+	$DATA_AT_PATH


Should this be +DATA_AT_PATH? maybe add a separator between the two.

This is the confusing part of using "$" for bulk strings. I am using it in all of the documentation to represent a variable (ala bash). So this should stay variable "DATA_AT_PATH" because we cannot say what type it will be.

proppy · 2016-02-12T18:47:14Z

We should also take a look at alternative like https://github.com/jeelabs/esp-link#outbound-http-rest-requests-and-mqtt-client (with https://github.com/jeelabs/el-client client) or https://github.com/tuanpmt/espduino and see how the protocol/usage compare to it.

Their protocol is apparently based on SLIP.

ed7coyne · 2016-02-16T17:45:35Z

We should take a look at them. In general I would be against using the SLIP protocol though. I think human readable is really the right approach for us, ease of comprehension and debugging should be pretty high on our priority list.

But that is my personal bias, this was the reason I selected redis for my personal projects.

ed7coyne · 2016-02-16T19:55:11Z

serial_protocol.md

+
+The event stream will continue until you send CANCEL_STREAM.
+
+When there is an ongoing event stream no other commands can be processed until you call END_STREAM as the event stream owns the return line. 


Added this note in here as well. I don't see a clean way around this, let me know if you disagree.

do we need different command for CANCEL_STREAM and END_STREAM?

re: "no other commands can be processed", since our protocol is serial based the client will have to poll if Serial.available() Serial.readline() for stream result, maybe it wouldn't hurt to make that more explicit with a command? that way this could be multiplexed with other operation.

Changed CANCEL_STREAM to END_STREAM, I think they should be the same.

I am not certain what you mean by the second comment. What would you like to make explicit? Do you want to cache the stream on the modem and have the client call "NEXT_EVENT"? We may be able to do that but memory could be an issue.

Isn't memory already an issue if we print something to Serial and nobody is reading on the other side?

We could have it read from the wifi connection only when we receive the NEXT_EVENT command?

I'm not sure which one is better, I was just thinking out loud.

Once the serial buffer fills up I think the serial write will just block until it is read. Also if the client can't do anything else with the connection they are less likely to leave it blocked than if they can just start a stream then ignore it while they do other things.

If we leave it in the wifi queue we are back to the state of not being able to do other actions while streaming. All other commands would have to return an error.

That said we can address the issue directly and measure memory usage, if it exceeds a certain amount shutdown the stream and return an error when they call "NEXT_EVENT".

However if we can stream directly to the serial line than I think we can handle much larger/faster streams more efficiently. Maybe are back to adding more modes?

If we leave it in the wifi queue we are back to the state of not being able to do other actions while streaming. All other commands would have to return an error.

So practically, it would have to be a different "connection", since the stream endpoint is different from the API endpoint (remember the 2x redirect you can after calling /stream)

proppy · 2016-04-08T18:59:40Z

serial_protocol.md

+
+Every time a serial connection is established we will expect a “BEGIN” call after which any subsequent calls can be made, until the connection is closed. 
+
+In the following examples we use $ to represent variables. For example $Host represents your firebase host.


nit: since $ is already used in the protocol, maybe we should using a different delimeter for placeholder like %VALUE%, [VALUE] or <VALUE>

proppy · 2016-04-08T19:05:32Z

LGTM just some nits.

ed7coyne added 2 commits February 2, 2016 15:24

Create serial_protocol.md

5c75177

Initial copy from google doc

Update serial_protocol.md

eff6087

ed7coyne reviewed Feb 3, 2016
View reviewed changes

Update serial_protocol.md

7bddea6

proppy reviewed Feb 3, 2016
View reviewed changes

Update serial_protocol.md

81788aa

remove GET_VALUE

proppy reviewed Feb 3, 2016
View reviewed changes

ed7coyne added 4 commits February 3, 2016 15:57

Update serial_protocol.md

04bdf6e

Update serial_protocol.md

8557658

Update serial_protocol.md

67f51e6

Update serial_protocol.md

e323a1c

proppy reviewed Feb 4, 2016
View reviewed changes

Update serial_protocol.md

8a3a56c

proppy reviewed Feb 5, 2016
View reviewed changes

ed7coyne added 2 commits February 9, 2016 10:49

Update serial_protocol.md

6150e54

Update serial_protocol.md

3d96c9c

ed7coyne added 7 commits February 9, 2016 17:11

Update serial_protocol.md

c39e371

updated example for changes to protocol

6ca12be

Merge branch 'Serial' of https://github.com/ed7coyne/firebase-arduino …

c485ef7

…into Serial

Update serial_protocol.md

6be74cd

Update serial_protocol.md

e8c9623

Move serial client to another branch

d9bdee6

Merge branch 'Serial' of https://github.com/ed7coyne/firebase-arduino …

6530949

…into Serial

proppy reviewed Feb 12, 2016
View reviewed changes

Update serial_protocol.md

f004ae5

Update serial_protocol.md

bba8045

ed7coyne reviewed Feb 16, 2016
View reviewed changes

Update serial_protocol.md

8ced4fb

proppy reviewed Apr 8, 2016
View reviewed changes

switched variable marker from $ to %%

e5570d6

ed7coyne merged commit d674692 into FirebaseExtended:master Apr 8, 2016

ed7coyne deleted the Serial branch April 8, 2016 20:22


		The event stream will continue until you send CANCEL_STREAM.

		When there is an ongoing event stream no other commands can be processed until you call END_STREAM as the event stream owns the return line.


		Every time a serial connection is established we will expect a “BEGIN” call after which any subsequent calls can be made, until the connection is closed.

		In the following examples we use $ to represent variables. For example $Host represents your firebase host.

Add serial protocol #59

Add serial protocol #59

Conversation

ed7coyne commented Feb 3, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

proppy commented Feb 3, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

proppy commented Feb 4, 2016

proppy commented Feb 4, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ed7coyne commented Feb 10, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

proppy commented Feb 12, 2016

ed7coyne commented Feb 16, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

proppy commented Apr 8, 2016