Skip to content
This repository was archived by the owner on Mar 17, 2025. It is now read-only.

Add serial protocol #59

Merged
merged 39 commits into from
Apr 8, 2016
Merged
Changes from 8 commits
Commits
Show all changes
39 commits
Select commit Hold shift + click to select a range
5c75177
Create serial_protocol.md
ed7coyne Feb 2, 2016
eff6087
Update serial_protocol.md
ed7coyne Feb 3, 2016
7bddea6
Update serial_protocol.md
ed7coyne Feb 3, 2016
81788aa
Update serial_protocol.md
ed7coyne Feb 3, 2016
04bdf6e
Update serial_protocol.md
ed7coyne Feb 3, 2016
8557658
Update serial_protocol.md
ed7coyne Feb 4, 2016
67f51e6
Update serial_protocol.md
ed7coyne Feb 4, 2016
e323a1c
Update serial_protocol.md
ed7coyne Feb 4, 2016
8a3a56c
Update serial_protocol.md
ed7coyne Feb 4, 2016
6150e54
Update serial_protocol.md
ed7coyne Feb 9, 2016
3d96c9c
Update serial_protocol.md
ed7coyne Feb 9, 2016
d554f5d
Update serial_protocol.md
ed7coyne Feb 9, 2016
246f2e8
Update serial_protocol.md
ed7coyne Feb 9, 2016
d3da4e9
Update serial_protocol.md
ed7coyne Feb 9, 2016
07c43db
Update serial_protocol.md
ed7coyne Feb 9, 2016
ebb181b
Update serial_protocol.md
ed7coyne Feb 9, 2016
51921e3
Update serial_protocol.md
ed7coyne Feb 9, 2016
1b318b3
Update serial_protocol.md
ed7coyne Feb 9, 2016
6297660
Update serial_protocol.md
ed7coyne Feb 9, 2016
ecea516
Update serial_protocol.md
ed7coyne Feb 9, 2016
6f30401
Update serial_protocol.md
ed7coyne Feb 9, 2016
8dac3f3
Update serial_protocol.md
ed7coyne Feb 9, 2016
b0d90d2
Added Serial client example.
ed7coyne Feb 9, 2016
724fde5
Merge branch 'master' of https://github.com/ed7coyne/firebase-arduino
ed7coyne Feb 9, 2016
1c422b9
Merge branch 'Serial' of https://github.com/ed7coyne/firebase-arduino…
ed7coyne Feb 9, 2016
fc6a0f4
Update FirebaseSerialClient.ino
ed7coyne Feb 9, 2016
9f146de
Added buttons and led logic
ed7coyne Feb 10, 2016
76c746c
merged
ed7coyne Feb 10, 2016
c39e371
Update serial_protocol.md
ed7coyne Feb 10, 2016
6ca12be
updated example for changes to protocol
ed7coyne Feb 10, 2016
c485ef7
Merge branch 'Serial' of https://github.com/ed7coyne/firebase-arduino…
ed7coyne Feb 10, 2016
6be74cd
Update serial_protocol.md
ed7coyne Feb 10, 2016
e8c9623
Update serial_protocol.md
ed7coyne Feb 10, 2016
d9bdee6
Move serial client to another branch
ed7coyne Feb 10, 2016
6530949
Merge branch 'Serial' of https://github.com/ed7coyne/firebase-arduino…
ed7coyne Feb 10, 2016
f004ae5
Update serial_protocol.md
ed7coyne Feb 16, 2016
bba8045
Update serial_protocol.md
ed7coyne Feb 16, 2016
8ced4fb
Update serial_protocol.md
ed7coyne Mar 25, 2016
e5570d6
switched variable marker from $ to %%
ed7coyne Apr 8, 2016
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
72 changes: 72 additions & 0 deletions serial_protocol.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
#Protocol:
During the first use, or when the chiplet changes environments a “NETWORK” call is expected to initialize the wifi parameters.

Every time a serial connection is established we will expect a “INIT” call after which any subsequent calls can be made, until the connection is closed.

In the following examples we use $ to represent variables. For example $Host represents your firebase host.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: since $ is already used in the protocol, maybe we should using a different delimeter for placeholder like %VALUE%, [VALUE] or <VALUE>


##Response Format Byte
All responses will be prefixed with one of the following bytes signifying the response type.
```
+ If response is ok and a raw value
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we also have something for raw number and booleans ( maybe : and ? ?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

* If response is ok and a raw value prefixed by count of bytes in response.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need size prefix for leaf raw value like number and simple strings they are unlikely to grow that big.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is actually where I saw the most use for it. Say you wanted to store a bunch of text to scroll on a display with line breaks. Or say you wanted to push a large binary blob, like a OTA update, to your client. This is where it is nice to know the size of the package so you can prepare memory and storage for it before you start reading it.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

True, since the GET command already have variant result (either an int, or a string, or a boolean or some json) maybe we could just have the result being auto promoted to a bulk string once it pass a fixed size? what do you think.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was thinking about that as well. The only use case that doesn't support is multiline strings, we cannot know if the value is a multiline string without reading the whole value. But I would like to keep the protocol able to work without the chiplet having to read the whole value into memory. It would be more flexible if it were able to stream data directly from the http connection to the serial connection. It is very possible that the chiplet will have far less ram than the client's MCU.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

actually as I think about handling multiline strings will be a pain anyway. I think we should just pass them back to the client and allow them to un-escape the newlines.
Since we don't know how many new lines there are we cannot just use the http body size as the size of our payload since everytime we translate the escaped line to a \n we will reduce the size. Technically we can just pad the output, since we would always be the stated size or less it shouldn't cause a problem but it seems extra work for something that isn't actually hard for the client to do.
Also the closer we keep our wireformat to the response from the REST call the better I think. That way they can just use curl for debugging to see what the value should look like.

Alright I am sold, lets just determine a threshold to autopromote a string to a bulk string.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can also use the content-length as an hint to promote to bulk string.

The trick is that for some "growing" value the client code might have to adapt to get predictable output.

Maybe we should have typed GET command where the client can express more intent:
GET$ to force bulk strings
GET: to force numbers
GET+ to force simple strings
GET? to force bool
GET do your best

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well I think as long as we are always prefixing the response than the user has the tools they need to resolve a "growing" value. When the prefix changes they can adapt.

That said I don't see any harm in providing typed get commands that will just return an error if the type is not matched.

& If response is ok and json formatted
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel like we could always use $ and size prefix for json value (sorry if I went back and forth on this)

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am ok with that, not a problem to change your mind. It is easy to do at this stage, better than when we have actually implemented things.

$ If response is ok and json formatted and prefixed by count of bytes in response.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe add a \r\n after the size to match Redis spec for bulk strings

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

! If response is an error
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we should use - like redis

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

```
##NETWORK
Only needs to be called when the chiplet is in a new environment and needs to connect to a new network. This setting will be stored by the chiplet and persisted through power cycles.
###Usage
NETWORK $SSID
NETWORK $SSID $PASSWORD
###Response
CONNECTED - When connected to new network
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should this be +CONNECTED :)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

on nvm, this is then correct in the example.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed

UNABLE_TO_CONNECT - When unable to connect
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

-UNABLE_TO_CONNECT?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same here :)

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed

###Examples
>> NETWORK home-guest
<< +CONNECTED
>> NETWORK home-private MySecretPassword
<< +CONNECTED
>> NETWORK home-guest
<< !UNABLE_TO_CONNECT

##INIT
Must be called after creating a Serial connection, it can take either just a host for accessing public variables or you may also provide a secret for accessing protected variables in the database.
###Usage
INIT $Host
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe we could have that be FIREBASE $HOST $SECRET

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm not sure about that one, INIT implies that it needs to be called everytime to initialize the connection.

Maybe CONNECT $HOST $SECRET?

Just It seems these should all be commands to the system and phrased as such where FIREBASE isn't really a command.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was thinking that if the CHIPLET was programmed with multiple APIs, it might expose similar commands derived from REST verbs over different APIs (hence the explicit 'FIREBASE').

But that may be a premature discussion, what about 'BEGIN' to make it Arduinoish?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah got ya. Hadn't considered the other APIs. BEGIN works too, we can do BEGIN_FIREBASE if we want to leave the option open for other APIs.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's do only begin for now ;) I agree we should worry about multiple API when we have more than 1 binding ;) sorry for the disgression.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Then what about:

API firebase version
BEGIN someurl

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not really sold on adding an extra call. I think the fewer calls you need to make to get the env setup to start communicating the better.

I guess for versions I would just change the begin call. i.e. create a BEGIN_FIREBASE2 if we ever need to implement a breaking change.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What I liked about the API prelude is that it makes it feels declarative. Like an #include or an import. Later we might add some extra calls that allow you to manage the lifecycle of the APIs loaded in your chiplet: list version, OTA, etc.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alright I am on board with something like that. We can have a concept of "default api" that will load at startup.

So for now we can just leave this out all together and address it later on when we introduce other APIs or versions.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alright I am on board with something like that. We can have a concept of "default api" that will load at startup.

Yes, so for now I think it's fine to just have BEGIN url

INIT $Host $Secret
###Response
OK - Accepted initialization parameters
###Examples
>> INIT https://samplechat.firebaseio-demo.com
<< +OK
>> INIT https://samplechat.firebaseio-demo.com nnz...sdf
<< +OK
##GET
Fetches the value at $Path and returns it on the serial line. If $PATH points to a leaf node you will get the raw value back, if it points to an internal node you will get a JSON string with all children.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

s/Path/PATH` to match the "Usage"?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done.

###Usage
GET $PATH
###Response
$DATA_AT_PATH
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be +DATA_AT_PATH? maybe add a separator between the two.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the confusing part of using "$" for bulk strings. I am using it in all of the documentation to represent a variable (ala bash). So this should stay variable "DATA_AT_PATH" because we cannot say what type it will be.

###Examples
>>GET /user/aturing/first
<<+Alan
>>GET /user/aturing
<<&{ "first" : "Alan", "last" : "Turing" }

##GET_BULK
Same as GET but returns value with size prefix. useful when value is expected to be large so you can know the size before accepting value.
Also only returns values at leaf nodes, if called on internal node returns error.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was thinking that the bulk MODE would return the data always as it is coming from fbase (json escaped), otherwise it's a strict subset of GET behavior.

Maybe we need separate flag/mode for json/size/type and keep the default very simple and readable (leaf value, un escaped, no prefix), wdyt?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am ok with just dropping the bulk mode for now. We can add it back if we see a need for it in practice.
I do think the return from GET needs a prefix though, to allow reliable testing to see if it is an error or valid response before you try to parse it.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd love if we added all of those as optional stateful flags you could turn on and off so that we can experiment with the wire protocol and see how it feels without adding all the different combination as separate commands.

Then we can debate which one is the default, but everything you listed sounds nice to have:

  • type prefix
  • size prefix
  • decoded leaf value
  • disabling non leaf value

###Usage
GET_BULK $PATH
###Response
$DATA_BYTE_COUNT $DATA_AT_PATH
###Examples
>>GET_BULK /user/aturing/first
<<*4 Alan
>>GET /user/aturing
<<$39 { "first" : "Alan", "last" : "Turing" }
##Set
##Push
##Remove
##Stream