Skip to content
This repository was archived by the owner on Mar 17, 2025. It is now read-only.

Add serial protocol #59

Merged
merged 39 commits into from
Apr 8, 2016
Merged

Add serial protocol #59

Changes from 3 commits
Commits
Show all changes
39 commits
Select commit Hold shift + click to select a range
5c75177
Create serial_protocol.md
ed7coyne Feb 2, 2016
eff6087
Update serial_protocol.md
ed7coyne Feb 3, 2016
7bddea6
Update serial_protocol.md
ed7coyne Feb 3, 2016
81788aa
Update serial_protocol.md
ed7coyne Feb 3, 2016
04bdf6e
Update serial_protocol.md
ed7coyne Feb 3, 2016
8557658
Update serial_protocol.md
ed7coyne Feb 4, 2016
67f51e6
Update serial_protocol.md
ed7coyne Feb 4, 2016
e323a1c
Update serial_protocol.md
ed7coyne Feb 4, 2016
8a3a56c
Update serial_protocol.md
ed7coyne Feb 4, 2016
6150e54
Update serial_protocol.md
ed7coyne Feb 9, 2016
3d96c9c
Update serial_protocol.md
ed7coyne Feb 9, 2016
d554f5d
Update serial_protocol.md
ed7coyne Feb 9, 2016
246f2e8
Update serial_protocol.md
ed7coyne Feb 9, 2016
d3da4e9
Update serial_protocol.md
ed7coyne Feb 9, 2016
07c43db
Update serial_protocol.md
ed7coyne Feb 9, 2016
ebb181b
Update serial_protocol.md
ed7coyne Feb 9, 2016
51921e3
Update serial_protocol.md
ed7coyne Feb 9, 2016
1b318b3
Update serial_protocol.md
ed7coyne Feb 9, 2016
6297660
Update serial_protocol.md
ed7coyne Feb 9, 2016
ecea516
Update serial_protocol.md
ed7coyne Feb 9, 2016
6f30401
Update serial_protocol.md
ed7coyne Feb 9, 2016
8dac3f3
Update serial_protocol.md
ed7coyne Feb 9, 2016
b0d90d2
Added Serial client example.
ed7coyne Feb 9, 2016
724fde5
Merge branch 'master' of https://github.com/ed7coyne/firebase-arduino
ed7coyne Feb 9, 2016
1c422b9
Merge branch 'Serial' of https://github.com/ed7coyne/firebase-arduino…
ed7coyne Feb 9, 2016
fc6a0f4
Update FirebaseSerialClient.ino
ed7coyne Feb 9, 2016
9f146de
Added buttons and led logic
ed7coyne Feb 10, 2016
76c746c
merged
ed7coyne Feb 10, 2016
c39e371
Update serial_protocol.md
ed7coyne Feb 10, 2016
6ca12be
updated example for changes to protocol
ed7coyne Feb 10, 2016
c485ef7
Merge branch 'Serial' of https://github.com/ed7coyne/firebase-arduino…
ed7coyne Feb 10, 2016
6be74cd
Update serial_protocol.md
ed7coyne Feb 10, 2016
e8c9623
Update serial_protocol.md
ed7coyne Feb 10, 2016
d9bdee6
Move serial client to another branch
ed7coyne Feb 10, 2016
6530949
Merge branch 'Serial' of https://github.com/ed7coyne/firebase-arduino…
ed7coyne Feb 10, 2016
f004ae5
Update serial_protocol.md
ed7coyne Feb 16, 2016
bba8045
Update serial_protocol.md
ed7coyne Feb 16, 2016
8ced4fb
Update serial_protocol.md
ed7coyne Mar 25, 2016
e5570d6
switched variable marker from $ to %%
ed7coyne Apr 8, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
61 changes: 61 additions & 0 deletions serial_protocol.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
#Protocol:
During the first use, or when the chiplet changes environments a “NETWORK” call is expected to initialize the wifi parameters.

Every time a serial connection is established we will expect a “INIT” call after which any subsequent calls can be made, until the connection is closed.

In the following examples we use $ to represent variables. For example $Host represents your firebase host.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: since $ is already used in the protocol, maybe we should using a different delimeter for placeholder like %VALUE%, [VALUE] or <VALUE>

##NETWORK
Only needs to be called when the chiplet is in a new environment and needs to connect to a new network. This setting will be stored by the chiplet and persisted through power cycles.
###Usage
NETWORK $SSID
NETWORK $SSID $PASSWORD
###Response
OK - When connected to new network
FAIL - When unable to connect
###Examples
>> NETWORK home-guest
<< OK
>> NETWORK home-private MySecretPassword
<< OK

##INIT
Must be called after creating a Serial connection, it can take either just a host for accessing public variables or you may also provide a secret for accessing protected variables in the database.
###Usage
INIT $Host
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe we could have that be FIREBASE $HOST $SECRET

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm not sure about that one, INIT implies that it needs to be called everytime to initialize the connection.

Maybe CONNECT $HOST $SECRET?

Just It seems these should all be commands to the system and phrased as such where FIREBASE isn't really a command.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was thinking that if the CHIPLET was programmed with multiple APIs, it might expose similar commands derived from REST verbs over different APIs (hence the explicit 'FIREBASE').

But that may be a premature discussion, what about 'BEGIN' to make it Arduinoish?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah got ya. Hadn't considered the other APIs. BEGIN works too, we can do BEGIN_FIREBASE if we want to leave the option open for other APIs.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's do only begin for now ;) I agree we should worry about multiple API when we have more than 1 binding ;) sorry for the disgression.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Then what about:

API firebase version
BEGIN someurl

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not really sold on adding an extra call. I think the fewer calls you need to make to get the env setup to start communicating the better.

I guess for versions I would just change the begin call. i.e. create a BEGIN_FIREBASE2 if we ever need to implement a breaking change.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What I liked about the API prelude is that it makes it feels declarative. Like an #include or an import. Later we might add some extra calls that allow you to manage the lifecycle of the APIs loaded in your chiplet: list version, OTA, etc.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alright I am on board with something like that. We can have a concept of "default api" that will load at startup.

So for now we can just leave this out all together and address it later on when we introduce other APIs or versions.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alright I am on board with something like that. We can have a concept of "default api" that will load at startup.

Yes, so for now I think it's fine to just have BEGIN url

INIT $Host $Secret
###Response
OK - Accepted initialization parameters
###Examples
>> INIT https://samplechat.firebaseio-demo.com
<< OK
>> INIT https://samplechat.firebaseio-demo.com nnz...sdf
<< OK
##GET
Fetches the value, in json, at $Path and returns it on the serial line.
###Usage
GET $PATH
###Response
$DATA_BYTE_COUNT $DATA
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if that wouldn't be easier to just have it line delimited (since you can easily readline from Serial), if we want more fancy protocol framing we can have SPI and i2c transport.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You would have to escape newline in the value. Also it is nice to know how large the incoming message is so you can allocate a buffer for it.

I guess it depends whether the common case is just grabbing an int value or grabbing data blobs or text.

So what I currently use as a backend for my IoT projects in redis mostly because it has a very simple wire protocol that I can hack a library for in O(minutes). http://redis.io/topics/protocol

They use the first byte of the response to dictate a type, being error, string, blob(bulkstring). I like the idea to address this problem as only large data types benefit from complexity like a size. But for those data types it is far easier to deal with them if you have the size ahead of time.

We could just use their protocol, which would be nice for compatibility.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They use the first byte of the response to dictate a type, being error, string, blob(bulkstring).

Nice! We used something similar for error and logging in the [old serial bridge] 566c303#diff-dc3ebc3d309c06217829e6e5ed607be0R71, everything that started with a ! could be ignored.

I'm wondering if we need to "always" prefix the type since the chiplet is meant to be used directly with Serial.readStringUntil and it's like that you already know if you want to read a json number, a string or a json body depending on the path you're GETing.

It would look something like:

Serial.println("GET /foo");
float foo = atof(Serial.readStringUntil("\n").c_str());

But maybe I'm being too naive.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ya I think this is getting back to two modes of operation. The "store a simple value" and "Store a binary blob or text".

Maybe we should just separate them and have GET return the value alone but have a GET_BULK that will return the value prefixed by it's size.

Eitherway when I think about it your example probably isn't what we want to encourage though. If we return an error than that will crash the board. Maybe we should have the forward prefix that you should always check and strip off so it would be like:
Serial.println("GET /foo");
if (Serial.read() == "!") {
// handle error
}
float foo = atof(Serial.readStringUntil("\n").c_str());

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we should have the forward prefix that you should always check and strip off.

Yes, you're right and if we end up prefixing the the result we might as well use that prefix a type hint: sold!

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

alright updated with prefixes and bulk mode.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Im thinking that the BULK mode would be better as a stateful MODE rather than a separate comment. Since you're likely to always want one or the other.

re: prefix only on error: just for completeness of the argument what do think of the following client code:

String val = Serial.readStringUntil('\n');
If (val[0] == '!') {
  // error
}
float n = atof(val.c_str());

Maybe the type prefix could come as a separate stateful mode, since you're likely to always want them or never.

###Examples
>>GET /user/aturing
<<39 { "first" : "Alan", "last" : "Turing" }

##GET_VALUE
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@proppy
First place that might be controversial. I think it is important to not force the user to parse json. I would almost be inclined to not include the primary "get" at all. But I could see it as useful for some users.

I feel that a goal of this project is to provide cloud backends for non-cloud users who have sofar stayed local or offline due to a reluctance to face the learning curve of cloud tech. json isn't a common protocol in the embedded world and is relatively hard/expensive to decode.

Fetches the value stored $Path and returns it on the serial line. $PATH must point to a leaf node in the tree as this call returns one value but does so without json wrapper.
###Usage
GET_VALUE $PATH
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you think of having GET yield value if it's a leaf node and json otherwise. Most client are likely to know what they are reading: if that's a leaf node they can atoi and if that's not they can JsonStaticBuffer::parseObject.

For you second suggestion (disabling json by default):
you could have a stateful ENABLE JSON command that turn on json on GET.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, I didn't realize that was how the REST interface already worked, I thought it always returned json. Ya if that will match the REST interface I am on board.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, I didn't realize that was how the REST interface already worked,

Oups, sorry for the missunderstanding: the REST interface always return JSON, what I'm saying is that here we could return something closer to what the user is waiting for, i.e: if he is querying a leaf string or number, it's unlikely he want to unquote it himself.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does it?
If I do:
curl -X GET 'https://edcoyne-iot.firebaseio.com/logs/-K94eMV35X3g4scJlA8K.json?auth=$AUTH_KEY

(from the example app) I just get a raw integer back. I guess that raw integer could be valid json but it is not formatted in any special way.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, raw integer is a valid json number, but everything json raw string needs to be "double quoted".

###Response
$DATA_BYTE_COUNT $DATA
ERROR_NOT_LEAF_NODE
###Examples
>>GET_VALUE /user/aturing/first
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

s/GET_VALUE/GET/g

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

<<4 Alan
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

another issue with having the size as TEXT is that is also a variable length unless we do some padding.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

true but you can just read until whitespace. there is also Serial.parseInt()

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Then what's the improvement over reading until \n? That you're guaranteed the size will never cross a given threshold (a few digits).

Maybe this could come in the same kind of stateful MODEs than type prefix and forced json strings?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you know on the outset the size of the value coming back you can prepare for it. For example you can allocate a buffer large enough to hold it without having to allocate a smaller buffer and grow it. You can also quickly decide if something is too large to process and close the connection.

We could also return text without the json escaping which might be nice so the user doesn't need to unescape it themselves. For example if they wanted to send it directly to a display.

>>GET_VALUE /user/aturing/last
<<7 Turing
>>GET_VALUE /user/aturing
<<ERROR_NOT_LEAF_NODE

##Set
##Push
##Remove
##Stream