[Do not merge] Add a test for custom bindings

[anthonychu]

Does the new support for raw data types allow bindings that are not known to the worker? I wasn't able to get the SignalRConnectionInfo binding working. Here's a test to show what I'm trying to do. Thanks.

[pragnagopa]

cc: @maiqbal11 @asavaritayal @anirudhgarg

@elprans - Can you please take a look? Should the dataType be set in function.json?

[elprans]

@pragnagopa Yes, for custom bindings the dataType should be set in function.json

[anthonychu]

@elprans Thanks. Do you have an example?

[elprans]

@anthonychu

azure-functions-python-worker/tests/blob_functions/get_blob_as_bytes/function.json

Lines 1 to 24 in b31ede8

	{
	"scriptFile": "main.py",
	"disabled": false,
	"bindings": [
	{
	"type": "httpTrigger",
	"direction": "in",
	"name": "req"
	},
	{
	"type": "blob",
	"direction": "in",
	"name": "file",
	"dataType": "binary",
	"connection": "AzureWebJobsStorage",
	"path": "python-worker-tests/test-bytes.txt"
	},
	{
	"type": "http",
	"direction": "out",
	"name": "$return"
	}
	]
	}

[anthonychu]

I tested this out and it doesn't work for signalRConnectionInfo because the binding returns an object and it doesn't convert to a string. I suspect other bindings that do not convert to string will have the same problems.

Is it possible for it to work like the JavaScript worker when the dataType is not specified and allow binding to a dict or list? Example:

    {
      "type": "signalRConnectionInfo",
      "direction": "in",
      "name": "connectionInfo",
      "hubName": "chat"
    }

def main(req: func.HttpRequest, connectionInfo: dict) -> func.HttpResponse:
      return func.HttpResponse(json.dumps(connectionInfo), mimetype='application/json')

[elprans]

because the binding returns an object and it doesn't convert to a string.

Can you elaborate here? On the protocol level everything is either a string or an array of bytes. It seems what you are looking for is the ability to say dataType: "json", but AFAIK, the host doesn't support that. /cc @pragnagopa

[anthonychu]

I believe an undefined dataType is essentially the same as dataType: "json". I think we should be able to handle this case for custom bindings. Here's what I'm thinking of:

If dataType is undefined:

• pytype is str: pass the JSON string
• pytype is dict: pass a dict if it can deserialize into one, else fail
• pytype is list: pass a list if it can deserialize into one, else fail
• pytype is None: pass the result of json.loads()

[elprans]

Undefined dataType does not actually imply JSON or any structure that the worker should be able to decode. The current handling is that an unspecified dataType means "use the rich binding if any". The current design does not require the use of type annotations in user functions, so we cannot rely on them, only on what's been specified in function.json. There has been a lengthy discussion about this in #66.

[anthonychu]

That makes sense. But even without dataType or type annotations, can we still perform a best effort conversion as I outlined above? That would make the Python implementation match the JavaScript one.

[elprans]

Matching the JavaScript implementation is not a goal. The Python implementation provides "rich" bindings by default, so the behavior is closer to C# rather than JavaScript. Doing an implicit json.loads() without an explicitly specified dataType in function.json prevents possible future implementation of a rich binding for an extension in a consistent way. We decided in #66 (and in other discussions around it) that any form of "generic" binding must require an explicit opt-in in function.json.

[pragnagopa]

@elprans - if dataType is set to string input can be a json string or just a string. Python worker needs to check if the input is valid json first and fall back to string otherwise. Is this possible?

@anthonychu - let me know if providing singnalRInfo as json would unblock your scenario.

[pragnagopa]

cc @jeffhollan for more context

[elprans]

Python worker needs to check if the input is valid json first and fall back to string otherwise.

@pragnagopa Why can't we add an explicit "json" setting? Implicit decoding attempt seems very fragile to me. Changing the type of the value based on the input (as opposed to an explicit declaration) isn't a good thing.

Unless we declare and document otherwise, the worker cannot rely on the type annotations. Otherwise we create a weird situation where it's not clear whether function.json is authoritative or the function declaration is.

[jeffhollan]

@anthonychu I imagine your goal here is to make sure the bindings behave even if the user doesn't explicitly set a dataType?

@elprans is the alternative here so that a blank / default dataType would work to provide a "rich" binding for the SignalR stuff @anthonychu is working on? I'm guessing a rich data type provides a bit more control for the binding to default to a desired type?

[anthonychu]

I think it would be okay to set a dataType, but I don't think string would be the right one because (I think) this requires every extension used in this manner to implement string converters. If there's no converter available, as in the case of the SignalR bindings, the host will throw an exception when dataType is set to string.

If dataType must be explicitly set, I like @elprans's suggestion of adding a new value such as json. The host would treat this exactly the same if it was undefined. But the Python worker would use it to opt into running json.loads() on the value.

[pragnagopa]

json is not a dataType. Functions host is built on WebJobs SDK which deserializes input to a Json object if the input data is valid json. Language workers are designed to be data converters between functions host and user code.

I think right approach is to use type annotations if the function expects a custom type such as
singnalRConnectionInfo which is not a basic type like: string or byte[] or rich binding type defined in azure-functions-python-library

def main(req: func.HttpRequest, connectionInfo: signalRConnectionInfo) -> func.HttpResponse:
    return func.HttpResponse(connectionInfo)

Python worker needs to convert the input data sent by the functions host to the type requested by the function. This is the goal of issue #135 as well.

Here is the flow for the above example:

• User function includes reference to the library where signalRConnectionInfo is defined
• On function invocation, host sends input data as String/Json
• Python worker examines the type annotation and converts input data to type signalRConnectionInfo
• If type conversion fails, throw error
  Note: This is the first step of adding support to live objects i.e convert blob input to type 'blob defined in azure storage python sdk

@elprans - Any suggestions on how above flow can be achieved?

[anthonychu]

Resolved with #409