Worker Indexing Pipeline

The Python Worker is being modified to enable it to perform indexing on a function app. This new functionality will enable Python developers to structure their function app directories in the way that they see fit.

Changes Needed For Implementation

New GRPC Messages

TODO: ADD LINK TO NEW MESSAGES IN PROTOBUF REPO ONCE PR IS APPROVED

The first step to implement this pipeline is to ensure that proper communication logic is put in place between the host and the worker. To do this, we need new GRPC messages to handle worker indexing requests and responses. Here are the messages that have been created to facilitate this communication:

WorkerFunctionMetadataRequest (host to worker)

Sends the function app directory to the worker.

{
    "directory": "some/script/path"
}

WorkerFunctionMetadataResponse (worker to host)

Includes a list of WorkerFunctionMetadata messages and an overall StatusResult attribute.

{
    "results": [
        {
            "name": "HttpTrigger1",
            "directory": "some/file/path1",
            "script_file": "some_file1"
        }, 
        {
            "name": "HttpTrigger2",
            "directory": "some/file/path2",
            "script_file": "some_file2"
        }
    ], 
    "overall_status": "success"
}

WorkerFunctionMetadata (worker to host)

Includes all information necessary to populate FunctionMetadata and convey the indexing status for each function (ie. success, failure, etc.). Each of these messages corresponds to a single function.

{
    "name": "HttpTrigger",
    "directory": "some/file/path",
    "script_file": "some_file",
    "entry_point": "some_entry_point",
    "id": "abc123",
    "bindings": [
        "binding1",
        "binding2"
    ],
    "is_proxy": "false",
    "status": "success",
    "language": "python"
}

---

New handler in dispatcher.py for incoming WorkerFunctionMetadataRequest messages

A new handler is needed to initiate the indexing process and return the function metadata through a new WorkerFunctionMetadataResponse message. A basic skeleton of the handler is shown below:

async def _handle__worker_function_metadata_request(self, req):
        metadata_request = req.worker_function_metadata_request
        directory = metadata_request.directory

        # get list of function metadata
        function_metadata = someFile.index_functions(directory)

        return protos.StreamingMessage(
            request_id=req.request_id,
            worker_function_metadata_response=protos.WorkerFunctionMetadataResponse(
                results=function_metadata,
                overall_status=protos.StatusResult(
                    status=protos.StatusResult.Success)))

---

New file to house the indexing logic (ie. indexer.py?)

• Function that can find the functions files
• Function that can scan through the functions files and find all metadata