Skip to content

Make ready_to_execute channel more general #22

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 7 commits into from
May 15, 2024

Conversation

Marenz
Copy link
Contributor

@Marenz Marenz commented Apr 4, 2024

  • Rename ready_to_execute channel to running_status_change reflecting it's broader purpose.
  • Small re-designs and changes about the interfaces in general
  • Add own Dispatch type with useful functions on top of the original base class

@github-actions github-actions bot added part:tests Affects the unit, integration and performance (benchmarks) tests part:actor Affects the dispatching actor labels Apr 4, 2024
@Marenz Marenz force-pushed the interface-updates branch from 1988d99 to 3dc5439 Compare April 10, 2024 17:30
@Marenz Marenz force-pushed the interface-updates branch from 3dc5439 to 4521cac Compare April 17, 2024 17:21
@github-actions github-actions bot added the part:tooling Affects the development tooling (CI, deployment, dependency management, etc.) label Apr 17, 2024
@Marenz Marenz force-pushed the interface-updates branch 3 times, most recently from 6af21be to 62bc687 Compare April 25, 2024 16:59
@llucax
Copy link
Contributor

llucax commented Apr 29, 2024

I see this was updated recently. Is it still a thing? Please let me know if you need more input from me.

@Marenz
Copy link
Contributor Author

Marenz commented Apr 29, 2024

This is very much a thing, one that will be updated now that the dependencies are merged :)

@Marenz
Copy link
Contributor Author

Marenz commented Apr 29, 2024

But apart from rebasing, the code is ready for review

@Marenz Marenz force-pushed the interface-updates branch from 62bc687 to fbb9f21 Compare April 29, 2024 10:14
@github-actions github-actions bot added the part:docs Affects the documentation label Apr 29, 2024
@Marenz Marenz marked this pull request as ready for review April 29, 2024 10:16
@Marenz Marenz requested a review from a team as a code owner April 29, 2024 10:16
@Marenz Marenz enabled auto-merge April 29, 2024 10:31
Returns:
True if the dispatch is currently meant to be running, False otherwise.
"""
if self.type != type_:
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am just thinking, maybe this should return None for this case, as in "Information not available"

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or maybe a three-state enum?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about raising an exception? I think a 3-state enum will be annoying to use and None will be, for most cases, the same as using False, as people will be still able to use it like if dispatch.running(): ...

If type changes are not allowed by the API this could only happen due to bugs or intentionally bad players, right?

If you don't like the exception approach I guess returning None might be the most convenient option, if it is relevant to anyone, they can do a if dispatch.running() is None: and if you don´t care you can use it as a bool (the only issue is it will probably lead to people not thinking about the None option at all, which could potentially mask issues).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mmm, looking at the type_ argument I'm more lost than before, this interface looks very confusing to me. Why are we checking against a type passed by the user at all? Can't the user check this by itself if they are interested in it?

Copy link
Contributor Author

@Marenz Marenz Apr 29, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Every actor receives every dispatch, so they have to check if it is for them.

To make sure they have to check, the running() method requires the type and if the types don't match no action at all is expected.

If the users don't handle the three-state enum, they instead need to filter the dispatch. Seems at the end the same amount of work to me, only that this way forces the user and makes it impossible to forget

Copy link
Contributor

@llucax llucax Apr 30, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably for a separate PR, but I was thinking that maybe this is the wrong approach. If we know any users will only be interested in a sub-set of dispatch types, then it would probably be better to have an interface like:

class FcrDispatch(Dispatch):
    def __init__(self, dispatch: Dispatch) -> None:
        ...
        # Probably parsing the payload here too, using pydantic or some other schema
        # we could even offer tools for this
        super().__init__(dispatch, FcrPayloadSchema())

class OtherDispatch(Dispatch):
    ...
        
async for dispatch in dispatcher.active.new_receiver({"FCR": FcrDispatch, "OTHER": OtherDispatch}):
    match dispatch:
        case FcrDispatch() as fcr_dispatch:
            ...
        case OtherDispatch() as other_dispatch:
            ...
        case _ as unhandled:
            assert_never(unhandled)

This way you are forced to check the type, and you can have a more specific interface for the types you care about.

So the new_receiver() method would become more complex, it needs to do the filtering and the conversion of the general Dispatch type to the specific one provided by the user, and be generic. It might be a challenge to do it properly. I guess worse case we can always only just do it with a simple T and then use: dispatcher.active.new_receiver[FcrDispatch | OtherDispatch]({"FCR": FcrDispatch, "OTHER": OtherDispatch}) which is a bit too verbose, but well.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So we decided to go with this approach, and also the class approach on top of it, so simple dispatch executing actors should be trivial to implement, and if you need something a bit more complex, there is still a pretty high-level interface that takes most of the complexity out.

But we'll do this in a separate PR. @Marenz do you want to create a new issue with this info or do you plan to start implementing it immediately after this is merged?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll implement it right a way one FCR dispatching is up and running

@Marenz Marenz force-pushed the interface-updates branch from 3e298c5 to 1a15050 Compare April 29, 2024 11:37
Copy link
Contributor

@llucax llucax left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like this in general, but I have a few minor comments and some more fundamental, like about the running() function and the channel name.

Comment on lines +13 to +15
from frequenz.client.dispatch.types import Dispatch as BaseDispatch
from frequenz.client.dispatch.types import Frequency, Weekday
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(unrelated) I don't remember the rationale about using .types here but to me it is still funny-looking that it is not just from frequenz.client.dispatch import ....

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also unrelated and just purely brainstorming, but it we reversed the namespaces here, it might be more natural to use with google style imports:

from frequenz.dispatch import client

...

class Dispatch(client.Dispatch):
    ...

Returns:
True if the dispatch is currently meant to be running, False otherwise.
"""
if self.type != type_:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about raising an exception? I think a 3-state enum will be annoying to use and None will be, for most cases, the same as using False, as people will be still able to use it like if dispatch.running(): ...

If type changes are not allowed by the API this could only happen due to bugs or intentionally bad players, right?

If you don't like the exception approach I guess returning None might be the most convenient option, if it is relevant to anyone, they can do a if dispatch.running() is None: and if you don´t care you can use it as a bool (the only issue is it will probably lead to people not thinking about the None option at all, which could potentially mask issues).

"""Mark the dispatch as deleted."""
object.__setattr__(self, "deleted", True)

def running_status_notified(self) -> bool:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not @property?

And this also looks like is not intended to be used by the end user. If this is the case, I would make it private.

Comment on lines 129 to 135
Example: Creating a new dispatch and then modifying it. Note that this uses
the lower-level `Client` class to create and update the dispatch.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you check how this documentation renders? I recommend using a shorter title anyway, and adding a longer description as part of the example:

Suggested change
Example: Creating a new dispatch and then modifying it. Note that this uses
the lower-level `Client` class to create and update the dispatch.
Example: Creating a new dispatch and then modifying it
Note that this uses the lower-level `Client` class to create and update the dispatch.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also pending, it should be a quick fix.

# Create a new dispatch
new_dispatch = await dispatcher.client.create(
microgrid_id=microgrid_id,
_type="ECHO_FREQUENCY", # replace with your own type
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BTW, why _type? type is not a reserved word, flake8/pylint triggers a warning if it is used by I think we should silence it and use it anyway in this case.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.


@property
def ready_to_execute(self) -> ReceiverFetcher[Dispatch]:
"""Return ready dispatches receiver.
def running_status_change(self) -> ReceiverFetcher[Dispatch]:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not happy about this name, since we are streamming Dispatch objects, I think it should be named accordingly, like which types of dispatch objects are put here. I think currently running dispatches are put here, or active dispatches.

So what about either just active, or currently_active? Either that, or we call it something like _events like lifecycle_events and wrap the dispatches inside some other object that tells which kind of update is it.

For me it looks simpler to just use active, but it might be actually misleading, as it is actually a sort of stream of status changes as you say, but then we should have some wrapper object and track the changes and report the changes instead of just sending the dispatch object itself. I'm really unsure about this though, as it is a lot of work and the actual current needs are pretty well covered by this interface, so maybe it makes more sense to go with the simple active approach and clarify in the docs, it seems like no name will be clear and good enough to convey all that users will need to do to properly handle dispatches that are changing while running.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Active seems wrong as well, given that it also sends deleted and specifically inactive dispatches, too...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

like which types of dispatch objects are put here. I think currently running dispatches are put here, or active dispatches.

Any "type" that makes a receiver start, stop or reconfigure.

So what about either just active, or currently_active?

It includes deleted, inactive and finished dispatches, as well.

or we call it something like _events like lifecycle_events

We could call it running_status_events 🤷

and wrap the dispatches inside some other object that tells which kind of update is it.

Not sure what help that is, there is no specific "kind" or "type" of update.
Well, or if you want, you could categorize it as

  1. request to start
  2. request to stop
  3. request to use different configuration

all those cases are covered by the simple usage example already shown, basically for 1) and 3) you check if you're running, if yes, reconfigure, if no, start. (or stop if it's 2)

Copy link
Contributor

@llucax llucax Apr 30, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It includes deleted, inactive and finished dispatches, as well.

Yeah, but considering both the previous and new state, then the dispatch was active (either it needs to be active now, or it was active when the change was received). It is a broad definition of active but I think it could still work given that it is very complex what we are trying to describe and no sensible symbol name will be able to fully describe it.

We could call it running_status_events 🤷

Yeah, sure, but in that case I don't think we should stream Dispatch objects, we should stream some RunningStatusEvent, which makes the interface more complicated, because what do you put there?

Not sure what help that is, there is no specific "kind" or "type" of update. Well, or if you want, you could categorize it as

1. request to start
2. request to stop
3. request to use different configuration

Mmmm, maybe. For some reason I'm not convinced to add so much meaning to an update. What we are really receiving is an update to the dispatch, which could end up in those 3 points or not. But I don't know exactly why I don't like this mapping, maybe is just a general feeling of leaking abstractions, maybe you are right and this approach is OK.

But then, when the dispatch finishes normally you won't get a notification this way, right? Or will you? I thought we only send something here if the duration changed, or it was disabled, or something else that makes the dispatch not be active anymore, but users should still make sure the dispatch is stopped after the duration even if they don't receive any notification, right? My fear here is that if users believe they will always be notified about a stopping event, then they will not look into the duration anymore.

Actually thinking about it, the actor could send a stop request and keep the timer(s) itself to trigger the event after the duration. I guess the only risk here is the dispatch actor crashing and the users not being notified about the stop. We could potentially handle this correctly and recover the timers after the crash, so only an infinite restart loop could a real problem, but maybe in that case is reasonable to assume everything went to hell.

Another issue with handling the duration timer in the dispatch actor is that we'll have to set a timer for every dispatch, even if nobody cares about it. Let's say for example a FCR dispatch has an invalid payload parameter, so we don't run it, we'll still have a timer in the dispatch actor for it. If we combine it with the feature of automatically filtering by type in the Dispatcher, then I guess this problem is not that bad, as it should be rare that someone requested for a type of Dispatch and then they don't really run it.

OK, if we go this route I think it could work: We have running_status_events, which streams something like Start | Stop | Update events, we provide type filtering via the new_receiver() and we make the actor handle the dispatch timer. This will make the dispatch actor much more complex, but I guess it might be worth it if we remove that complexity from the user...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But then, when the dispatch finishes normally you won't get a notification this way, right?

correct.

Or will you? I thought we only send something here if the duration changed, or it was disabled, or something else that makes the dispatch not be active anymore, but users should still make sure the dispatch is stopped after the duration even if they don't receive any notification, right? My fear here is that if users believe they will always be notified about a stopping event, then they will not look into the duration anymore.

Though I feel we could offer this as a client side ActorDispatcher class like here so that the end-user would really just have the start() and stop() and then reconfigure() or so functions to care about.

Actually thinking about it, the [..]

Well, or we go that way, of course. I do like the new_receiver(filter) idea, that makes whatever I've been doing with running() obsolete


self._dispatches[dispatch.id] = Dispatch(client_dispatch)
old_dispatch = old_dispatches.pop(dispatch.id, None)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Be careful with this, if the actor crashes, you might lose this dispatch in the list of old_dispatches, not sure if it would be a problem or not.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm it might not be a problem to just re-run all dispatches that are currently supposed to be running?

README.md Outdated
Comment on lines 13 to 15
* [Dispatcher][frequenz.dispatch.Dispatcher]: The entry point for the API.
* [Dispatch][frequenz.dispatch.Dispatch]: A dispatch type with lots of useful extra functionality.
* [Created][frequenz.dispatch.Created], [Updated][frequenz.dispatch.Updated], [Deleted][frequenz.dispatch.Deleted]: Dispatch event types.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You shouldn't use cross-reference syntax in the README, it doesn't render well in GitHub, which will be the primary place where this is seen.

image

I recommend to only provide an example in the README and then pointing to the generated documentation for more details. This kind of overview should be in the User Guide IMHO (i.e. probably the __init__.py docstring).

You can see the channels repo as an example to how to structure a user guide where most of the docs still lives in the docstrings, so it is still accessible in code editors.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed links

Returns:
True if the dispatch is currently meant to be running, False otherwise.
"""
if self.type != type_:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So we decided to go with this approach, and also the class approach on top of it, so simple dispatch executing actors should be trivial to implement, and if you need something a bit more complex, there is still a pretty high-level interface that takes most of the complexity out.

But we'll do this in a separate PR. @Marenz do you want to create a new issue with this info or do you plan to start implementing it immediately after this is merged?

return self._until(now)

@property
def missed_runs(self) -> Iterator[datetime]: # noqa: DOC405
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe? 🙏 😆

"""
return self.running_state_change_synced == self.update_time

def _set_running_status_notified(self) -> None:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
def _set_running_status_notified(self) -> None:
@_running_status_notified.setter
def _running_status_notified(self) -> None:

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure how I'd call this setter given that I provide no value...

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, sorry, I missed there was no argument. I think it would be better to either take an argument or maybe rename to something like _mark_as_sent or something conveying what this is actually doing better. But it is a minor thing if we plan to change this in the future, I'm fine with merging as is if it is going to be changed soon.

Comment on lines 129 to 135
Example: Creating a new dispatch and then modifying it. Note that this uses
the lower-level `Client` class to create and update the dispatch.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also pending, it should be a quick fix.

@llucax llucax changed the title Dispatch python interface change suggestion Make ready_to_execute channel more general. May 3, 2024
@llucax
Copy link
Contributor

llucax commented May 3, 2024

@Marenz I updated the title to make it more suitable as a commit message. Can you please also update the PR description to be suitable as a commit message body? This will end up in the merge commit.

@Marenz Marenz force-pushed the interface-updates branch from f671bc8 to f80fef2 Compare May 6, 2024 16:28
llucax
llucax previously approved these changes May 7, 2024
Copy link
Contributor

@llucax llucax left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd be happy if the noqa comment is updated, but all are minor comments, so approving in case you want to merge as-is.

"""
return self.running_state_change_synced == self.update_time

def _set_running_status_notified(self) -> None:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, sorry, I missed there was no argument. I think it would be better to either take an argument or maybe rename to something like _mark_as_sent or something conveying what this is actually doing better. But it is a minor thing if we plan to change this in the future, I'm fine with merging as is if it is going to be changed soon.

return self._until(now)

@property
def missed_runs(self) -> Iterator[datetime]: # noqa: DOC405
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See the new comment. We should probably also report the issue, it shouldn't be hard to fix I guess, just treating a return without a value specially.

@Marenz Marenz added this pull request to the merge queue May 7, 2024
@Marenz Marenz removed this pull request from the merge queue due to a manual request May 7, 2024
@Marenz Marenz force-pushed the interface-updates branch from 8309f42 to 4e601cd Compare May 7, 2024 10:23
@Marenz Marenz enabled auto-merge May 7, 2024 10:23
@Marenz Marenz disabled auto-merge May 8, 2024 07:56
@Marenz Marenz mentioned this pull request May 8, 2024
@Marenz Marenz force-pushed the interface-updates branch from 4e601cd to 4810bb4 Compare May 8, 2024 17:26
@Marenz
Copy link
Contributor Author

Marenz commented May 13, 2024

@llucax anything missing for a merge? :)

@Marenz Marenz enabled auto-merge May 14, 2024 07:28
@Marenz Marenz disabled auto-merge May 14, 2024 07:28
@llucax
Copy link
Contributor

llucax commented May 14, 2024

Oh, sorry, I didn't see it was updated, will have a look now.

Copy link
Contributor

@llucax llucax left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The typo in the comment is super minor but very confusing if left as is, so ideally fix that and we are set.

Comment on lines +15 to +18
The [`Dispatcher` class](https://frequenz-floss.github.io/frequenz-dispatch-python/v0.1/reference/frequenz/dispatch/#frequenz.dispatch.Dispatcher), the main entry point for the API, provides two channels:

* [Lifecycle events](https://frequenz-floss.github.io/frequenz-dispatch-python/v0.1/reference/frequenz/dispatch/#frequenz.dispatch.Dispatcher.lifecycle_events): A channel that sends a message whenever a [Dispatch][frequenz.dispatch.Dispatch] is created, updated or deleted.
* [Running status change](https://frequenz-floss.github.io/frequenz-dispatch-python/v0.1/reference/frequenz/dispatch/#frequenz.dispatch.Dispatcher.running_status_change): Sends a dispatch message whenever a dispatch is ready to be executed according to the schedule or the running status of the dispatch changed in a way that could potentially require the actor to start, stop or reconfigure itself.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

About these links, we were using redirect for version aliases because GitHub didn't support symlinks. With redirects we can't use direct links to an alias, but with symlinks we can. You could update the alias_type configuration from redirect to symlink and then link to the latest alias for example, instead of v0.1.

That said, any linking to any particular version will be bad as things are now, because the README is also rendered as the home page of the project, and if you are visiting the docs for any particular version, the links in that version home page will point to a different version.

This is why I'm more inclined to not provide deep links to docs in the README, just explain in plain text and add a link to the docs home page (without any version specification, just https://frequenz-floss.github.io/frequenz-dispatch-python/) for people looking for more info.

That, or stop including the README entirely in the docs/index.md and do some partial inclusions (and leave stuff with deep links to docs out of the included parts) like we do with channels:

Happy to leave this for a future PR though.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's do that in the next iteration that we already plan :) I am afraid to add more stuff here currently

Marenz added 7 commits May 14, 2024 12:25
Signed-off-by: Mathias L. Baumann <[email protected]>
Signed-off-by: Mathias L. Baumann <[email protected]>
When you want to manipulate dispatches, you will need direct access
to the client.

Signed-off-by: Mathias L. Baumann <[email protected]>
Signed-off-by: Mathias L. Baumann <[email protected]>
They are meant to be used only by the dispatch actor.

Signed-off-by: Mathias L. Baumann <[email protected]>
@Marenz Marenz force-pushed the interface-updates branch from 4810bb4 to addb81d Compare May 14, 2024 10:25
@Marenz Marenz enabled auto-merge May 14, 2024 10:26
@Marenz Marenz requested a review from llucax May 14, 2024 10:56
@llucax llucax changed the title Make ready_to_execute channel more general. Make ready_to_execute channel more general May 15, 2024
@Marenz Marenz added this pull request to the merge queue May 15, 2024
Merged via the queue into frequenz-floss:v0.x.x with commit 69a72ec May 15, 2024
14 checks passed
@Marenz Marenz deleted the interface-updates branch May 15, 2024 08:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
part:actor Affects the dispatching actor part:docs Affects the documentation part:tests Affects the unit, integration and performance (benchmarks) tests part:tooling Affects the development tooling (CI, deployment, dependency management, etc.)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants