-
Notifications
You must be signed in to change notification settings - Fork 5
Make ready_to_execute
channel more general
#22
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
1988d99
to
3dc5439
Compare
3dc5439
to
4521cac
Compare
6af21be
to
62bc687
Compare
I see this was updated recently. Is it still a thing? Please let me know if you need more input from me. |
This is very much a thing, one that will be updated now that the dependencies are merged :) |
But apart from rebasing, the code is ready for review |
62bc687
to
fbb9f21
Compare
Returns: | ||
True if the dispatch is currently meant to be running, False otherwise. | ||
""" | ||
if self.type != type_: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am just thinking, maybe this should return None
for this case, as in "Information not available"
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Or maybe a three-state enum?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What about raising an exception? I think a 3-state enum will be annoying to use and None
will be, for most cases, the same as using False
, as people will be still able to use it like if dispatch.running():
...
If type changes are not allowed by the API this could only happen due to bugs or intentionally bad players, right?
If you don't like the exception approach I guess returning None
might be the most convenient option, if it is relevant to anyone, they can do a if dispatch.running() is None:
and if you don´t care you can use it as a bool
(the only issue is it will probably lead to people not thinking about the None
option at all, which could potentially mask issues).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Mmm, looking at the type_
argument I'm more lost than before, this interface looks very confusing to me. Why are we checking against a type passed by the user at all? Can't the user check this by itself if they are interested in it?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Every actor receives every dispatch, so they have to check if it is for them.
To make sure they have to check, the running()
method requires the type and if the types don't match no action at all is expected.
If the users don't handle the three-state enum, they instead need to filter the dispatch. Seems at the end the same amount of work to me, only that this way forces the user and makes it impossible to forget
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Probably for a separate PR, but I was thinking that maybe this is the wrong approach. If we know any users will only be interested in a sub-set of dispatch types, then it would probably be better to have an interface like:
class FcrDispatch(Dispatch):
def __init__(self, dispatch: Dispatch) -> None:
...
# Probably parsing the payload here too, using pydantic or some other schema
# we could even offer tools for this
super().__init__(dispatch, FcrPayloadSchema())
class OtherDispatch(Dispatch):
...
async for dispatch in dispatcher.active.new_receiver({"FCR": FcrDispatch, "OTHER": OtherDispatch}):
match dispatch:
case FcrDispatch() as fcr_dispatch:
...
case OtherDispatch() as other_dispatch:
...
case _ as unhandled:
assert_never(unhandled)
This way you are forced to check the type, and you can have a more specific interface for the types you care about.
So the new_receiver()
method would become more complex, it needs to do the filtering and the conversion of the general Dispatch
type to the specific one provided by the user, and be generic. It might be a challenge to do it properly. I guess worse case we can always only just do it with a simple T
and then use: dispatcher.active.new_receiver[FcrDispatch | OtherDispatch]({"FCR": FcrDispatch, "OTHER": OtherDispatch})
which is a bit too verbose, but well.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So we decided to go with this approach, and also the class approach on top of it, so simple dispatch executing actors should be trivial to implement, and if you need something a bit more complex, there is still a pretty high-level interface that takes most of the complexity out.
But we'll do this in a separate PR. @Marenz do you want to create a new issue with this info or do you plan to start implementing it immediately after this is merged?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'll implement it right a way one FCR dispatching is up and running
3e298c5
to
1a15050
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I like this in general, but I have a few minor comments and some more fundamental, like about the running()
function and the channel name.
from frequenz.client.dispatch.types import Dispatch as BaseDispatch | ||
from frequenz.client.dispatch.types import Frequency, Weekday |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
(unrelated) I don't remember the rationale about using .types
here but to me it is still funny-looking that it is not just from frequenz.client.dispatch import ...
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also unrelated and just purely brainstorming, but it we reversed the namespaces here, it might be more natural to use with google style imports:
from frequenz.dispatch import client
...
class Dispatch(client.Dispatch):
...
Returns: | ||
True if the dispatch is currently meant to be running, False otherwise. | ||
""" | ||
if self.type != type_: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What about raising an exception? I think a 3-state enum will be annoying to use and None
will be, for most cases, the same as using False
, as people will be still able to use it like if dispatch.running():
...
If type changes are not allowed by the API this could only happen due to bugs or intentionally bad players, right?
If you don't like the exception approach I guess returning None
might be the most convenient option, if it is relevant to anyone, they can do a if dispatch.running() is None:
and if you don´t care you can use it as a bool
(the only issue is it will probably lead to people not thinking about the None
option at all, which could potentially mask issues).
src/frequenz/dispatch/_dispatch.py
Outdated
"""Mark the dispatch as deleted.""" | ||
object.__setattr__(self, "deleted", True) | ||
|
||
def running_status_notified(self) -> bool: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why not @property
?
And this also looks like is not intended to be used by the end user. If this is the case, I would make it private.
src/frequenz/dispatch/_dispatcher.py
Outdated
Example: Creating a new dispatch and then modifying it. Note that this uses | ||
the lower-level `Client` class to create and update the dispatch. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Did you check how this documentation renders? I recommend using a shorter title anyway, and adding a longer description as part of the example:
Example: Creating a new dispatch and then modifying it. Note that this uses | |
the lower-level `Client` class to create and update the dispatch. | |
Example: Creating a new dispatch and then modifying it | |
Note that this uses the lower-level `Client` class to create and update the dispatch. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also pending, it should be a quick fix.
# Create a new dispatch | ||
new_dispatch = await dispatcher.client.create( | ||
microgrid_id=microgrid_id, | ||
_type="ECHO_FREQUENCY", # replace with your own type |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
BTW, why _type
? type
is not a reserved word, flake8
/pylint
triggers a warning if it is used by I think we should silence it and use it anyway in this case.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
|
||
@property | ||
def ready_to_execute(self) -> ReceiverFetcher[Dispatch]: | ||
"""Return ready dispatches receiver. | ||
def running_status_change(self) -> ReceiverFetcher[Dispatch]: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not happy about this name, since we are streamming Dispatch
objects, I think it should be named accordingly, like which types of dispatch objects are put here. I think currently running dispatches are put here, or active dispatches.
So what about either just active
, or currently_active
? Either that, or we call it something like _events
like lifecycle_events
and wrap the dispatches inside some other object that tells which kind of update is it.
For me it looks simpler to just use active
, but it might be actually misleading, as it is actually a sort of stream of status changes as you say, but then we should have some wrapper object and track the changes and report the changes instead of just sending the dispatch object itself. I'm really unsure about this though, as it is a lot of work and the actual current needs are pretty well covered by this interface, so maybe it makes more sense to go with the simple active
approach and clarify in the docs, it seems like no name will be clear and good enough to convey all that users will need to do to properly handle dispatches that are changing while running.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Active seems wrong as well, given that it also sends deleted and specifically inactive dispatches, too...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
like which types of dispatch objects are put here. I think currently running dispatches are put here, or active dispatches.
Any "type" that makes a receiver start, stop or reconfigure.
So what about either just active, or currently_active?
It includes deleted, inactive and finished dispatches, as well.
or we call it something like _events like lifecycle_events
We could call it running_status_events
🤷
and wrap the dispatches inside some other object that tells which kind of update is it.
Not sure what help that is, there is no specific "kind" or "type" of update.
Well, or if you want, you could categorize it as
- request to start
- request to stop
- request to use different configuration
all those cases are covered by the simple usage example already shown, basically for 1) and 3) you check if you're running, if yes, reconfigure, if no, start. (or stop if it's 2)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It includes deleted, inactive and finished dispatches, as well.
Yeah, but considering both the previous and new state, then the dispatch was active (either it needs to be active now, or it was active when the change was received). It is a broad definition of active but I think it could still work given that it is very complex what we are trying to describe and no sensible symbol name will be able to fully describe it.
We could call it
running_status_events
🤷
Yeah, sure, but in that case I don't think we should stream Dispatch
objects, we should stream some RunningStatusEvent
, which makes the interface more complicated, because what do you put there?
Not sure what help that is, there is no specific "kind" or "type" of update. Well, or if you want, you could categorize it as
1. request to start 2. request to stop 3. request to use different configuration
Mmmm, maybe. For some reason I'm not convinced to add so much meaning to an update. What we are really receiving is an update to the dispatch, which could end up in those 3 points or not. But I don't know exactly why I don't like this mapping, maybe is just a general feeling of leaking abstractions, maybe you are right and this approach is OK.
But then, when the dispatch finishes normally you won't get a notification this way, right? Or will you? I thought we only send something here if the duration changed, or it was disabled, or something else that makes the dispatch not be active anymore, but users should still make sure the dispatch is stopped after the duration even if they don't receive any notification, right? My fear here is that if users believe they will always be notified about a stopping event, then they will not look into the duration anymore.
Actually thinking about it, the actor could send a stop request and keep the timer(s) itself to trigger the event after the duration. I guess the only risk here is the dispatch actor crashing and the users not being notified about the stop. We could potentially handle this correctly and recover the timers after the crash, so only an infinite restart loop could a real problem, but maybe in that case is reasonable to assume everything went to hell.
Another issue with handling the duration timer in the dispatch actor is that we'll have to set a timer for every dispatch, even if nobody cares about it. Let's say for example a FCR dispatch has an invalid payload parameter, so we don't run it, we'll still have a timer in the dispatch actor for it. If we combine it with the feature of automatically filtering by type in the Dispatcher
, then I guess this problem is not that bad, as it should be rare that someone requested for a type of Dispatch and then they don't really run it.
OK, if we go this route I think it could work: We have running_status_events
, which streams something like Start | Stop | Update
events, we provide type filtering via the new_receiver()
and we make the actor handle the dispatch timer. This will make the dispatch actor much more complex, but I guess it might be worth it if we remove that complexity from the user...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
But then, when the dispatch finishes normally you won't get a notification this way, right?
correct.
Or will you? I thought we only send something here if the duration changed, or it was disabled, or something else that makes the dispatch not be active anymore, but users should still make sure the dispatch is stopped after the duration even if they don't receive any notification, right? My fear here is that if users believe they will always be notified about a stopping event, then they will not look into the duration anymore.
Though I feel we could offer this as a client side ActorDispatcher
class like here so that the end-user would really just have the start()
and stop()
and then reconfigure()
or so functions to care about.
Actually thinking about it, the [..]
Well, or we go that way, of course. I do like the new_receiver(filter)
idea, that makes whatever I've been doing with running()
obsolete
|
||
self._dispatches[dispatch.id] = Dispatch(client_dispatch) | ||
old_dispatch = old_dispatches.pop(dispatch.id, None) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Be careful with this, if the actor crashes, you might lose this dispatch in the list of old_dispatches
, not sure if it would be a problem or not.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hmm it might not be a problem to just re-run all dispatches that are currently supposed to be running?
69e4772
to
f671bc8
Compare
README.md
Outdated
* [Dispatcher][frequenz.dispatch.Dispatcher]: The entry point for the API. | ||
* [Dispatch][frequenz.dispatch.Dispatch]: A dispatch type with lots of useful extra functionality. | ||
* [Created][frequenz.dispatch.Created], [Updated][frequenz.dispatch.Updated], [Deleted][frequenz.dispatch.Deleted]: Dispatch event types. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You shouldn't use cross-reference syntax in the README, it doesn't render well in GitHub, which will be the primary place where this is seen.
I recommend to only provide an example in the README and then pointing to the generated documentation for more details. This kind of overview should be in the User Guide IMHO (i.e. probably the __init__.py
docstring).
You can see the channels repo as an example to how to structure a user guide where most of the docs still lives in the docstrings, so it is still accessible in code editors.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
fixed links
Returns: | ||
True if the dispatch is currently meant to be running, False otherwise. | ||
""" | ||
if self.type != type_: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So we decided to go with this approach, and also the class approach on top of it, so simple dispatch executing actors should be trivial to implement, and if you need something a bit more complex, there is still a pretty high-level interface that takes most of the complexity out.
But we'll do this in a separate PR. @Marenz do you want to create a new issue with this info or do you plan to start implementing it immediately after this is merged?
return self._until(now) | ||
|
||
@property | ||
def missed_runs(self) -> Iterator[datetime]: # noqa: DOC405 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe? 🙏 😆
""" | ||
return self.running_state_change_synced == self.update_time | ||
|
||
def _set_running_status_notified(self) -> None: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
def _set_running_status_notified(self) -> None: | |
@_running_status_notified.setter | |
def _running_status_notified(self) -> None: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am not sure how I'd call this setter given that I provide no value...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
OK, sorry, I missed there was no argument. I think it would be better to either take an argument or maybe rename to something like _mark_as_sent
or something conveying what this is actually doing better. But it is a minor thing if we plan to change this in the future, I'm fine with merging as is if it is going to be changed soon.
src/frequenz/dispatch/_dispatcher.py
Outdated
Example: Creating a new dispatch and then modifying it. Note that this uses | ||
the lower-level `Client` class to create and update the dispatch. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also pending, it should be a quick fix.
ready_to_execute
channel more general.
@Marenz I updated the title to make it more suitable as a commit message. Can you please also update the PR description to be suitable as a commit message body? This will end up in the merge commit. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'd be happy if the noqa
comment is updated, but all are minor comments, so approving in case you want to merge as-is.
""" | ||
return self.running_state_change_synced == self.update_time | ||
|
||
def _set_running_status_notified(self) -> None: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
OK, sorry, I missed there was no argument. I think it would be better to either take an argument or maybe rename to something like _mark_as_sent
or something conveying what this is actually doing better. But it is a minor thing if we plan to change this in the future, I'm fine with merging as is if it is going to be changed soon.
return self._until(now) | ||
|
||
@property | ||
def missed_runs(self) -> Iterator[datetime]: # noqa: DOC405 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
See the new comment. We should probably also report the issue, it shouldn't be hard to fix I guess, just treating a return
without a value specially.
@llucax anything missing for a merge? :) |
Oh, sorry, I didn't see it was updated, will have a look now. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The typo in the comment is super minor but very confusing if left as is, so ideally fix that and we are set.
The [`Dispatcher` class](https://frequenz-floss.github.io/frequenz-dispatch-python/v0.1/reference/frequenz/dispatch/#frequenz.dispatch.Dispatcher), the main entry point for the API, provides two channels: | ||
|
||
* [Lifecycle events](https://frequenz-floss.github.io/frequenz-dispatch-python/v0.1/reference/frequenz/dispatch/#frequenz.dispatch.Dispatcher.lifecycle_events): A channel that sends a message whenever a [Dispatch][frequenz.dispatch.Dispatch] is created, updated or deleted. | ||
* [Running status change](https://frequenz-floss.github.io/frequenz-dispatch-python/v0.1/reference/frequenz/dispatch/#frequenz.dispatch.Dispatcher.running_status_change): Sends a dispatch message whenever a dispatch is ready to be executed according to the schedule or the running status of the dispatch changed in a way that could potentially require the actor to start, stop or reconfigure itself. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
About these links, we were using redirect for version aliases because GitHub didn't support symlinks. With redirects we can't use direct links to an alias, but with symlinks we can. You could update the alias_type
configuration from redirect
to symlink
and then link to the latest
alias for example, instead of v0.1
.
That said, any linking to any particular version will be bad as things are now, because the README is also rendered as the home page of the project, and if you are visiting the docs for any particular version, the links in that version home page will point to a different version.
This is why I'm more inclined to not provide deep links to docs in the README, just explain in plain text and add a link to the docs home page (without any version specification, just https://frequenz-floss.github.io/frequenz-dispatch-python/) for people looking for more info.
That, or stop including the README entirely in the docs/index.md
and do some partial inclusions (and leave stuff with deep links to docs out of the included parts) like we do with channels:
- https://raw.githubusercontent.com/frequenz-floss/frequenz-channels-python/v1.x.x/README.md
- https://raw.githubusercontent.com/frequenz-floss/frequenz-channels-python/v1.x.x/docs/index.md
Happy to leave this for a future PR though.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let's do that in the next iteration that we already plan :) I am afraid to add more stuff here currently
Signed-off-by: Mathias L. Baumann <[email protected]>
Signed-off-by: Mathias L. Baumann <[email protected]>
When you want to manipulate dispatches, you will need direct access to the client. Signed-off-by: Mathias L. Baumann <[email protected]>
Signed-off-by: Mathias L. Baumann <[email protected]>
Signed-off-by: Mathias L. Baumann <[email protected]>
Signed-off-by: Mathias L. Baumann <[email protected]>
They are meant to be used only by the dispatch actor. Signed-off-by: Mathias L. Baumann <[email protected]>
ready_to_execute
channel more general.ready_to_execute
channel more general
ready_to_execute
channel torunning_status_change
reflecting it's broader purpose.Dispatch
type with useful functions on top of the original base class