Add support for informers

[macsux]

This PR adds support for informers that form building block for creating controllers. It follows the spirit of original informers, but differs in implementation found in Go and Java in a few significant ways.

• The entire library is built on reactive paradigm using Reactive Extensions. RX is a natural fit as we're dealing with streams of events, allowing leverage powerful built in functional folds over live streams of object change data. This offers advantages in:

  • intuitive LINQ like operators to filter / transform and collect live information being streamed when attached to k8s
  • reuse of single connection for internal multicasting
  • scheduler abstractions to provide multithreaded synchronization and concurrency
  • virtual time for unit testing

• Flexible abstraction model based on IInformer interface that allows modeling observable resources over resources other then K8S. This allows observing a SET of resources of similar type while conforming to same logical interface, easily allowing connecting one resource type to another. IInformer abstracts out a List/Watch operations into a single observable stream of events. Each event in the stream includes a bitvise enum mask that describes event purpose and how receivers of event should treat it, as well the current state of the object.

  • List operations are marked by event type "Reset", with first and last elements having ResetStart/ResetEnd flags. Reset also implies that any previously tracked cache state should be discard and new one used.
  • If list operation is empty and is followed by watch, ResetEmpty flag is used to denote transition point
  • Add/Modify/Delete flags denote changes to existing state
  • (Re)Sync events to denote periodic refreshes of same data already sent

  The intended vision is to allow reuseable set of informers for other APIs to be created and used as building blocks for creating controllers. For example a reusable informer can be written for twitter feed, leveraging push based websocket features. Legacy APIs without efficient change detection such an equivalent of Watch can be abstracted to provide push based events via polling.

• Concrete implementations of IInformer follow individual API rules, as well as provide reasonable failure recovery models.

• Generic List operation for Kubernates object that can work with any existing object or CRD.

  • Future enhancement will add similar features for modifying the state

• Included K8S informer provides supports

  • List followed by Watch
  • Fault tolerance policy with default backoff retry
  • Seamless connection recovery over transient error codes, including support for bookmarks
  • Server side support for namespace filtering
  • Future todo: server side label filtering

• Shared informer that is not K8S specific acts as an in-memory broadcaster and cache synchronizer. It follows lazy loading semantics, attaching multiple logical in-memory subscribers connection to single master Informer. This allows multiple controllers to share a single physical connection to the API server. Connection is automatically established when first subscription is made, and detached when it's dropped. List type queries for internal subscribers are simulated from local synchronized cache

• Shared informer factory that dynamically allocates specific shared informer based on resource type and server side query

• Support for shared versioned cache that allows resources that follow versioning semantics to be reused across multiple logical caches. This is useful when two informers are monitoring a subset of same resource where overlaps are possible (such as a server side label query), but one does not want to maintain two sets of objects with same key/version on the heap.

• Logical reactive fold helper that allows new caches to be created based on arbitrary key selector. This is an alternative to indexed caches that are used in Go implementation. This is a preferred model because it doesn't require complex global cache synchronization issues that exist with Go implementation.

• A functional fold for detection of discrepancies between relists and simulation of missed events. For example a List operation that returned [ObjectA.v1, ObjectB.v1] followed by disconnect and a relist that yields only [ObjectA.v1], it computes that the discrepency with local cache on ObjectB is a missed delete operation and changes the stream accordingly. This can be leveraged as needed simply by adding .ComputeMissedEventsBetweenResets extension method to observable.

• A worker with "events per object key" semantics, allowing processing of all events related to single event as a single unit of work. This offers similar functions as Go's DeltaFifo implementation. This is based on TPL Dataflows which are included in .NET Core and offer powerful in-memory actor model which can be used to develop additional queuing and processing semantics.

• Extensive unit test coverage

  • Virtual time based on RX test scheduler
  • Reusable scheduled event "scenarios" for writing theories
  • Improved test readability via use of NSubstitute and FluentAssertions.
  • Integration tests with mocking of real K8S via WireMock.NET. This is more powerful then existing mock object which doesn't currently support mocking multiple requests or streaming results.

• Sample informer project that demonstrates how to use informer as a starting point to write a simple controller

Things to note

• this PR bumps up required .NET Framework version to 4.6.1 as some of the libraries target NetStandard 2.0.
• The current .NET Kubernetes library is becoming quite monolithic and has too many dependencies. I propose that it be modularized into separate nuget packages. I'm happy to refactor informers into its own package if this is what maintainers want to do. The next phase for controllers will introduce the need for even more dependencies, so it makes sense to do this now. See sample project on where this could be taken

[macsux]

/assign @krabhishek8260

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: macsux
To complete the pull request process, please assign brendandburns
You can assign the PR to them by writing /assign @brendandburns in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[macsux]

@brendandburns wondering if there's anything I can do to help with reviewing this PR. i understand that this is a large chunk of code I created, so I would be happy to jump on a zoom call if you need a walkthrough.

[brendandburns]

Yeah I need to find an hour or two in my schedule to review the code :) I will try to block that out tomorrow

…

--brendan

________________________________ From: Andrew Stakhov <[email protected]> Sent: Tuesday, April 7, 2020 2:05:41 PM To: kubernetes-client/csharp <[email protected]> Cc: Brendan Burns <[email protected]>; Mention <[email protected]> Subject: Re: [kubernetes-client/csharp] Add support for informers (#394) @brendandburns<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fbrendandburns&data=02%7C01%7Cbburns%40microsoft.com%7C21966c9aeb9246f1414208d7db376e98%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637218903438875611&sdata=rL0miaJZM7XLPgl5MoVIyDeMl52IHf%2Fv3PQPPeHI0Fo%3D&reserved=0> wondering if there's anything I can do to help with reviewing this PR. i understand that this is a large chunk of code I created, so I would be happy to jump on a zoom call if you need a walkthrough. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fkubernetes-client%2Fcsharp%2Fpull%2F394%23issuecomment-610620219&data=02%7C01%7Cbburns%40microsoft.com%7C21966c9aeb9246f1414208d7db376e98%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637218903438875611&sdata=h0DRbthxbi1MDYfwH6oVpePe%2BeamdTSenJBfr1fcg9c%3D&reserved=0>, or unsubscribe<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FABL4HAQIWQSEFTPANATIOMDRLOISLANCNFSM4LYIPYWA&data=02%7C01%7Cbburns%40microsoft.com%7C21966c9aeb9246f1414208d7db376e98%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637218903438885611&sdata=Br%2F9xC%2FldaTucjjmb2K6pUERSatCezJ6btReq5gk3HI%3D&reserved=0>.

[tg123]

I am doing it too, but has to do after work :)

[tg123]

anyway to keep informer in the main package?
I do not think we could implement informer on net452

BTW, I do not want to support net45 either. but it is widely used.
the code would be even more clean if dropping old netframework support.
or stop supporting net452 when 3.0 (would be k8s 1.9) ? @brendandburns

[macsux]

I think it's not a good idea to try to keep everything in one package. This is consistent with core direction and moving away from monolithic releases. RX requires netstandard20 so there is no way to keep it in main package. There are more things coming down the pipe and new dependencies being introduced where developer should choose what they need. For example there is strong case to be made for DI extension methods, controller pieces which will probably be core only.

…

On Wed., Apr. 22, 2020, 7:47 a.m. Boshi Lian, ***@***.***> wrote: anyway to keep informer in the main package? I do not think we could implement informer on net452 BTW, I do not want to support net45 either. but it is widely used. the code would be even more clean if dropping old netframework support. or stop supporting net452 when 3.0 (would be k8s 1.9) ? @brendandburns <https://github.com/brendandburns> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#394 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAINFWCT47QBTBDOCRPAUDTRN3KLNANCNFSM4LYIPYWA> .

[k8s-ci-robot]

@macsux: PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[brendandburns]

Having this in a separate package makes sense to me. I'm going to try to allocate another hour or so to re-review this code once it gets rebased. It's a really big PR so it's challenging to find the time to give it an adequate review.

[macsux]

I've recently got agreement from project steeltoe.io which I contribute to and is mainly maintained by efforts from my current company (VMWare) to take this on as part of their codebase. The way I structured informers makes the codebase high-level LINQ abstraction over resources that can be watched for changes, with Kubernetes being just one of the potentially many implementations. Similar relationship as EF has with underlying drivers. Given a relatively loose dependency on Kubernetes, unless there's a strong desire to keep it here, I'm going to move this chunk over to steeltoe.io codebase with dependency on Kubernetes client.

The plan right now is to introduce two new packages as part of that codebase:
Steeltoe.Informers.InformersBase which will provide the core abstraction model
Steeltoe.Informers.KubernetesInformer which will provide Kubernetes specific implementation with dependency on KubernetesClient.

That codebase is already setup to be multi-package so its easier to integrate, and easier to introduce large code changes like this because of how the release cycle is structured.

I would like to hear your thoughts before I close this PR.

[brendandburns]

Obviously it's fine with me if you want to move this to a different package.

I think it's also possible to make it work within this repo and I'm happy to help get it there, but if you feel like the Steeltoe project is a better fit, then go for it.

fwiw, we will probably add informers to this package at some point, so we may look to the code in steeltoe for inspiration eventually.

[macsux]

I wrote an extensive blog post about idea behind informer. @brendandburns if you wanna read through it and let me know your thoughts - it should give you an idea of what this is trying to accomplish. It would be good to leave Kubernetes specific informer piece in here (as a driver) for the base informer framework.

[macsux]

Woops, forgot the link
https://stakhov.pro/informers-intro/

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[RichiCoder1]

Would it be possible to revive this PR in some fashion?

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[macsux]

This code will added as part of project SteelToe.io over the next few months. Main reason is that I'm using newer apis which prevent me from contributing them here. Second reason is that given the large scope of work, I need a release channel where people can try this work and provide feedback before api is finalized. Kubernetes client lacks stabilization branching strategy, which makes this contribution problematic. Finally the way I structured the codebase makes kubernetes just one of underlying implementations for what is essentially a 'live query' problem. In a sense kubernetes integration is almost like a driver for EF. I'll update this pr as I get closer to finishing it.

…

On Sat., Aug. 29, 2020, 6:09 p.m. Richard Simpson, ***@***.***> wrote: Would it be possible to revive this PR in some fashion? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#394 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAINFWHUNYK2CUZ5XNYJON3SDF37ZANCNFSM4LYIPYWA> .

[RichiCoder1]

This code will added as part of project SteelToe.io over the next few
months. Main reason is that I'm using newer apis which prevent me from
contributing them here. Second reason is that given the large scope of
work, I need a release channel where people can try this work and provide
feedback before api is finalized. Kubernetes client lacks stabilization
branching strategy, which makes this contribution problematic. Finally the
way I structured the codebase makes kubernetes just one of underlying
implementations for what is essentially a 'live query' problem. In a sense
kubernetes integration is almost like a driver for EF. I'll update this pr
as I get closer to finishing it.

Basically build it separately, stabilize, and then backport the k8s specific bits? Sounds good 👍. If you could post here when there's SteelToe bits available, I'd be interested!

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closed this PR.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[ddieruf]

So 10 months later, @macsux did you ever make any progress with the Steeltoe project?