Remove reliance on rsync

[nhooyr]

Right now we require rsync on the remote server.

[nhooyr]

(gcp vm's do not have rsync by default)

[ammario]

How else would this work?

[nhooyr]

Not sure, just would be nice to have no deps by default. Maybe we use one of the Go rsync libs?

[ammario]

That seems reasonable.. we don't need the advanced features of rsync that necessitate a remote binary.

Curious about how slow it woukd be to read all those remote modtimes/file sizes over SSH .

[nhooyr]

What if instead of rsync, we just mounted the local extensions folder?

[ammario]

Avg folder is 300MB and VS Code does a full scan of it on startup

[nhooyr]

In that case, I think requiring rsync to be installed for syncing is fair. Users who do not have it can use --skipsync and we can clarify the error message to indicate that.

[Belval]

FYI: Even with -skipsync, the connection will fail if the server does not have rsync.

[nhooyr]

@Belval can you post the error you're getting? I just looked at the code again, rsync is never executed on the remote site if -skipsync is passed.

[Belval]

I don't have my laptop right now, but it was something along the lines of rsync not found.

EDIT: I'll reply when I have the actual message.

[nhooyr]

Thanks, found and fixed.

[nhooyr]

Should be able to integrate mutagen here at some point to avoid using rsync.

See https://github.com/mutagen-io/mutagen/blob/cd6f4e1a5e68979668abfb22174c32d877e46000/cmd/mutagen/daemon/main.go

[nhooyr]

@ammario I know you recently worked with it so how was mutagen's API?

[ammario]

Didn't interface directly with Mutagen's API, but the architecture requires that the user has all the mutagen agents locally (it uploads the mutagen agent as the first step in the sync) which introduces its own complexity.

cc @havoc-io correct me if wrong.

[Merith-TK]

Oaky so I want to try to work on this concept. But can someone explain exactly what the frack "mutagen" is in this context?

…

On Mon, May 4, 2020, 7:20 AM Ammar Bandukwala ***@***.***> wrote: Didn't interface directly with Mutagen's API, but the architecture requires that the user has all the mutagen agents locally (it uploads the mutagen agent as the first step in the sync) which introduces its own complexity. cc @havoc-io <https://github.com/havoc-io> correct me if wrong. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#11 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACPQOXR7BLDCT2OMZZK5ZZ3RP3FJ7ANCNFSM4HHIT2CQ> .

[ammario]

@Merith-TK https://mutagen.io/ ... And please don't work on anything without an in depth design consensus with @nhooyr. We want to make sure you're as productive as possible.

[Merith-TK]

Don't worry, almost all of my code goes through a rough draft stage where I make it work. And a cleanup stage where I make it make bloody sense and follow what is actually needed.

…

On Mon, May 4, 2020, 8:40 AM Ammar Bandukwala ***@***.***> wrote: @Merith-TK <https://github.com/Merith-TK> https://mutagen.io/ ... And please don't work on anything without an in depth design consensus with @nhooyr <https://github.com/nhooyr>. We want to make sure you're as productive as possible. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#11 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACPQOXWHWFOMDQTJROJUZZLRP3OXRANCNFSM4HHIT2CQ> .

[xenoscopic]

@ammario In theory, you can avoid needing Mutagen's agent binaries locally if one is preinstalled at the expected location in the target ($HOME/.mutagen/agents/x.y.z/mutagen-agent, where x.y.z is the Mutagen client version). Mutagen will try to execute an existing agent first before injecting one, so you could potentially avoid the injection step and the agent bundle dependency. But Mutagen currently expects an exact version match because the internal protocol isn't stable across versions¹, so it will attempt to inject a matching version if none is present. If you're going to vendor/version-lock the Mutagen version that you ship, you might be able to use this strategy to skip the injection step.

If you do need to carry around the agent bundle locally, you can slim it down to only include the required agents (probably just linux/amd64 in your case). The archive just contains a bunch of binaries named <GOOS>_<GOARCH>, so a simple script can prune these and save a lot of space.

---

¹ Actually it is stable across minor release series, but a full version match, including the patch component and any -tag suffix is still required. This may change as the protocol further stabilizes.

[Merith-TK]

Maybe the mutagen client/server stuff can be acquired in the same way as how code-server binaries are aquired?

…

On Mon, May 4, 2020, 9:17 AM Jacob Howard ***@***.***> wrote: @ammario <https://github.com/ammario> In theory, you can avoid needing Mutagen's agent binaries locally if one is preinstalled at the expected location in the target ($HOME/.mutagen/agents/x.y.z/mutagen-agent, where x.y.z is the Mutagen client version). Mutagen will try to execute an existing agent first before injecting one, so you could potentially avoid the injection step and the agent bundle dependency. But Mutagen currently expects an exact version match because the internal protocol isn't stable across versions1, so it will attempt to inject a matching version if none is present. If you're going to vendor/version-lock the Mutagen version that you ship, you might be able to use this strategy to skip the injection step. If you do need to carry around the agent bundle locally, you can slim it down to only include the required agents (probably just linux/amd64 in your case). The archive just contains a bunch of binaries named <GOOS>_<GOARCH>, so a simple script can prune these and save a lot of space. ------------------------------ 1 Actually it is stable across minor release series, but a full version match, including the patch component and any -tag suffix is still required. This may change as the protocol further stabilizes. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#11 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACPQOXR2XN4OPOHJXTNC2GTRP3TCBANCNFSM4HHIT2CQ> .

[Merith-TK]

also in theory, incorperating mutagen should be as easy as replacing rsync function