From ddefc203523df11f7abf40a86802a5761a5bc06c Mon Sep 17 00:00:00 2001 From: Julien Richard-Foy Date: Thu, 11 May 2023 16:49:19 +0200 Subject: [PATCH 1/3] Add follow-up blog about the sbt community repository --- ...5-20-sbt-community-repository-follow-up.md | 38 +++++++++++++++++++ 1 file changed, 38 insertions(+) create mode 100644 blog/_posts/2023-05-20-sbt-community-repository-follow-up.md diff --git a/blog/_posts/2023-05-20-sbt-community-repository-follow-up.md b/blog/_posts/2023-05-20-sbt-community-repository-follow-up.md new file mode 100644 index 000000000..029a5066f --- /dev/null +++ b/blog/_posts/2023-05-20-sbt-community-repository-follow-up.md @@ -0,0 +1,38 @@ +--- +layout: blog-detail +post-type: blog +by: Julien Richard-Foy, Scala Center +title: "sbt Community Repository (Follow-Up)" +--- + +[Last month]({% link blog/_posts/2023-04-20-sbt-plugins-community-repository.md %}), I published a blog article where I described the sbt community repository. In this follow-up article, I present our plan to strengthen the infrastructure that supports the sbt ecosystem. + +This blog article is also an initial response to the related Advisory Board proposal [SCP-029 sbt community repository](https://github.com/scalacenter/advisoryboard/blob/main/proposals/029-sbt-community-repository.md), which has been approved by the Advisory Board during the meeting of the [27th of April](https://github.com/scala/scala.epfl.ch/blob/main/minutes/_posts/2023-04-27-april-27-2023.md). + +## Current State + +The sbt community repository was set up a couple of years ago with two primary purposes: +1. host “old” versions of sbt plugins so that “old” Scala projects continue to work today and in the future, +2. give some time to the ecosystem of sbt plugins to migrate to Maven Central. + +On average, the community repository serves 6 TB of data per month and 700 requests per second. These numbers are not huge, but I think they are higher than they should be, especially when we take into account the fact that the content delivered by the community repository is usually downloaded once per client and then cached forever. I would like to thank again JFrog for their continued support of the sbt project and the Scala ecosystem. + +The reason why these numbers are so high is that the migration to Maven Central did not completely happen and many sbt plugins still have their latest stable version on the community repository. +As a consequence, any new developer joining a Scala project or any continuous integration system building a Scala project with a fresh cache will probably download artifacts from the community repository, even if the project has up-to-date dependencies. + +The event of the [7th of April]({% link blog/_posts/2023-04-20-sbt-plugins-community-repository.md %}) has shown that the sbt community repository plays a critical role in the ecosystem. Therefore, we need to increase our safety margins. + +## Action Plan + +Our goal is to ensure all the Scala projects can be built in the future regardless of which versions of sbt plugins they use. + +To achieve this, we will: + +1. Accelerate the migration of the actively used sbt plugins to Maven Central. This will reduce the load on the community repository and make it easier to host mirrors of it. We will need the help of the plugin authors to make progress on this item. +2. Host the Linux packages of sbt releases in a separate place. This will avoid the need to create regular backups of the community repository since all its content will be read-only. + +Concretely, we have already collected a backup of the content of the community repository, and set up a backup server hosting a mirror of the repository. We are currently investigating which plugins are actively used and should be migrated to Maven Central. The next step we will be to explore alternative solutions to host the Linux packages of sbt releases. + +## Conclusion + +The sbt community repository is currently hosted by the company JFrog, which generously provides an Artifactory instance. We are taking steps to reduce the load on their infrastructure in order to both make sure our partnership with JFrog will continue, and to reduce the costs for making the repository redundant. From 21781906f7a0ebee10f6acf4be4b843513872c10 Mon Sep 17 00:00:00 2001 From: Julien Richard-Foy Date: Fri, 12 May 2023 09:58:29 +0200 Subject: [PATCH 2/3] Address feedback from review --- ...023-05-20-sbt-community-repository-follow-up.md | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/blog/_posts/2023-05-20-sbt-community-repository-follow-up.md b/blog/_posts/2023-05-20-sbt-community-repository-follow-up.md index 029a5066f..7a577a2e2 100644 --- a/blog/_posts/2023-05-20-sbt-community-repository-follow-up.md +++ b/blog/_posts/2023-05-20-sbt-community-repository-follow-up.md @@ -5,17 +5,19 @@ by: Julien Richard-Foy, Scala Center title: "sbt Community Repository (Follow-Up)" --- -[Last month]({% link blog/_posts/2023-04-20-sbt-plugins-community-repository.md %}), I published a blog article where I described the sbt community repository. In this follow-up article, I present our plan to strengthen the infrastructure that supports the sbt ecosystem. +[Last month]({% link blog/_posts/2023-04-20-sbt-plugins-community-repository.md %}), we published a blog article where we described the sbt community repository. In this follow-up article, we present our plan to strengthen the infrastructure that supports the sbt ecosystem. This blog article is also an initial response to the related Advisory Board proposal [SCP-029 sbt community repository](https://github.com/scalacenter/advisoryboard/blob/main/proposals/029-sbt-community-repository.md), which has been approved by the Advisory Board during the meeting of the [27th of April](https://github.com/scala/scala.epfl.ch/blob/main/minutes/_posts/2023-04-27-april-27-2023.md). ## Current State The sbt community repository was set up a couple of years ago with two primary purposes: -1. host “old” versions of sbt plugins so that “old” Scala projects continue to work today and in the future, -2. give some time to the ecosystem of sbt plugins to migrate to Maven Central. +1. To host the sbt plugins that were published before 2021 so that the Scala projects that use on them continue to work today and in the future +2. To give some time to the ecosystem of sbt plugins to migrate to Maven Central -On average, the community repository serves 6 TB of data per month and 700 requests per second. These numbers are not huge, but I think they are higher than they should be, especially when we take into account the fact that the content delivered by the community repository is usually downloaded once per client and then cached forever. I would like to thank again JFrog for their continued support of the sbt project and the Scala ecosystem. +Firstly, I would like to again thank JFrog for their continued support of the sbt project and the Scala ecosystem. + +On average, the community repository serves 6 TB of data per month and 700 requests per second. These numbers are not huge, but I think they are higher than they should be, especially when we take into account the fact that the content delivered by the community repository is usually downloaded once per client and then cached indefinitely. The reason why these numbers are so high is that the migration to Maven Central did not completely happen and many sbt plugins still have their latest stable version on the community repository. As a consequence, any new developer joining a Scala project or any continuous integration system building a Scala project with a fresh cache will probably download artifacts from the community repository, even if the project has up-to-date dependencies. @@ -28,10 +30,10 @@ Our goal is to ensure all the Scala projects can be built in the future regardle To achieve this, we will: -1. Accelerate the migration of the actively used sbt plugins to Maven Central. This will reduce the load on the community repository and make it easier to host mirrors of it. We will need the help of the plugin authors to make progress on this item. +1. Accelerate the migration of the actively used sbt plugins to Maven Central. This will reduce the load on the community repository and make it easier to host mirrors of it. We will need the help of the plugin authors to make this happen. 2. Host the Linux packages of sbt releases in a separate place. This will avoid the need to create regular backups of the community repository since all its content will be read-only. -Concretely, we have already collected a backup of the content of the community repository, and set up a backup server hosting a mirror of the repository. We are currently investigating which plugins are actively used and should be migrated to Maven Central. The next step we will be to explore alternative solutions to host the Linux packages of sbt releases. +Concretely, we have already collected a backup of the content of the community repository, and set up a backup server hosting a mirror of the repository. We are currently investigating which plugins are actively used and should be migrated to Maven Central. The next step is to explore alternative solutions to host the Linux packages of sbt releases. ## Conclusion From d75158b36cd3a238ba6991253b3655006b001013 Mon Sep 17 00:00:00 2001 From: Julien Richard-Foy Date: Mon, 15 May 2023 09:06:01 +0200 Subject: [PATCH 3/3] Remove the new blog post and update the old one --- ...-04-20-sbt-plugins-community-repository.md | 2 + ...5-20-sbt-community-repository-follow-up.md | 40 ------------------- 2 files changed, 2 insertions(+), 40 deletions(-) delete mode 100644 blog/_posts/2023-05-20-sbt-community-repository-follow-up.md diff --git a/blog/_posts/2023-04-20-sbt-plugins-community-repository.md b/blog/_posts/2023-04-20-sbt-plugins-community-repository.md index 6004fe198..fc5b3d038 100644 --- a/blog/_posts/2023-04-20-sbt-plugins-community-repository.md +++ b/blog/_posts/2023-04-20-sbt-plugins-community-repository.md @@ -9,6 +9,8 @@ On the 7th of April 2023, many Scala builds stopped working because an outage af In this blog post, we describe the sbt community repository, report on the incident, and outline actions taken so far. There will be more action in the future and a followup blog post. +Update: we have announced our action plan on the [Contributors Forum](https://contributors.scala-lang.org/t/roadmap-for-the-sbt-community-repository/6195). + ## The Incident On the 7th of April 2023, many projects of the Scala ecosystem could not resolve diff --git a/blog/_posts/2023-05-20-sbt-community-repository-follow-up.md b/blog/_posts/2023-05-20-sbt-community-repository-follow-up.md deleted file mode 100644 index 7a577a2e2..000000000 --- a/blog/_posts/2023-05-20-sbt-community-repository-follow-up.md +++ /dev/null @@ -1,40 +0,0 @@ ---- -layout: blog-detail -post-type: blog -by: Julien Richard-Foy, Scala Center -title: "sbt Community Repository (Follow-Up)" ---- - -[Last month]({% link blog/_posts/2023-04-20-sbt-plugins-community-repository.md %}), we published a blog article where we described the sbt community repository. In this follow-up article, we present our plan to strengthen the infrastructure that supports the sbt ecosystem. - -This blog article is also an initial response to the related Advisory Board proposal [SCP-029 sbt community repository](https://github.com/scalacenter/advisoryboard/blob/main/proposals/029-sbt-community-repository.md), which has been approved by the Advisory Board during the meeting of the [27th of April](https://github.com/scala/scala.epfl.ch/blob/main/minutes/_posts/2023-04-27-april-27-2023.md). - -## Current State - -The sbt community repository was set up a couple of years ago with two primary purposes: -1. To host the sbt plugins that were published before 2021 so that the Scala projects that use on them continue to work today and in the future -2. To give some time to the ecosystem of sbt plugins to migrate to Maven Central - -Firstly, I would like to again thank JFrog for their continued support of the sbt project and the Scala ecosystem. - -On average, the community repository serves 6 TB of data per month and 700 requests per second. These numbers are not huge, but I think they are higher than they should be, especially when we take into account the fact that the content delivered by the community repository is usually downloaded once per client and then cached indefinitely. - -The reason why these numbers are so high is that the migration to Maven Central did not completely happen and many sbt plugins still have their latest stable version on the community repository. -As a consequence, any new developer joining a Scala project or any continuous integration system building a Scala project with a fresh cache will probably download artifacts from the community repository, even if the project has up-to-date dependencies. - -The event of the [7th of April]({% link blog/_posts/2023-04-20-sbt-plugins-community-repository.md %}) has shown that the sbt community repository plays a critical role in the ecosystem. Therefore, we need to increase our safety margins. - -## Action Plan - -Our goal is to ensure all the Scala projects can be built in the future regardless of which versions of sbt plugins they use. - -To achieve this, we will: - -1. Accelerate the migration of the actively used sbt plugins to Maven Central. This will reduce the load on the community repository and make it easier to host mirrors of it. We will need the help of the plugin authors to make this happen. -2. Host the Linux packages of sbt releases in a separate place. This will avoid the need to create regular backups of the community repository since all its content will be read-only. - -Concretely, we have already collected a backup of the content of the community repository, and set up a backup server hosting a mirror of the repository. We are currently investigating which plugins are actively used and should be migrated to Maven Central. The next step is to explore alternative solutions to host the Linux packages of sbt releases. - -## Conclusion - -The sbt community repository is currently hosted by the company JFrog, which generously provides an Artifactory instance. We are taking steps to reduce the load on their infrastructure in order to both make sure our partnership with JFrog will continue, and to reduce the costs for making the repository redundant.