Schedule plexus and apache.sqoop providers for removal (#36208)

The providers have been agreed to be removed in the LAZY CONSENSUS
threads:

* https://lists.apache.org/thread/6so1qh8kf088spd2y4nl0o5tpbyn0x3b
* https://lists.apache.org/thread/wnov3n2m2v0blnxl3q413mj8vphdrz4b

(cherry picked from commit 97e8f58)

Author: potiuk

.github/ISSUE_TEMPLATE/airflow_providers_bug_report.yml

@@ -40,7 +40,6 @@ body:
         - apache-pig
         - apache-pinot
         - apache-spark
-        - apache-sqoop
         - apprise
         - arangodb
         - asana
@@ -88,7 +87,6 @@ body:
         - papermill
         - pgvector
         - pinecone
-        - plexus
         - postgres
         - presto
         - redis

CONTRIBUTING.rst

@@ -670,18 +670,18 @@ This is the full list of those extras:
   .. START EXTRAS HERE
 aiobotocore, airbyte, alibaba, all, all_dbs, amazon, apache.atlas, apache.beam, apache.cassandra,
 apache.drill, apache.druid, apache.flink, apache.hdfs, apache.hive, apache.impala, apache.kafka,
-apache.kylin, apache.livy, apache.pig, apache.pinot, apache.spark, apache.sqoop, apache.webhdfs,
-apprise, arangodb, asana, async, atlas, atlassian.jira, aws, azure, cassandra, celery, cgroups,
-cloudant, cncf.kubernetes, cohere, common.io, common.sql, crypto, databricks, datadog, dbt.cloud,
+apache.kylin, apache.livy, apache.pig, apache.pinot, apache.spark, apache.webhdfs, apprise,
+arangodb, asana, async, atlas, atlassian.jira, aws, azure, cassandra, celery, cgroups, cloudant,
+cncf.kubernetes, cohere, common.io, common.sql, crypto, databricks, datadog, dbt.cloud,
 deprecated_api, devel, devel_all, devel_ci, devel_hadoop, dingding, discord, doc, doc_gen, docker,
 druid, elasticsearch, exasol, facebook, ftp, gcp, gcp_api, github, github_enterprise, google,
 google_auth, grpc, hashicorp, hdfs, hive, http, imap, influxdb, jdbc, jenkins, kerberos, kubernetes,
 ldap, leveldb, microsoft.azure, microsoft.mssql, microsoft.psrp, microsoft.winrm, mongo, mssql,
 mysql, neo4j, odbc, openai, openfaas, openlineage, opensearch, opsgenie, oracle, otel, pagerduty,
-pandas, papermill, password, pgvector, pinecone, pinot, plexus, postgres, presto, rabbitmq, redis,
-s3, s3fs, salesforce, samba, saml, segment, sendgrid, sentry, sftp, singularity, slack, smtp,
-snowflake, spark, sqlite, ssh, statsd, tableau, tabular, telegram, trino, vertica, virtualenv,
-weaviate, webhdfs, winrm, yandex, zendesk
+pandas, papermill, password, pgvector, pinecone, pinot, postgres, presto, rabbitmq, redis, s3, s3fs,
+salesforce, samba, saml, segment, sendgrid, sentry, sftp, singularity, slack, smtp, snowflake,
+spark, sqlite, ssh, statsd, tableau, tabular, telegram, trino, vertica, virtualenv, weaviate,
+webhdfs, winrm, yandex, zendesk
   .. END EXTRAS HERE
 
 Provider packages

INSTALL

@@ -96,18 +96,18 @@ The list of available extras:
 # START EXTRAS HERE
 aiobotocore, airbyte, alibaba, all, all_dbs, amazon, apache.atlas, apache.beam, apache.cassandra,
 apache.drill, apache.druid, apache.flink, apache.hdfs, apache.hive, apache.impala, apache.kafka,
-apache.kylin, apache.livy, apache.pig, apache.pinot, apache.spark, apache.sqoop, apache.webhdfs,
-apprise, arangodb, asana, async, atlas, atlassian.jira, aws, azure, cassandra, celery, cgroups,
-cloudant, cncf.kubernetes, cohere, common.io, common.sql, crypto, databricks, datadog, dbt.cloud,
+apache.kylin, apache.livy, apache.pig, apache.pinot, apache.spark, apache.webhdfs, apprise,
+arangodb, asana, async, atlas, atlassian.jira, aws, azure, cassandra, celery, cgroups, cloudant,
+cncf.kubernetes, cohere, common.io, common.sql, crypto, databricks, datadog, dbt.cloud,
 deprecated_api, devel, devel_all, devel_ci, devel_hadoop, dingding, discord, doc, doc_gen, docker,
 druid, elasticsearch, exasol, facebook, ftp, gcp, gcp_api, github, github_enterprise, google,
 google_auth, grpc, hashicorp, hdfs, hive, http, imap, influxdb, jdbc, jenkins, kerberos, kubernetes,
 ldap, leveldb, microsoft.azure, microsoft.mssql, microsoft.psrp, microsoft.winrm, mongo, mssql,
 mysql, neo4j, odbc, openai, openfaas, openlineage, opensearch, opsgenie, oracle, otel, pagerduty,
-pandas, papermill, password, pgvector, pinecone, pinot, plexus, postgres, presto, rabbitmq, redis,
-s3, s3fs, salesforce, samba, saml, segment, sendgrid, sentry, sftp, singularity, slack, smtp,
-snowflake, spark, sqlite, ssh, statsd, tableau, tabular, telegram, trino, vertica, virtualenv,
-weaviate, webhdfs, winrm, yandex, zendesk
+pandas, papermill, password, pgvector, pinecone, pinot, postgres, presto, rabbitmq, redis, s3, s3fs,
+salesforce, samba, saml, segment, sendgrid, sentry, sftp, singularity, slack, smtp, snowflake,
+spark, sqlite, ssh, statsd, tableau, tabular, telegram, trino, vertica, virtualenv, weaviate,
+webhdfs, winrm, yandex, zendesk
 # END EXTRAS HERE
 
 # For installing Airflow in development environments - see CONTRIBUTING.rst

airflow/providers/MANAGING_PROVIDERS_LIFECYCLE.rst

@@ -514,14 +514,15 @@ providers (and then remove all the code and documentation related to the provide
 
 The "removed: true" flag will cause the provider to be available for the following commands (note that such
 provider has to be explicitly added as selected to the package - such provider will not be included in
-the available list of providers):
+the available list of providers or when documentation is built unless --include-removed-providers
+flag is used):
 
 * ``breeze build-docs``
 * ``breeze release-management prepare-provider-documentation``
 * ``breeze release-management prepare-provider-packages``
 * ``breeze release-management publish-docs``
 
-For all those commands, release manager needs to specify such to-be-removed provider explicitly as extra
-command during the release process. Except the changelog that needs to be maintained manually, all other
-documentation (main page of the provider documentation, PyPI README), will be automatically updated
-to include removal notice.
+For all those commands, release manager needs to specify ``--include-removed-providers`` when all providers
+are built or must add the provider id explicitly during the release process.
+Except the changelog that needs to be maintained manually, all other documentation (main page of the provider
+documentation, PyPI README), will be automatically updated to include removal notice.

airflow/providers/apache/sqoop/provider.yaml

@@ -21,9 +21,11 @@ name: Apache Sqoop
 description: |
   `Apache Sqoop <https://sqoop.apache.org/>`__
 
-suspended: false
-source-date-epoch: 1700827444
+removed: true
+suspended: true
+source-date-epoch: 1701983361
 versions:
+  - 4.2.0
   - 4.1.0
   - 4.0.0
   - 3.2.1

airflow/providers/plexus/provider.yaml

@@ -21,9 +21,11 @@ name: Plexus
 description: |
     `Plexus <https://plexus.corescientific.com/>`__
 
-suspended: false
-source-date-epoch: 1700827488
+removed: true
+suspended: true
+source-date-epoch: 1701983408
 versions:
+  - 3.4.0
   - 3.3.0
   - 3.2.2
   - 3.2.1

dev/breeze/tests/test_packages.py

@@ -109,17 +109,17 @@ def test_get_provider_requirements():
 
 def test_get_removed_providers():
     # Modify it every time we schedule provider for removal or remove it
-    assert ["daskexecutor"] == get_removed_provider_ids()
+    assert ["apache.sqoop", "daskexecutor", "plexus"] == get_removed_provider_ids()
 
 
 def test_get_suspended_provider_ids():
     # Modify it every time we suspend/resume provider
-    assert ["daskexecutor"] == get_suspended_provider_ids()
+    assert ["apache.sqoop", "daskexecutor", "plexus"] == get_suspended_provider_ids()
 
 
 def test_get_suspended_provider_folders():
     # Modify it every time we suspend/resume provider
-    assert ["daskexecutor"] == get_suspended_provider_folders()
+    assert ["apache/sqoop", "daskexecutor", "plexus"] == get_suspended_provider_folders()
 
 
 @pytest.mark.parametrize(

docs/apache-airflow/extra-packages-ref.rst

@@ -144,8 +144,6 @@ custom bash/python providers).
 +---------------------+-----------------------------------------------------+------------------------------------------------+
 | apache.spark        | ``pip install 'apache-airflow[apache.spark]'``      | All Spark related operators & hooks            |
 +---------------------+-----------------------------------------------------+------------------------------------------------+
-| apache.sqoop        | ``pip install 'apache-airflow[apache.sqoop]'``      | All Sqoop related operators & hooks            |
-+---------------------+-----------------------------------------------------+------------------------------------------------+
 | apache.webhdfs      | ``pip install 'apache-airflow[apache.webhdfs]'``    | HDFS hooks and operators                       |
 +---------------------+-----------------------------------------------------+------------------------------------------------+
 
@@ -204,8 +202,6 @@ These are extras that add dependencies needed for integration with external serv
 +---------------------+-----------------------------------------------------+-----------------------------------------------------+
 | pinecone            | ``pip install 'apache-airflow[pinecone]'``          | Pinecone Operators and Hooks                        |
 +---------------------+-----------------------------------------------------+-----------------------------------------------------+
-| plexus              | ``pip install 'apache-airflow[plexus]'``            | Plexus service of CoreScientific.com AI platform    |
-+---------------------+-----------------------------------------------------+-----------------------------------------------------+
 | salesforce          | ``pip install 'apache-airflow[salesforce]'``        | Salesforce hook                                     |
 +---------------------+-----------------------------------------------------+-----------------------------------------------------+
 | sendgrid            | ``pip install 'apache-airflow[sendgrid]'``          | Send email using sendgrid                           |

generated/provider_dependencies.json

@@ -198,13 +198,6 @@
     ],
     "excluded-python-versions": []
   },
-  "apache.sqoop": {
-    "deps": [
-      "apache-airflow>=2.6.0"
-    ],
-    "cross-providers-deps": [],
-    "excluded-python-versions": []
-  },
   "apprise": {
     "deps": [
       "apache-airflow>=2.6.0",
@@ -751,14 +744,6 @@
     "cross-providers-deps": [],
     "excluded-python-versions": []
   },
-  "plexus": {
-    "deps": [
-      "apache-airflow>=2.6.0",
-      "arrow>=0.16.0"
-    ],
-    "cross-providers-deps": [],
-    "excluded-python-versions": []
-  },
   "postgres": {
     "deps": [
       "apache-airflow-providers-common-sql>=1.3.1",