[GR-58469] Cut ties with Sulong managed.

PullRequest: graalpython/3503

Author: timfel

README.md

@@ -4,8 +4,8 @@
 [![](https://img.shields.io/badge/pyenv-graalpy-blue)](#start-replacing-cpython-with-graalpy) 
 </a> [![Join Slack][badge-slack]][slack] [![GraalVM on Twitter][badge-twitter]][twitter] [![License](https://img.shields.io/badge/license-UPL-green)](#license)
 
-GraalPy is a high-performance implementation of the [Python](https://www.python.org/) language for the JVM built on [GraalVM](https://www.graalvm.org/). 
-GraalPy is a Python 3.11 compliant runtime. 
+GraalPy is a high-performance implementation of the Python language for the JVM built on [GraalVM](https://www.graalvm.org/python). 
+GraalPy is a Python 3.11 compliant runtime.
 It has first-class support for embedding in Java and can turn Python applications into fast, standalone binaries.
 GraalPy is ready for production running pure Python code and has experimental support for many popular native extension modules.
 
@@ -24,7 +24,7 @@ GraalPy is ready for production running pure Python code and has experimental su
 * Support for native extension modules is considered experimental, but you can already install [packages](docs/user/Python-Runtime.md#installing-packages) like *NumPy*, *PyTorch*, or *Tensorflow*; run [Hugging Face](https://huggingface.co/) models like *Stable Diffusion* or *GPT*
 ![](docs/user/assets/mcd.svg#gh-light-mode-only)![](docs/user/assets/mcd-dark.svg#gh-dark-mode-only)<sup>
 We run the tests of the [most depended on PyPI packages](https://libraries.io/pypi) every day.
-For 96% of those packages a recent version can be installed on GraalPy and GraalPy passes about 50% of all tests of all packages combined.
+For 97% of those packages a recent version can be installed on GraalPy and GraalPy passes over 60% of all tests of all packages combined.
 We assume that CPython not passing 100% of all tests is due to problems in our infrastructure that may also affect GraalPy.
 Packages where CPython fails all tests are marked as "not tested" for both CPython and GraalPy.
 </sup>

bisect-benchmark.ini

@@ -3,7 +3,7 @@
 # Usage:
 # - Create a temporary branch based on the main branch (or the bad commit)
 # - Fill in this configuration file, preferably using the automated script
-#   graalpython-apptests/scripts/create-bisect-config
+#   graal-enterprise/graalpython-enterprise/scripts/create-bisect-config
 # - Commit and push the file
 # - The push command output should give you a link to create a PR. Open it, but
 #   don't create a PR. Instead, you should execute the job on your commit using

ci.jsonnet

@@ -1 +1 @@
-{ "overlay": "b8527c2fa08fc88554559a4a54c9294155c84c4f" }
+{ "overlay": "5f2c190a2db5461ac877a32df039c3584ae08148" }

docs/contributor/CONTRIBUTING.md

@@ -273,7 +273,7 @@ There are also different options for how the Python interpreter is run, passed v
  * `default` - run using the standard options
  * `default-multi` - run using a shared engine, which is the mode that is recommended to embedders that want to spawn multiple isolated Python contexts
  * `native` - same as `default`, its name is due to the fact that it runs C extensions using a mixture of LLVM bitcode interpreted and compiled via GraalVM and real native libraries
- * `sandboxed` - this name is historical - this configuration requires Oracle GraalVM and runs all C extensions purely as LLVM bitcode on the GraalVM, without any access to the native OS libraries, i.e., using the `--llvm.managed` option for GraalVM.
+ * `sandboxed` - this name is historical - this configuration runs some modules such as the posix and sha3 modules with Java code instead of with native access.
 
 ### Configuration of the underlying GraalVM runtime
 

docs/user/Embedding-Permissions.md

@@ -35,7 +35,6 @@ This approach is the most compatible with CPython and provides bare access to th
 By default, this implementation bypasses the Truffle abstraction layer, and therefore it is not sandboxed and does not support custom implementations of [Truffle FileSystem service-provider](https://www.graalvm.org/truffle/javadoc/org/graalvm/polyglot/io/FileSystem.html), and other Polyglot API providers related to system interfaces.
 
 The native backend is chosen by default when GraalPy is started via the `graalpy` or any other Python related launcher.
-The exceptions are Python related launchers with `-managed` suffix available only in Oracle GraalVM (for example, `graalpy-managed`), which by default use the `java` POSIX backend.
 
 #### Limitations of the Native Backend
 
@@ -49,7 +48,7 @@ Known limitations are:
 This backend uses the [Truffle abstraction layer](https://github.com/oracle/graal/blob/master/truffle/docs/README.md) and therefore supports custom Polyglot API providers related to system interfaces and sandboxing.
 Because this abstraction is POSIX agnostic, it does not expose all the necessary functionality. Some functionality is emulated, and some functionality is unsupported.
 
-The Java backend is the default when GraalPy is run via the `Context` API, that is, [embedded in Java applications](https://github.com/oracle/graal/blob/master/docs/reference-manual/embedding/embed-languages.md), or when it is launched using Python-related launchers with the `-managed` suffix (available only in Oracle GraalVM).
+The Java backend is the default when GraalPy is run via the `Context` API, that is, [embedded in Java applications](https://github.com/oracle/graal/blob/master/docs/reference-manual/embedding/embed-languages.md).
 
 #### Limitations of the Java Backend
 

docs/user/Python-Standalone-Applications.md

@@ -9,7 +9,8 @@ redirect_from: /reference-manual/python/standalone-binaries/
 # Python Standalone Applications
 
 GraalPy enables you to create a Python application or library as a native application or JAR file with no external dependencies.
-The [Truffle framework](https://github.com/oracle/graal/tree/master/truffle) on which GraalPy is built, combined with the [Sulong LLVM runtime](https://github.com/oracle/graal/tree/master/sulong) that GraalPy leverages for managed execution of Python's native extensions, completely virtualizes all filesystem accesses, including those to the standard library and installed packages.
+The [Truffle framework](https://github.com/oracle/graal/tree/master/truffle) on which GraalPy is built virtualizes all filesystem accesses, including those to the standard library and installed pure Python packages.
+Packages that include native code can still circumvent this, however!
 
 GraalPy includes a module named `standalone` to create a Python binary for Linux, macOS, and Windows. 
 The modules bundles all your application's resources into a single file.

docs/user/assets/mcd-dark.svg

@@ -47,14 +47,13 @@ if(WIN32)
 require_var(CAPI_INC_DIR)
 require_var(TRUFFLE_H_INC)
 require_var(TRUFFLE_NFI_H_INC)
-require_var(LLVM_MODE)
 require_var(GRAALPY_EXT)
 
 if(NOT DEFINED SRC_DIR)
     set(SRC_DIR "${CMAKE_SOURCE_DIR}")
 endif()
 
-set(TARGET_LIBPYTHON "python-${LLVM_MODE}")
+set(TARGET_LIBPYTHON "python-native")
 
 ######################################################################
 # common variables and compile/link options (for all build targets)
@@ -75,11 +74,7 @@ add_compile_definitions(
     GRAALVM_PYTHON_LLVM
 )
 
-if(${LLVM_MODE} STREQUAL "native")
-    add_compile_definitions(GRAALVM_PYTHON_LLVM_NATIVE)
-else()
-    add_compile_definitions(GRAALVM_PYTHON_LLVM_MANAGED)
-endif()
+add_compile_definitions(GRAALVM_PYTHON_LLVM_NATIVE)
 
 
 if(WIN32)

docs/user/assets/mcd.svg

@@ -48,20 +48,8 @@
 #define HANDLE_BASE 0x8000000000000000ULL
 #define IMMORTAL_REFCNT (INT64_MAX >> 1)
 
-#ifdef GRAALVM_PYTHON_LLVM_MANAGED
-/*
- * We can't include sulong headers here, they include stdbool.h and CPython has a test that
- * Python.h won't pull in stdbool.h
- */
-PyAPI_FUNC(int) points_to_py_handle_space(void* ptr);
-#define stub_to_pointer(STUB_PTR) (STUB_PTR)
-#define pointer_to_stub(O) (O)
-#else /* GRAALVM_PYTHON_LLVM_MANAGED */
-
 #define points_to_py_handle_space(PTR) ((((uintptr_t) (PTR)) & HANDLE_BASE) != 0)
 #define stub_to_pointer(STUB_PTR) (((uintptr_t) (STUB_PTR)) | HANDLE_BASE)
 #define pointer_to_stub(PTR) ((PyObject *)(((uintptr_t) (PTR)) & ~HANDLE_BASE))
 
-#endif /* GRAALVM_PYTHON_LLVM_MANAGED */
-
 #endif /* SRC_HANDLES_H_ */

graalpython/com.oracle.graal.python.cext/CMakeLists.txt

@@ -195,10 +195,6 @@ static inline void _PyObject_GC_UNTRACK(
 // Macros to accept any type for the parameter, and to automatically pass
 // the filename and the filename (if NDEBUG is not defined) where the macro
 // is called.
-#ifdef GRAALVM_PYTHON_LLVM_MANAGED
-#  define _PyObject_GC_TRACK(op)
-#  define _PyObject_GC_UNTRACK(op)
-#else
 #ifdef NDEBUG
 #  define _PyObject_GC_TRACK(op) \
         _PyObject_GC_TRACK(_PyObject_CAST(op))
@@ -210,7 +206,6 @@ static inline void _PyObject_GC_UNTRACK(
 #  define _PyObject_GC_UNTRACK(op) \
         _PyObject_GC_UNTRACK(__FILE__, __LINE__, _PyObject_CAST(op))
 #endif
-#endif
 
 #ifdef Py_REF_DEBUG
 extern void _PyDebug_PrintTotalRefs(void);

graalpython/com.oracle.graal.python.cext/include/graalpy/handles.h

@@ -139,7 +139,7 @@ static inline Py_ssize_t Py_REFCNT(PyObject *ob) {
 // bpo-39573: The Py_SET_TYPE() function must be used to set an object type.
 PyAPI_FUNC(PyTypeObject*) PyTruffle_TYPE(PyObject *ob);
 static inline PyTypeObject* Py_TYPE(PyObject *ob) {
-#if defined(GRAALVM_PYTHON) && !defined(GRAALVM_PYTHON_LLVM_MANAGED) && defined(NDEBUG)
+#if defined(GRAALVM_PYTHON) && defined(NDEBUG)
     return (pointer_to_stub(ob)->ob_type);
 #else
     return PyTruffle_TYPE(ob);

graalpython/com.oracle.graal.python.cext/include/internal/pycore_object.h

@@ -45,12 +45,6 @@
 
 #include "pycore_gc.h" // _PyGC_InitState
 
-#ifdef GRAALVM_PYTHON_LLVM_MANAGED
-int points_to_py_handle_space(void* ptr) {
-   return polyglot_is_value(ptr);
-}
-#endif
-
 typedef struct arrayobject {
     PyObject_VAR_HEAD
     char *ob_item;
@@ -282,15 +276,11 @@ PyObject* _Py_EllipsisObjectReference;
 PyObject* _Py_NoneStructReference;
 PyObject* _Py_NotImplementedStructReference;
 
-#ifndef GRAALVM_PYTHON_LLVM_MANAGED
 THREAD_LOCAL PyThreadState *tstate_current = NULL;
-#endif /* GRAALVM_PYTHON_LLVM_MANAGED */
 
 static void initialize_globals() {
-#ifndef GRAALVM_PYTHON_LLVM_MANAGED
     // store the thread state into a thread local variable
     tstate_current = GraalPyTruffleThreadState_Get(&tstate_current);
-#endif /* GRAALVM_PYTHON_LLVM_MANAGED */
     _Py_NoneStructReference = GraalPyTruffle_None();
     _Py_NotImplementedStructReference = GraalPyTruffle_NotImplemented();
     _Py_EllipsisObjectReference = GraalPyTruffle_Ellipsis();
@@ -890,9 +880,7 @@ PyAPI_FUNC(void) initialize_graal_capi(TruffleEnv* env, void **builtin_closures,
         TRUFFLE_CONTEXT = (*env)->getTruffleContext(env);
     }
 
-#ifndef GRAALVM_PYTHON_LLVM_MANAGED
     _PyGC_InitState(gc);
-#endif
 
     initialize_builtins(builtin_closures);
     PyTruffle_Log(PY_TRUFFLE_LOG_FINE, "initialize_builtins: %fs", ((double) (clock() - t)) / CLOCKS_PER_SEC);

graalpython/com.oracle.graal.python.cext/include/object.h

@@ -83,10 +83,6 @@
 #include "pycore_global_objects.h" // _PY_NSMALLPOSINTS
 #include "pycore_gc.h" // PyGC_Head
 
-#ifdef GRAALVM_PYTHON_LLVM_MANAGED
-#include <graalvm/llvm/polyglot.h>
-#endif
-
 #define SRC_CS "utf-8"
 
 /* Flags definitions representing global (debug) options. */
@@ -159,9 +155,7 @@ CAPI_BUILTINS
 
 PyAPI_DATA(uint32_t) Py_Truffle_Options;
 
-#ifndef GRAALVM_PYTHON_LLVM_MANAGED
 extern THREAD_LOCAL Py_LOCAL_SYMBOL PyThreadState *tstate_current;
-#endif /* GRAALVM_PYTHON_LLVM_MANAGED */
 
 extern Py_LOCAL_SYMBOL int graalpy_finalizing;
 

graalpython/com.oracle.graal.python.cext/src/capi.c

@@ -301,7 +301,6 @@ PyFloat_AsDouble(PyObject *op)
 {
     // GraalPy change: read from native object stub or upcall for managed
     if (points_to_py_handle_space(op)) {
-#ifndef GRAALVM_PYTHON_LLVM_MANAGED
         if (PyFloat_Check(op)) {
             double val = ((GraalPyFloatObject*) pointer_to_stub(op))->ob_fval;
 #ifndef NDEBUG
@@ -311,7 +310,6 @@ PyFloat_AsDouble(PyObject *op)
 #endif
             return val;
         }
-#endif /* GRAALVM_PYTHON_LLVM_MANAGED */
         return GraalPyTruffleFloat_AsDouble(op);
     }
 

graalpython/com.oracle.graal.python.cext/src/capi.h

@@ -2426,8 +2426,6 @@ PyInit_gc(void)
 }
 #endif // GraalPy change
 
-#ifndef GRAALVM_PYTHON_LLVM_MANAGED
-
 /* C API for controlling the state of the garbage collector */
 int
 PyGC_Enable(void)
@@ -2503,37 +2501,6 @@ _PyGC_CollectNoFail(PyThreadState *tstate)
     gcstate->collecting = 0;
     return n;
 }
-#else
-int
-PyGC_Enable(void)
-{
-    return 0;
-}
-
-int
-PyGC_Disable(void)
-{
-    return 0;
-}
-
-int
-PyGC_IsEnabled(void)
-{
-    return 0;
-}
-
-Py_ssize_t
-PyGC_Collect(void)
-{
-    return 0;
-}
-
-PyAPI_FUNC(Py_ssize_t)
-_PyGC_CollectNoFail(PyThreadState *tstate)
-{
-    return 0;
-}
-#endif
 
 #if 0 // GraalPy change
 void
@@ -2614,7 +2581,6 @@ _PyGC_Fini(PyInterpreterState *interp)
 }
 #endif // GraalPy change
 
-#ifndef GRAALVM_PYTHON_LLVM_MANAGED
 /* for debugging */
 void
 _PyGC_Dump(PyGC_Head *g)
@@ -2708,28 +2674,6 @@ _PyObject_GC_Link(PyObject *op)
     }
 }
 
-#else
-
-void
-_PyGC_Dump(PyGC_Head *g) {}
-
-void
-PyObject_GC_Track(void *op_raw) {}
-
-void
-PyObject_GC_UnTrack(void *op_raw) {}
-
-int
-PyObject_IS_GC(PyObject *obj)
-{
-    return 0;
-}
-
-void
-_PyObject_GC_Link(PyObject *op) {}
-
-#endif
-
 static PyObject *
 gc_alloc(size_t basicsize, size_t presize)
 {
@@ -2783,8 +2727,6 @@ _PyObject_GC_NewVar(PyTypeObject *tp, Py_ssize_t nitems)
     return op;
 }
 
-#ifndef GRAALVM_PYTHON_LLVM_MANAGED
-
 PyVarObject *
 _PyObject_GC_Resize(PyVarObject *op, Py_ssize_t nitems)
 {
@@ -2869,42 +2811,3 @@ _GraalPyObject_GC_NotifyOwnershipTransfer(PyObject *op)
         CALL_TRAVERSE(traverse, op, visit_strong_reachable, NULL);
     }
 }
-#else
-
-
-PyVarObject *
-_PyObject_GC_Resize(PyVarObject *op, Py_ssize_t nitems)
-{
-    return NULL;
-}
-
-void
-PyObject_GC_Del(void *op) {}
-
-int
-PyObject_GC_IsTracked(PyObject* obj)
-{
-    return 0;
-}
-
-int
-PyObject_GC_IsFinalized(PyObject *obj)
-{
-    return 1;
-}
-
-
-void
-GraalPyObject_GC_Del(void *op) {}
-
-/* Exposes 'gc_collect_impl' such that we can call it from Java. */
-PyAPI_FUNC(Py_ssize_t)
-GraalPyGC_Collect(int generation)
-{
-    return 0;
-}
-
-PyAPI_FUNC(void)
-_GraalPyObject_GC_NotifyOwnershipTransfer(PyObject *op) {}
-
-#endif

graalpython/com.oracle.graal.python.cext/src/floatobject.c

@@ -43,7 +43,6 @@
 static int
 _list_clear(PyListObject *a)
 {
-#ifndef GRAALVM_PYTHON_LLVM_MANAGED
     int64_t i;
     PyObject **item;
 
@@ -62,14 +61,12 @@ _list_clear(PyListObject *a)
     /* Never fails; the return value can be ignored.
        Note that there is no guarantee that the list is actually empty
        at this point, because XDECREF may have populated it again! */
-#endif
     return 0;
 }
 
 static int
 list_traverse(PyListObject *o, visitproc visit, void *arg)
 {
-#ifndef GRAALVM_PYTHON_LLVM_MANAGED
     int64_t size, i;
     PyObject **ob_item;
 
@@ -86,7 +83,6 @@ list_traverse(PyListObject *o, visitproc visit, void *arg)
 
     for (i = size; --i >= 0; )
         Py_VISIT(ob_item[i]);
-#endif
     return 0;
 }