Fix #6706 – Update sources for compatibility with NumPy 2 (#6724)

* Explicitly convert NumPy ndarray of np.bool to Python bool

In NumPy 2 (and possibly earlier versions), lines 478-480 produced a
deprecation warning:

```
DeprecationWarning: In future, it will be an error
for 'np.bool' scalars to be interpreted as an index
```

This warning is somewhat misleading: it _is_ the case that Booleans
are involved, but they are not being used as indices.

The fields `rs`, `xs`, and `zs` of CliffordTableau as defined in file
`cirq-core/cirq/qis/clifford_tableau.py` have type
`Optional[np.ndarray]`, and the values in the ndarray have NumPy type
`bool` in practice. The protocol buffer version of CliffordTableau
defined in file `cirq-google/cirq_google/api/v2/program_pb2.pyi`
defines those fields as `collections.abc.Iterable[builtins.bool]`. At
first blush, you might think they're arrays of Booleans in both cases,
but unfortunately, there's a wrinkle: Python defines its built-in
`bool` type as being derived from `int` (see PEP 285), while NumPy
explicitly does _not_ drive its `bool` from its integer class (see
<https://numpy.org/doc/2.0/reference/arrays.scalars.html#numpy.bool>).
The warning about converting `np.bool` to index values (i.e.,
integers) probably arises when the `np.bool` values in the ndarray are
coerced into Python Booleans.

At first, I thought the obvious solution would be to use `np.asarray`
to convert the values to `builtins.bool`, but this did not work:

```
>>> import numpy as np
>>> import builtins
>>> arr = np.array([True, False], dtype=np.bool)
>>> arr
array([ True, False])
>>> type(arr[0])
<class 'numpy.bool'>
>>> newarr = np.asarray(arr, dtype=builtins.bool)
>>> newarr
array([ True, False])
>>> type(newarr[0])
<class 'numpy.bool'>
```

They still end up being NumPy bools. Some other variations on this
approach all failed to produce proper Python Booleans. In the end,
what worked was to use `map()` to apply `builtins.bool` to every value
in the incoming arrays. This may not be as efficient as possible; a
possible optimization for the future is to look for a more efficient
way to cast the types, or avoid having to do it at all.

* Avoid a construct deprecated in NumPy 2

The NumPy 2 Migration Guide [explicitly recommends
changing](https://numpy.org/doc/stable/numpy_2_0_migration_guide.html#adapting-to-changes-in-the-copy-keyword)
constructs of the form

```python
np.array(state, copy=False)
```

to

```python
np.asarray(state)
```

* Avoid implicitly converting 2-D arrays of single value to scalars

NumPy 2 raises deprecation warnings about converting an ndarray with
dimension > 0 of values likle `[[0]]` to a scalar value like `0`. The
solution is to get the value using `.item()`.

* Add pytest option --warn-numpy-data-promotion

This adds a new option to make NumPy warn about data promotion behavior that has changed in NumPy 2. This new promotion can lead to lower precision results when working with floating-point scalars, and errors or overflows when working with integer scalars. Invoking pytest with `--warn-numpy-data-promotion` will cause warnings warnings to be emitted when dissimilar data types are used in an operation in such a way that NumPy ends up changing the data type of the result value.

Although this new option for Cirq's pytest code is most useful during Cirq's migration to NumPy 2, the flag will likely remain for some time afterwards too, because developers will undoubtely need time to adjust to the new NumPy behavior.

For more information about the NumPy warning enabled by this option, see
<https://numpy.org/doc/2.0/numpy_2_0_migration_guide.html#changes-to-numpy-data-type-promotion>.

* Update requirements to use NumPy 2

This updates the minimum NumPy version requirement to 2.0, and updates
a few other packages to versions that are compatible with NumPy 2.0.

Note: NumPy 2.1 was released 3 weeks ago, but at this time, Cirq can
only upgrade to 2.0. This is due to the facts that (a) Google's
internal codebase is moving to NumPy 2.0.2, and not 2.1 yet; and (b)
conflicts arise with some other packages used by Cirq if NumPy 2.1 is
required right now. These considerations will no doubt change in the
near future, at which time we can update Cirq to use NumPy 2.1 or
higher.

* Address NumPy 2 data type promotion warnings

One of the changes in NumPy 2 is to the [behavior of type
promotion](https://numpy.org/devdocs/numpy_2_0_migration_guide.html#changes-to-numpy-data-type-promotion).
A possible negative impact of the changes is that some operations
involving scalar types can lead to lower precision, or even overflow.
For example, `uint8(100) + 200` previously (in Numpy < 2.0) produced a
`unit16` value, but now results in a `unit8` value and an overflow
_warning_ (not error). This can have an impact on Cirq. For example,
in Cirq, simulator measurement result values are `uint8`'s, and in
some places, arrays of values are summed; this leads to overflows if
the sum > 128. It would not be appropriate to change measurement
values to be larger than `uint8`, so in cases like this, the proper
solution is probably to make sure that where values are summed or
otherwise numerically manipulated, `uint16` or larger values are
ensured.

NumPy 2 offers a new option
(`np._set_promotion_state("weak_and_warn")`) to produce warnings where
data types are changed. Commit 6cf50eb adds a new command-line to our
pytest framework, such that running

```bash
check/pytest --warn-numpy-data-promotion
```

will turn on this NumPy setting. Running `check/pytest` with this
option enabled revealed quite a lot of warnings. The present commit
changes code in places where those warnings were raised, in an effort
to eliminate as many of them as possible.

It is certainly the case that not all of the type promotion warnings
are meaningful. Unfortunately, I found it sometimes difficult to be
sure of which ones _are_ meaningful, in part because Cirq's code has
many layers and uses ndarrays a lot, and understanding the impact of a
type demotion (say, from `float64` to `float32`) was difficult for me
to do. In view of this, I wanted to err on the side of caution and try
to avoid losses of precision. The principles followed in the changes
are roughly the following:

* Don't worry about warnings about changes from `complex64` to
  `complex128`, as this obviously does not reduce precision.

* If a warning involves an operation using an ndarray, change the code
  to try to get the actual data type of the data elements in the array
  rather than use a specific data type. This is the reason some of the
  changes look like the following, where it reaches into an ndarray to
  get the dtype of an element and then later uses the `.type()` method
  of that dtype to cast the value of something else:

    ```python
    dtype = args.target_tensor.flat[0].dtype
    .....
    args.target_tensor[subspace] *= dtype.type(x)
    ```

* In cases where the above was not possible, or where it was obvious
  what the type must always be, the changes add type casts with
  explicit types like `complex(x)` or `np.float64(x)`.

It is likely that this approach resulted in some unnecessary
up-promotion of values and may have impacted run-time performance.
Some simple overall timing of `check/pytest` did not reveal a glaring
negative impact of the changes, but that doesn't mean real
applications won't be impacted. Perhaps a future review can evaluate
whether speedups are possible.

* NumPy 2 data promotion + minor refactoring

This commit for one file implements a minor refactoring of 3 test
functions to make them all use similar idioms (for greater ease of
reading) and to address the same NumPy 2 data promotion warnings for
the remaining files in commit eeeabef.

* Adjust dtypes per mypy warnings

Mypy flagged a couple of the previous data type declaration changes as
being incompatible with expected types. Changing them to satisfy mypy
did not affect Numpy data type promotion warnings.

* Fix Rigetti check for Aspen family device kind (#6734)

* Sync with new API for checking device family in qcs-sdk-python,
  Ref: rigetti/qcs-sdk-rust#463 in isa.pyi

* Require qcs-sdk-python-0.20.1 which introduced the new family API

Fixes #6732

* Adjustment for mypy: change 2 places where types are declared

Pytest was happy with the previous approach to declaring the value
types in a couple of expressions, but mypy was not. This new version
satisfies both.

* Avoid getting NumPy dtypes in printed (string) scalar values

As a consequence of [NEP
51](https://numpy.org/neps/nep-0051-scalar-representation.html#nep51),
the string representation of scalar numbers changed in NumPy 2 to
include type information. This affected printing Cirq circuit
diagrams: instead seeing numbers like 1.5, you would see
`np.float64(1.5)` and similar.

The solution is to avoid getting the repr output of NumPy scalars
directly, and instead doing `.item()` on them before passing them
to `format()` or other string-producing functions.

* Don't force Numpy 2; maintain compatibility with 1

The recent changes support NumPy 2 (as long as cirq-rigetti is removed
manually), but they don't require NumPy 2. We can maintain
compatibility with Numpy 1.x.

* Bump serve-static and express in /cirq-web/cirq_ts (#6731)

Bumps [serve-static](https://github.com/expressjs/serve-static) and [express](https://github.com/expressjs/express). These dependencies needed to be updated together.

Updates `serve-static` from 1.15.0 to 1.16.2
- [Release notes](https://github.com/expressjs/serve-static/releases)
- [Changelog](https://github.com/expressjs/serve-static/blob/v1.16.2/HISTORY.md)
- [Commits](expressjs/serve-static@v1.15.0...v1.16.2)

Updates `express` from 4.19.2 to 4.21.0
- [Release notes](https://github.com/expressjs/express/releases)
- [Changelog](https://github.com/expressjs/express/blob/4.21.0/History.md)
- [Commits](expressjs/express@4.19.2...4.21.0)

---
updated-dependencies:
- dependency-name: serve-static
  dependency-type: indirect
- dependency-name: express
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Michael Hucka <[email protected]>

* Silence pytest warnings about asyncio fixture scope

In the current version of pytest (8.3.3) with the pytest-asyncio
module version 0.24.0, we see the following warnings at the beginning
of a pytest run:

```
warnings.warn(PytestDeprecationWarning(_DEFAULT_FIXTURE_LOOP_SCOPE_UNSET))

..../lib/python3.10/site-packages/pytest_asyncio/plugin.py:208:
PytestDeprecationWarning: The configuration option
"asyncio_default_fixture_loop_scope" is unset. The event loop scope for
asynchronous fixtures will default to the fixture caching scope. Future
versions of pytest-asyncio will default the loop scope for asynchronous
fixtures to function scope. Set the default fixture loop scope explicitly in
order to avoid unexpected behavior in the future. Valid fixture loop scopes
are: "function", "class", "module", "package", "session"
```

A [currently-open issue and discussion over in the pytest-asyncio
repo](pytest-dev/pytest-asyncio#924) suggests that
this is an undesired side-effect of a recent change in pytest-asyncio and is
not actually a significant warning. Moreover, the discussion suggests the
warning will be removed or changed in the future.

In the meantime, the warning is confusing because it makes it sound like
something is wrong. This simple PR silences the warning by adding a suitable
pytest init flag to `pyproject.toml'.

* Fix wrong number of arguments to reshape()

Flagged by pylint.

* Fix formatting issues flagged by check/format-incremental

* Add coverage tests for changes in format_real()

* Remove import of kahypar after all

In commit eb98361 I added the import of kahypar, which (at least at the time) appeared to have been imported by Quimb. Double-checking this import in clean environments reveals that in fact, nothing depends on kahypar.

Taking it out via a separate commit because right now this package is causing our GitHub actions for commit checks to fail, and I want to leave a record of what caused the failures and how they were resolved.

* Simplify proper_repr

* No need to use bool from builtins

* Restore numpy casting to the state as in main

* Fix failing test_run_repetitions_terminal_measurement_stochastic

Instead of summing int8 ones count them.

* Simplify CircuitDiagramInfoArgs.format_radians

Handle np2 numeric types without outputting their dtype.

* `.item()` already collapses dimensions and converts to int

* Exclude cirq_rigetti from json_serialization_test when using numpy-2

This also enables the hash_from_pickle_test.py with numpy-2.

* pytest - apply warn_numpy_data_promotion option before test collection

* Add temporary requirements file for NumPy-2.0

* Adjust requirements for cirq-core

* allow numpy-1.24 which is still in the NEP-29 support window per
  https://numpy.org/neps/nep-0029-deprecation_policy.html

* require `scipy~=1.8` as scipy-1.8 is the first version that has
  wheels for Python 3.10

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: Pavol Juhas <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Author: 3 people

cirq-core/cirq/_compat.py

@@ -192,6 +192,9 @@ def _print(self, expr, **kwargs):
     if hasattr(value, "__qualname__"):
         return f"{value.__module__}.{value.__qualname__}"
 
+    if isinstance(value, np.number):
+        return str(value)
+
     return repr(value)
 
 

cirq-core/cirq/contrib/requirements.txt

@@ -4,5 +4,5 @@ ply>=3.6
 pylatex~=1.4
 
 # quimb
-quimb~=1.7
+quimb>=1.8
 opt_einsum

cirq-core/cirq/protocols/circuit_diagram_info_protocol.py

@@ -277,6 +277,8 @@ def format_radians(self, radians: Union[sympy.Basic, int, float]) -> str:
         if self.precision is not None and not isinstance(radians, sympy.Basic):
             quantity = self.format_real(radians / np.pi)
             return quantity + unit
+        if isinstance(radians, np.number):
+            return str(radians)
         return repr(radians)
 
     def copy(self):

cirq-core/cirq/protocols/circuit_diagram_info_protocol_test.py

@@ -208,6 +208,9 @@ def test_format_real():
     assert args.format_real(sympy.Symbol('t')) == 't'
     assert args.format_real(sympy.Symbol('t') * 2 + 1) == '2*t + 1'
 
+    assert args.format_real(np.float64(1.1)) == '1.1'
+    assert args.format_real(np.int32(1)) == '1'
+
     args.precision = None
     assert args.format_real(1) == '1'
     assert args.format_real(1.1) == '1.1'
@@ -252,6 +255,7 @@ def test_format_radians_without_precision():
     assert args.format_radians(-np.pi) == '-pi'
     assert args.format_radians(1.1) == '1.1'
     assert args.format_radians(1.234567) == '1.234567'
+    assert args.format_radians(np.float32(1.234567)) == '1.234567'
     assert args.format_radians(1 / 7) == repr(1 / 7)
     assert args.format_radians(sympy.Symbol('t')) == 't'
     assert args.format_radians(sympy.Symbol('t') * 2 + 1) == '2*t + 1'
@@ -261,6 +265,7 @@ def test_format_radians_without_precision():
     assert args.format_radians(-np.pi) == '-π'
     assert args.format_radians(1.1) == '1.1'
     assert args.format_radians(1.234567) == '1.234567'
+    assert args.format_radians(np.float32(1.234567)) == '1.234567'
     assert args.format_radians(1 / 7) == repr(1 / 7)
     assert args.format_radians(sympy.Symbol('t')) == 't'
     assert args.format_radians(sympy.Symbol('t') * 2 + 1) == '2*t + 1'

cirq-core/cirq/protocols/json_serialization_test.py

@@ -55,6 +55,13 @@ class _ModuleDeprecation:
     'non_existent_should_be_fine': None,
 }
 
+# TODO(#6706) remove after cirq_rigetti supports NumPy 2.0
+if np.__version__.startswith("2."):  # pragma: no cover
+    warnings.warn(
+        "json_serialization_test - ignoring cirq_rigetti due to incompatibility with NumPy 2.0"
+    )
+    del TESTED_MODULES["cirq_rigetti"]
+
 
 def _get_testspecs_for_modules() -> List[ModuleJsonTestSpec]:
     modules = []

cirq-core/cirq/qis/states.py

@@ -264,7 +264,7 @@ def quantum_state(
             dtype = DEFAULT_COMPLEX_DTYPE
         data = one_hot(index=state, shape=(dim,), dtype=dtype)
     else:
-        data = np.array(state, copy=False)
+        data = np.asarray(state)
         if qid_shape is None:
             qid_shape = infer_qid_shape(state)
         if data.ndim == 1 and data.dtype.kind != 'c':

cirq-core/cirq/sim/simulator.py

@@ -912,7 +912,7 @@ def __repr__(self) -> str:
     def __str__(self) -> str:
         def bitstring(vals):
             separator = ' ' if np.max(vals) >= 10 else ''
-            return separator.join(str(int(v)) for v in vals)
+            return separator.join(str(v.item()) for v in vals)
 
         results = sorted([(key, bitstring(val)) for key, val in self.measurements.items()])
         if not results:

cirq-core/cirq/sim/sparse_simulator_test.py

@@ -115,7 +115,7 @@ def test_run_repetitions_terminal_measurement_stochastic():
     q = cirq.LineQubit(0)
     c = cirq.Circuit(cirq.H(q), cirq.measure(q, key='q'))
     results = cirq.Simulator().run(c, repetitions=10000)
-    assert 1000 <= sum(v[0] for v in results.measurements['q']) < 9000
+    assert 1000 <= np.count_nonzero(results.measurements['q']) < 9000
 
 
 @pytest.mark.parametrize('dtype', [np.complex64, np.complex128])
@@ -255,7 +255,7 @@ def test_run_mixture(dtype: Type[np.complexfloating], split: bool):
     simulator = cirq.Simulator(dtype=dtype, split_untangled_states=split)
     circuit = cirq.Circuit(cirq.bit_flip(0.5)(q0), cirq.measure(q0))
     result = simulator.run(circuit, repetitions=100)
-    assert 20 < sum(result.measurements['q(0)'])[0] < 80
+    assert 20 < np.count_nonzero(result.measurements['q(0)']) < 80
 
 
 @pytest.mark.parametrize('dtype', [np.complex64, np.complex128])
@@ -265,8 +265,8 @@ def test_run_mixture_with_gates(dtype: Type[np.complexfloating], split: bool):
     simulator = cirq.Simulator(dtype=dtype, split_untangled_states=split, seed=23)
     circuit = cirq.Circuit(cirq.H(q0), cirq.phase_flip(0.5)(q0), cirq.H(q0), cirq.measure(q0))
     result = simulator.run(circuit, repetitions=100)
-    assert sum(result.measurements['q(0)'])[0] < 80
-    assert sum(result.measurements['q(0)'])[0] > 20
+    assert np.count_nonzero(result.measurements['q(0)']) < 80
+    assert np.count_nonzero(result.measurements['q(0)']) > 20
 
 
 @pytest.mark.parametrize('dtype', [np.complex64, np.complex128])

cirq-core/requirements.txt

@@ -4,10 +4,10 @@ attrs>=21.3.0
 duet>=0.2.8
 matplotlib~=3.0
 networkx>=2.4
-numpy~=1.22
+numpy>=1.24
 pandas
 sortedcontainers~=2.0
-scipy~=1.0
+scipy~=1.8
 sympy
 typing_extensions>=4.2
 tqdm

cirq-google/cirq_google/serialization/arg_func_langs.py

@@ -474,9 +474,9 @@ def clifford_tableau_arg_to_proto(
     msg = v2.program_pb2.CliffordTableau() if out is None else out
     msg.num_qubits = value.n
     msg.initial_state = value.initial_state
-    msg.xs.extend(value.xs.flatten())
-    msg.rs.extend(value.rs.flatten())
-    msg.zs.extend(value.zs.flatten())
+    msg.xs.extend(map(bool, value.xs.flatten()))
+    msg.rs.extend(map(bool, value.rs.flatten()))
+    msg.zs.extend(map(bool, value.zs.flatten()))
     return msg
 
 

conftest.py

@@ -13,6 +13,7 @@
 # limitations under the License.
 
 import pytest
+import numpy as np
 
 
 def pytest_addoption(parser):
@@ -25,6 +26,19 @@ def pytest_addoption(parser):
     parser.addoption(
         "--enable-slow-tests", action="store_true", default=False, help="run slow tests"
     )
+    parser.addoption(
+        "--warn-numpy-data-promotion",
+        action="store_true",
+        default=False,
+        help="enable NumPy 2 data type promotion warnings",
+    )
+
+
+def pytest_configure(config):
+    # If requested, globally enable verbose NumPy 2 warnings about data type
+    # promotion. See https://numpy.org/doc/2.0/numpy_2_0_migration_guide.html.
+    if config.option.warn_numpy_data_promotion:
+        np._set_promotion_state("weak_and_warn")
 
 
 def pytest_collection_modifyitems(config, items):

dev_tools/requirements/dev-np2.env.txt

@@ -0,0 +1,17 @@
+# This file describes a development environment for cirq with NumPy-2
+# It skips cirq_rigetti which has dependencies incompatible with NumPy-2.
+#
+# TODO(#6706) remove this file after cirq_rigetti supports NumPy 2.0
+
+# dev.env.txt expanded without cirq-rigetti ----------------------------------
+
+-r ../../cirq-aqt/requirements.txt
+-r ../../cirq-core/requirements.txt
+-r ../../cirq-google/requirements.txt
+-r ../../cirq-core/cirq/contrib/requirements.txt
+
+-r deps/dev-tools.txt
+
+# own rules ------------------------------------------------------------------
+
+numpy>=2

examples/bb84.py

@@ -128,7 +128,7 @@ def main(num_qubits=8):
     repetitions = 1
 
     result = cirq.Simulator().run(program=circuit, repetitions=repetitions)
-    result_bitstring = bitstring([int(result.measurements[str(q)]) for q in qubits])
+    result_bitstring = bitstring([result.measurements[str(q)].item() for q in qubits])
 
     # Take only qubits where bases match
     obtained_key = ''.join(
@@ -158,14 +158,14 @@ def main(num_qubits=8):
     # Run simulations.
     repetitions = 1
     result = cirq.Simulator().run(program=alice_eve_circuit, repetitions=repetitions)
-    eve_state = [int(result.measurements[str(q)]) for q in qubits]
+    eve_state = [result.measurements[str(q)].item() for q in qubits]
 
     eve_bob_circuit = make_bb84_circ(num_qubits, eve_basis, bob_basis, eve_state)
 
     # Run simulations.
     repetitions = 1
     result = cirq.Simulator().run(program=eve_bob_circuit, repetitions=repetitions)
-    result_bitstring = bitstring([int(result.measurements[str(q)]) for q in qubits])
+    result_bitstring = bitstring([result.measurements[str(q)].item() for q in qubits])
 
     # Take only qubits where bases match
     obtained_key = ''.join(