Merge pull request #83 from casework/release-0.8.0

Release 0.8.0

Author: kchason

.github/workflows/cicd.yml

@@ -32,7 +32,7 @@ jobs:
       matrix:
         python-version: 
           - '3.7'
-          - '3.10'
+          - '3.11'
 
     steps:
     - uses: actions/checkout@v2

CONTRIBUTE.md

@@ -27,10 +27,12 @@ pushd case_utils/ontology
   git add case-0.6.0.ttl  # Assuming CASE 0.6.0 was just released.
   # and/or
   git add uco-0.8.0.ttl   # Assuming UCO 0.8.0 was adopted in CASE 0.6.0.
+
+  git add ontology_and_version_iris.txt
 popd
 make check
 # Assuming `make check` passes:
-git commit -m "Build CASE 0.6.0 monolithic .ttl files" case_utils/ontology/case-0.6.0-subclasses.ttl case_utils/ontology/case-0.6.0.ttl
+git commit -m "Build CASE 0.6.0 monolithic .ttl files" case_utils/ontology/case-0.6.0-subclasses.ttl case_utils/ontology/case-0.6.0.ttl case_utils/ontology/ontology_and_version_iris.txt
 git commit -m "Update CASE ontology pointer to version 0.6.0" dependencies/CASE case_utils/ontology/version_info.py
 ```
 
@@ -43,4 +45,4 @@ pre-commit --version
 The `pre-commit` tool hooks into Git's commit machinery to run a set of linters and static analyzers over each change. To install `pre-commit` into Git's hooks, run:
 ```bash
 pre-commit install
-```
+```

README.md

@@ -55,12 +55,17 @@ To produce the validation report as a machine-readable graph output, the `--form
 case_validate --format turtle input.json > result.ttl
 ```
 
-To use one or more supplementary ontology files, the `--ontology-graph` flag can be used, more than once if desired, to supplement the selected CASE version:
+To use one or more supplementary ontology or shape files, the `--ontology-graph` flag can be used, more than once if desired, to supplement the selected CASE version:
 
 ```bash
-case_validate --ontology-graph internal_ontology.ttl --ontology-graph experimental_shapes.ttl input.json
+case_validate \
+  --ontology-graph internal_ontology.ttl \
+  --ontology-graph experimental_shapes.ttl \
+  input.json
 ```
 
+This tool uses the `--built-version` flag, described [below](#built-versions).
+
 Other flags are reviewable with `case_validate --help`.
 
 
@@ -87,6 +92,8 @@ These commands can be used with any RDF files to run arbitrary SPARQL queries.
 
 Note that prefixes used in the SPARQL queries do not need to be defined in the SPARQL query.  Their mapping will be inherited from their first definition in the input graph files.  However, input graphs are not required to agree on prefix mappings, so there is potential for confusion from input argument order mattering if two input graph files disagree on what a prefix maps to.  If there is concern of ambiguity from inputs, a `PREFIX` statement should be included in the query, such as is shown in [this test query](tests/case_utils/case_sparql_select/subclass.sparql).
 
+These tools use the `--built-version` flag, described [below](#built-versions).
+
 
 #### `case_sparql_construct`
 
@@ -116,6 +123,15 @@ case_sparql_select output.md input.sparql input.json [input-2.json ...]
 This [module](case_utils/local_uuid.py) provides a wrapper UUID generator, `local_uuid()`.  Its main purpose is making example data generate consistent identifiers, and intentionally includes mechanisms to make it difficult to activate this mode without awareness of the caller.
 
 
+### Built versions
+
+Several tools in this package include a flag `--built-version`.  This flag tailors the tool's behavior to a certain CASE ontology version; typically, this involves mixing the ontology graph into the data graph for certain necessary knowledge expansion for pattern matching (such as making queries aware of the OWL subclass hierarchy).
+
+If not provided, the tool will assume a default value of the latest ontology version.
+
+If the special value `none` is provided, none of the ontology builds this package ships will be included in the data graph.  The `none` value supports use cases that are wholly independent of CASE, such as running a test in a specialized vocabulary; and also suports use cases where a non-released CASE version is meant to be used, such as a locally revised version of CASE where some concept revisions are being reviewed.
+
+
 ## Development status
 
 This repository follows [CASE community guidance on describing development status](https://caseontology.org/resources/software.html#development_status), by adherence to noted support requirements.

case_utils/__init__.py

@@ -11,6 +11,6 @@
 #
 # We would appreciate acknowledgement if the software is used.
 
-__version__ = "0.7.0"
+__version__ = "0.8.0"
 
 from . import local_uuid  # noqa: F401

case_utils/case_sparql_construct/__init__.py

@@ -15,7 +15,7 @@
 This script executes a SPARQL CONSTRUCT query, returning a graph of the generated triples.
 """
 
-__version__ = "0.2.3"
+__version__ = "0.2.4"
 
 import argparse
 import logging
@@ -49,7 +49,7 @@ def main() -> None:
         "--built-version",
         choices=tuple(built_version_choices_list),
         default="case-" + CURRENT_CASE_VERSION,
-        help="Ontology version to use to supplement query, such as for subclass querying.  Does not require networking to use.  Default is most recent CASE release.",
+        help="Ontology version to use to supplement query, such as for subclass querying.  Does not require networking to use.  Default is most recent CASE release.  Passing 'none' will mean no pre-built CASE ontology versions accompanying this tool will be included in the analysis.",
     )
     parser.add_argument(
         "--disallow-empty-results",

case_utils/case_sparql_select/__init__.py

@@ -26,7 +26,7 @@
 Should a more complex query be necessary, an outer, wrapping SELECT query would let this script continue to function.
 """
 
-__version__ = "0.4.3"
+__version__ = "0.4.4"
 
 import argparse
 import binascii
@@ -63,7 +63,7 @@ def main() -> None:
         "--built-version",
         choices=tuple(built_version_choices_list),
         default="case-" + CURRENT_CASE_VERSION,
-        help="Ontology version to use to supplement query, such as for subclass querying.  Does not require networking to use.  Default is most recent CASE release.",
+        help="Ontology version to use to supplement query, such as for subclass querying.  Does not require networking to use.  Default is most recent CASE release.  Passing 'none' will mean no pre-built CASE ontology versions accompanying this tool will be included in the analysis.",
     )
     parser.add_argument(
         "--disallow-empty-results",

case_utils/case_validate/__init__.py

@@ -29,27 +29,47 @@
 details.)
 """
 
-__version__ = "0.1.3"
+__version__ = "0.2.0"
 
 import argparse
 import importlib.resources
 import logging
 import os
 import sys
 import typing
+import warnings
 
 import pyshacl  # type: ignore
-import rdflib.util
+import rdflib
 
 import case_utils.ontology
 from case_utils.ontology.version_info import (
     CURRENT_CASE_VERSION,
     built_version_choices_list,
 )
 
+NS_OWL = rdflib.OWL
+NS_RDF = rdflib.RDF
+NS_RDFS = rdflib.RDFS
+
 _logger = logging.getLogger(os.path.basename(__file__))
 
 
+class NonExistentCDOConceptWarning(UserWarning):
+    """
+    This class is used when a concept is encountered in the data graph that is not part of CDO ontologies, according to the --built-version flags and --ontology-graph flags.
+    """
+
+    pass
+
+
+def concept_is_cdo_concept(n_concept: rdflib.URIRef) -> bool:
+    concept_iri = str(n_concept)
+    return concept_iri.startswith(
+        "https://ontology.unifiedcyberontology.org/"
+    ) or concept_iri.startswith("https://ontology.caseontology.org/")
+
+
 def main() -> None:
     parser = argparse.ArgumentParser(
         description="CASE wrapper to pySHACL command line tool."
@@ -72,7 +92,7 @@ def main() -> None:
         "--built-version",
         choices=tuple(built_version_choices_list),
         default="case-" + CURRENT_CASE_VERSION,
-        help="Monolithic aggregation of CASE ontology files at certain versions.  Does not require networking to use.  Default is most recent CASE release.",
+        help="Monolithic aggregation of CASE ontology files at certain versions.  Does not require networking to use.  Default is most recent CASE release.  Passing 'none' will mean no pre-built CASE ontology versions accompanying this tool will be included in the analysis.",
     )
     parser.add_argument(
         "--ontology-graph",
@@ -86,10 +106,20 @@ def main() -> None:
         action="store_true",
         help="(As with pyshacl CLI) Abort on first invalid data.",
     )
+    parser.add_argument(
+        "--allow-info",
+        "--allow-infos",
+        dest="allow_infos",
+        action="store_true",
+        default=False,
+        help="(As with pyshacl CLI) Shapes marked with severity of Info will not cause result to be invalid.",
+    )
     parser.add_argument(
         "-w",
+        "--allow-warning",
         "--allow-warnings",
         action="store_true",
+        dest="allow_warnings",
         help="(As with pyshacl CLI) Shapes marked with severity of Warning or Info will not cause result to be invalid.",
     )
     parser.add_argument(
@@ -112,6 +142,14 @@ def main() -> None:
         default="none",
         help='(As with pyshacl CLI) Choose a type of inferencing to run against the Data Graph before validating. Default is "none".',
     )
+    parser.add_argument(
+        "-m",
+        "--metashacl",
+        dest="metashacl",
+        action="store_true",
+        default=False,
+        help="(As with pyshacl CLI) Validate the SHACL Shapes graph against the shacl-shacl Shapes Graph before validating the Data Graph.",
+    )
     parser.add_argument(
         "-o",
         "--output",
@@ -142,6 +180,71 @@ def main() -> None:
             _logger.debug("arg_ontology_graph = %r.", arg_ontology_graph)
             ontology_graph.parse(arg_ontology_graph)
 
+    # Construct set of CDO concepts for data graph concept-existence review.
+    cdo_concepts: typing.Set[rdflib.URIRef] = set()
+
+    for n_structural_class in [
+        NS_OWL.Class,
+        NS_OWL.AnnotationProperty,
+        NS_OWL.DatatypeProperty,
+        NS_OWL.ObjectProperty,
+        NS_RDFS.Datatype,
+    ]:
+        for ontology_triple in ontology_graph.triples(
+            (None, NS_RDF.type, n_structural_class)
+        ):
+            if not isinstance(ontology_triple[0], rdflib.URIRef):
+                continue
+            if concept_is_cdo_concept(ontology_triple[0]):
+                cdo_concepts.add(ontology_triple[0])
+    for n_ontology_predicate in [
+        NS_OWL.backwardCompatibleWith,
+        NS_OWL.imports,
+        NS_OWL.incompatibleWith,
+        NS_OWL.priorVersion,
+        NS_OWL.versionIRI,
+    ]:
+        for ontology_triple in ontology_graph.triples(
+            (None, n_ontology_predicate, None)
+        ):
+            assert isinstance(ontology_triple[0], rdflib.URIRef)
+            assert isinstance(ontology_triple[2], rdflib.URIRef)
+            cdo_concepts.add(ontology_triple[0])
+            cdo_concepts.add(ontology_triple[2])
+    for ontology_triple in ontology_graph.triples((None, NS_RDF.type, NS_OWL.Ontology)):
+        if not isinstance(ontology_triple[0], rdflib.URIRef):
+            continue
+        cdo_concepts.add(ontology_triple[0])
+
+    # Also load historical ontology and version IRIs.
+    ontology_and_version_iris_data = importlib.resources.read_text(
+        case_utils.ontology, "ontology_and_version_iris.txt"
+    )
+    for line in ontology_and_version_iris_data.split("\n"):
+        cleaned_line = line.strip()
+        if cleaned_line == "":
+            continue
+        cdo_concepts.add(rdflib.URIRef(cleaned_line))
+
+    data_cdo_concepts: typing.Set[rdflib.URIRef] = set()
+    for data_triple in data_graph.triples((None, None, None)):
+        for data_triple_member in data_triple:
+            if isinstance(data_triple_member, rdflib.URIRef):
+                if concept_is_cdo_concept(data_triple_member):
+                    data_cdo_concepts.add(data_triple_member)
+            elif isinstance(data_triple_member, rdflib.Literal):
+                if isinstance(data_triple_member.datatype, rdflib.URIRef):
+                    if concept_is_cdo_concept(data_triple_member.datatype):
+                        data_cdo_concepts.add(data_triple_member.datatype)
+
+    undefined_cdo_concepts = data_cdo_concepts - cdo_concepts
+    for undefined_cdo_concept in sorted(undefined_cdo_concepts):
+        warnings.warn(undefined_cdo_concept, NonExistentCDOConceptWarning)
+    undefined_cdo_concepts_message = (
+        "There were %d concepts with CDO IRIs in the data graph that are not in the ontology graph."
+        % len(undefined_cdo_concepts)
+    )
+
     # Determine output format.
     # pySHACL's determination of output formatting is handled solely
     # through the -f flag.  Other CASE CLI tools handle format
@@ -160,7 +263,9 @@ def main() -> None:
         shacl_graph=ontology_graph,
         ont_graph=ontology_graph,
         inference=args.inference,
+        meta_shacl=args.metashacl,
         abort_on_first=args.abort,
+        allow_infos=True if args.allow_infos else False,
         allow_warnings=True if args.allow_warnings else False,
         debug=True if args.debug else False,
         do_owl_imports=True if args.imports else False,
@@ -194,6 +299,13 @@ def main() -> None:
                 % type(validation_graph)
             )
 
+    if len(undefined_cdo_concepts) > 0:
+        warnings.warn(undefined_cdo_concepts_message)
+        if not args.allow_warnings:
+            undefined_cdo_concepts_alleviation_message = "The data graph is SHACL-conformant with the CDO ontologies, but nonexistent-concept references raise Warnings with this tool.  Please either correct the concept names in the data graph; use the --ontology-graph flag to pass a corrected CDO ontology file, also using --built-version none; or, use the --allow-warnings flag."
+            warnings.warn(undefined_cdo_concepts_alleviation_message)
+            conforms = False
+
     sys.exit(0 if conforms else 1)
 
 

case_utils/local_uuid.py

@@ -15,7 +15,7 @@
 This library is a wrapper for uuid, provided to generate repeatable UUIDs if requested.
 """
 
-__version__ = "0.3.0"
+__version__ = "0.3.1"
 
 import logging
 import os
@@ -38,7 +38,7 @@ def configure() -> None:
     if os.getenv("DEMO_UUID_REQUESTING_NONRANDOM") == "NONRANDOM_REQUESTED":
         warnings.warn(
             "Environment variable DEMO_UUID_REQUESTING_NONRANDOM is deprecated.  See case_utils.local_uuid.demo_uuid for usage notes on its replacement, CASE_DEMO_NONRANDOM_UUID_BASE.  Proceeding with random UUIDs.",
-            DeprecationWarning,
+            FutureWarning,
         )
         return
 
@@ -49,12 +49,14 @@ def configure() -> None:
     base_dir_original_path = pathlib.Path(env_base_dir_name)
     if not base_dir_original_path.exists():
         warnings.warn(
-            "Environment variable CASE_DEMO_NONRANDOM_UUID_BASE is expected to refer to an existing directory.  Proceeding with random UUIDs."
+            "Environment variable CASE_DEMO_NONRANDOM_UUID_BASE is expected to refer to an existing directory.  Proceeding with random UUIDs.",
+            RuntimeWarning,
         )
         return
     if not base_dir_original_path.is_dir():
         warnings.warn(
-            "Environment variable CASE_DEMO_NONRANDOM_UUID_BASE is expected to refer to a directory.  Proceeding with random UUIDs."
+            "Environment variable CASE_DEMO_NONRANDOM_UUID_BASE is expected to refer to a directory.  Proceeding with random UUIDs.",
+            RuntimeWarning,
         )
         return
 
@@ -108,9 +110,9 @@ def demo_uuid() -> str:
     """
     This function generates a repeatable UUID, drawing on non-varying elements of the environment and process call for entropy.
 
-    WARNING: This function was developed for use ONLY for reducing (but not eliminating) version-control edits to identifiers in sample data.  It creates UUIDs that are decidedly NOT random, and should remain consistent on repeated calls to the importing script.
+    WARNING: This function was developed for use ONLY for reducing (but not eliminating) version-control edits to identifiers when generating sample data.  It creates UUIDs that are decidedly NOT random, and should remain consistent on repeated calls to the importing script.
 
-    To prevent accidental non-random UUID usage, an environment variable must be set to a string provided by the caller.  The variable's required value is the path to some directory.  The variable's recommended value is the equivalent of the Make variable "top_srcdir" - that is, the root directory of the containing Git repository, some parent of the current process's current working directory.
+    To prevent accidental non-random UUID usage, an environment variable, CASE_DEMO_NONRANDOM_UUID_BASE, must be set to a string provided by the caller.  The variable's required value is the path to some directory.  The variable's recommended value is the equivalent of the Make variable "top_srcdir" - that is, the root directory of the containing Git repository, some parent of the current process's current working directory.
     """
     global DEMO_UUID_BASE
     global DEMO_UUID_COUNTER

case_utils/namespace.py

@@ -17,7 +17,7 @@
 To use, add "from case_utils.namespace import *".  Namespace variables starting with "NS_" are imported.  As needs are demonstrated in CASE tooling (both in case_utils and from downstream requests), namespaces will also be imported from rdflib for a consistent "NS_*" spelling.
 """
 
-__version__ = "0.2.0"
+__version__ = "0.2.1"
 
 import rdflib
 
@@ -37,6 +37,9 @@
 NS_UCO_ACTION = rdflib.Namespace(
     "https://ontology.unifiedcyberontology.org/uco/action/"
 )
+NS_UCO_ANALYSIS = rdflib.Namespace(
+    "https://ontology.unifiedcyberontology.org/uco/analysis/"
+)
 NS_UCO_CONFIGURATION = rdflib.Namespace(
     "https://ontology.unifiedcyberontology.org/uco/configuration/"
 )