Update linkml dependency to latest in main? #10

New issue

Open

opened 2026-09-15 22:09:48 +00:00 by jsheunis · 3 comments

jsheunis commented 2026-09-15 22:09:48 +00:00

Member

Copy link

While generating the exports from the updated dfg-cp-survey schema (see #9), I came across a linkml error when POSTing a Contributor record to a new dumpthings backend using the updated schema:

Validation error: Validation error for instance of Contributor: cannot use 'test.ParticipationFundingClassification' as a set element (unhashable type: 'ParticipationFundingClassification')

This seems to be a known issue in LinkML: https://github.com/linkml/linkml/issues/3590, which is fixed by https://github.com/linkml/linkml/pull/3596. My understanding is that the error relates to an enum being inlined as a list (or part of an object that is inlined as a list), and before the linkml fix the code would convert such enum options to a set, which would not be hashable.

I tested the latest version of linkml main, with my local dumpthings service, and the error disappeared:

uv run --active  --with "linkml @ git+https://github.com/linkml/linkml@main#subdirectory=packages/linkml"  dump-things-service

This leads to the question: should we make this concepts toolset dependent on latest linkml? or at least a recent commit that includes the fix(es)? That is, if there are no detrimental effects on the processes and outputs related to the other schemas in this toolset.

While generating the exports from the updated `dfg-cp-survey` schema (see https://hub.trr379.de/q02/concepts.trr379.de/pulls/9), I came across a linkml error when POSTing a `Contributor` record to a new `dumpthings` backend using the updated schema: ``` Validation error: Validation error for instance of Contributor: cannot use 'test.ParticipationFundingClassification' as a set element (unhashable type: 'ParticipationFundingClassification') ``` This seems to be a known issue in LinkML: https://github.com/linkml/linkml/issues/3590, which is fixed by https://github.com/linkml/linkml/pull/3596. My understanding is that the error relates to an enum being inlined as a list (or part of an object that is inlined as a list), and before the linkml fix the code would convert such enum options to a set, which would not be hashable. I tested the latest version of linkml `main`, with my local dumpthings service, and the error disappeared: ``` uv run --active --with "linkml @ git+https://github.com/linkml/linkml@main#subdirectory=packages/linkml" dump-things-service ``` This leads to the question: should we make this concepts toolset dependent on latest linkml? or at least a recent commit that includes the fix(es)? That is, if there are no detrimental effects on the processes and outputs related to the other schemas in this toolset.

jsheunis commented 2026-09-22 06:30:03 +00:00

Author

Member

Copy link

See 9d833ebddb

See https://hub.trr379.de/q02/concepts.trr379.de/commit/9d833ebddb8e0d4e9d4ba01e455eeb591587b9f6

jsheunis commented 2026-09-24 09:45:53 +00:00

Author

Member

Copy link

I did an in-depth investigation. Explaining my approach/progress to future me or anyone else:

The starting point is the dfg-cp-survey schema (specifically with the updates in #9) and the fact that there is a submission workflo bus as explained above.

So linkml needs to be updated to the latest commit in main, specifically for dumpthings, in order to solve this issue during record POSTing. I am going off of the assumption that it is good/useful to make the same linkml version update to the schema generation workflow (i.e. on the concepts side). But perhaps this assumption needs to be questioned?

I'm first moving forward from the concept side. I've generated the schema outputs using the previous setup (linkml v1.11.1, with existing patches, all succeeding).

And then I'm generating the same outputs with a new setup:

• python v3.14
• linkml @ git+https://github.com/linkml/linkml@b4c3261fddcad14f0221ba980f6e84f9d88eb873#subdirectory=packages/linkml

With the latter setup applying the shaclgen.py patches fails, so that needs to be fixed first.

Interim comment from @c.moench:

Q from JSH: wrt patching linkml, I know of the patches in the concepts repos (e.g. here for TRR379 https://hub.trr379.de/q02/concepts.trr379.de/src/branch/main/tools/patch_linkml) that are applied in order to generate the schema outputs used by shacl-vue and dumpthings. But are linkml patches also necessary for dumpthings to run correctly? And if so, do they have to be applied explicitly or does dumpthings do that internally when you run it from the command line?
Answer: dumpthigs requires some linkml-patches to code-generators and ifabsent-handling. All patches are included in dumpthings and are dynamically applied when the service is started. So there is no need to apply them explicitly.

Here is the updated patch for the state of linkml at the specific commit:

--- shaclgen.py
+++ shaclgen.py
@@ -98,6 +98,10 @@ class ShaclGenerator(Generator):
     """parameterized suffix to be appended. No suffix per default."""
     include_annotations: bool = False
     """True means include all class / slot / type annotations in generated Node or Property shapes"""
+    exclude_order: bool = False
+    """True means exclude the automatic addition of sh:order to Property shapes"""
+    include_stripped_prefixes: bool = False
+    """True means include the unused prefixes that are typically stripped during rdflib graph serialization"""
     exclude_imports: bool = False
     """If True, elements from imported ontologies won't be included in the generator's output"""
     use_class_uri_names: bool = True
@@ -196,7 +200,10 @@ class ShaclGenerator(Generator):
     def serialize(self, **args) -> str:
         g = self.as_graph()
         fmt = "turtle" if self.format in ["owl", "ttl"] else self.format
-        return canonicalize_rdf_graph(g, output_format=fmt)
+        data = canonicalize_rdf_graph(g, output_format=fmt)
+        if self.include_stripped_prefixes:
+            data = self._add_stripped_prefixes(data)
+        return data
 
     def as_graph(self) -> Graph:
         sv = self.schemaview
@@ -270,8 +277,6 @@ class ShaclGenerator(Generator):
                         g.add((pnode, p, Literal(v, lang=self._resolve_language(s))))
 
                 prop_pv(SH.path, slot_uri)
-                prop_pv_literal(SH.order, order)
-                order += 1
                 prop_pv_text(SH.name, s.title)
                 prop_pv_text(SH.description, s.description)
 
@@ -405,6 +410,17 @@ class ShaclGenerator(Generator):
                 if default_value:
                     prop_pv(SH.defaultValue, default_value)
 
+                # Add the order automatically unless the --exclude-order flag was supplied
+                if not self.exclude_order:
+                    # sh:order may already have been added to the graph via annotations
+                    if not (pnode, SH.order, None) in g:
+                        prop_pv_literal(SH.order, order)
+                        order += 1
+                
+                # sh:path may already have been added to the graph via annotations
+                if not (pnode, SH.path, None) in g:
+                    prop_pv(SH.path, slot_uri)
+
             if self.emit_rules:
                 self._add_rules(g, class_uri_with_suffix, c)
 
@@ -845,6 +861,17 @@ class ShaclGenerator(Generator):
 
         return list_node
 
+    def _add_stripped_prefixes(self, data):
+        # First remove all prefix lines from data string
+        lines = [ln for ln in data.splitlines() if not ln.strip().startswith("@prefix")]
+        # Now generate alphabetized lines from all prefixes:
+        prefix_lines = [
+            f"@prefix {str(pfx.prefix_prefix)}: <{pfx.prefix_reference}> ."
+            for pfx in sorted(self.schema.prefixes.values(), key=lambda p: str(p.prefix_prefix).lower())
+        ]
+        # return the new prefix lines joined with prefix-stripped data
+        return "\n".join(prefix_lines + [""] + lines)
+
 
 def add_simple_data_type(func: Callable, r: ElementName) -> None:
     for datatype in list(ShaclDataType):
@@ -873,6 +900,19 @@ def add_simple_data_type(func: Callable, r: ElementName) -> None:
     show_default=True,
     help="Use --include-annotations to include annotations of slots, types, and classes in the generated SHACL shapes.",
 )
+@click.option(
+    "--exclude-order/--include-order",
+    default=False,
+    show_default=True,
+    help="Use --exclude-order to exclude sh:order from being added automatically to generated SHACL property shapes. "
+    "Default behavior adds order. This flag has no influence on sh:order being added via annotations.",
+)
+@click.option(
+    "--include-stripped-prefixes/--exclude-stripped-prefixes",
+    default=False,
+    show_default=True,
+    help="Use --include-stripped-prefixes to include prefixes that are unused and hence stripped during rdflib serialization.",
+)
 @click.option(
     "--exclude-imports/--include-imports",
     default=False,

This now runs without failure.

Now I have dfg-cp-survey schema outputs for both versions of linkml.

First check: there is no difference between the two yaml files.

It is difficult to determine differences in shacl outputs, because linkml changed the way it renders the TTL (property shapes were embedded in node shapes previously, now they are only referenced)

**Second check: in the exported SHACL, the prefixes are different for the family_name slot of the Contributor class: **

• linkml 1.11.1: dlthings:Person -> dlthings:family_name
• linkml 1.11.1: trr379cps:Contributor -> dlthings:family_name
• linkml main: dlthings:Person -> dlthings:family_name
• linkml main: trr379cps:Contributor -> trr379cps:family_name

I did an in-depth investigation. Explaining my approach/progress to future me or anyone else: The starting point is the `dfg-cp-survey` schema (specifically with the updates in https://hub.trr379.de/q02/concepts.trr379.de/pulls/9) and the fact that there is a submission workflo bus as explained above. So `linkml` needs to be updated to the latest commit in `main`, specifically for `dumpthings`, in order to solve this issue during record POSTing. I am going off of the assumption that it is good/useful to make the same `linkml` version update to the schema generation workflow (i.e. on the concepts side). But perhaps this assumption needs to be questioned? I'm first moving forward from the concept side. I've generated the schema outputs using the previous setup (linkml v1.11.1, with existing patches, all succeeding). And then I'm generating the same outputs with a new setup: - python v3.14 - linkml @ git+https://github.com/linkml/linkml@b4c3261fddcad14f0221ba980f6e84f9d88eb873#subdirectory=packages/linkml With the latter setup applying the `shaclgen.py` patches fails, so that needs to be fixed first. Interim comment from @c.moench: > Q from JSH: wrt patching linkml, I know of the patches in the concepts repos (e.g. here for TRR379 https://hub.trr379.de/q02/concepts.trr379.de/src/branch/main/tools/patch_linkml) that are applied in order to generate the schema outputs used by shacl-vue and dumpthings. But are linkml patches also necessary for dumpthings to run correctly? And if so, do they have to be applied explicitly or does dumpthings do that internally when you run it from the command line? > Answer: dumpthigs requires some linkml-patches to code-generators and ifabsent-handling. All patches are included in dumpthings and are dynamically applied when the service is started. So there is no need to apply them explicitly. Here is the updated patch for the state of linkml at the specific commit: ```diff --- shaclgen.py +++ shaclgen.py @@ -98,6 +98,10 @@ class ShaclGenerator(Generator): """parameterized suffix to be appended. No suffix per default.""" include_annotations: bool = False """True means include all class / slot / type annotations in generated Node or Property shapes""" + exclude_order: bool = False + """True means exclude the automatic addition of sh:order to Property shapes""" + include_stripped_prefixes: bool = False + """True means include the unused prefixes that are typically stripped during rdflib graph serialization""" exclude_imports: bool = False """If True, elements from imported ontologies won't be included in the generator's output""" use_class_uri_names: bool = True @@ -196,7 +200,10 @@ class ShaclGenerator(Generator): def serialize(self, **args) -> str: g = self.as_graph() fmt = "turtle" if self.format in ["owl", "ttl"] else self.format - return canonicalize_rdf_graph(g, output_format=fmt) + data = canonicalize_rdf_graph(g, output_format=fmt) + if self.include_stripped_prefixes: + data = self._add_stripped_prefixes(data) + return data def as_graph(self) -> Graph: sv = self.schemaview @@ -270,8 +277,6 @@ class ShaclGenerator(Generator): g.add((pnode, p, Literal(v, lang=self._resolve_language(s)))) prop_pv(SH.path, slot_uri) - prop_pv_literal(SH.order, order) - order += 1 prop_pv_text(SH.name, s.title) prop_pv_text(SH.description, s.description) @@ -405,6 +410,17 @@ class ShaclGenerator(Generator): if default_value: prop_pv(SH.defaultValue, default_value) + # Add the order automatically unless the --exclude-order flag was supplied + if not self.exclude_order: + # sh:order may already have been added to the graph via annotations + if not (pnode, SH.order, None) in g: + prop_pv_literal(SH.order, order) + order += 1 + + # sh:path may already have been added to the graph via annotations + if not (pnode, SH.path, None) in g: + prop_pv(SH.path, slot_uri) + if self.emit_rules: self._add_rules(g, class_uri_with_suffix, c) @@ -845,6 +861,17 @@ class ShaclGenerator(Generator): return list_node + def _add_stripped_prefixes(self, data): + # First remove all prefix lines from data string + lines = [ln for ln in data.splitlines() if not ln.strip().startswith("@prefix")] + # Now generate alphabetized lines from all prefixes: + prefix_lines = [ + f"@prefix {str(pfx.prefix_prefix)}: <{pfx.prefix_reference}> ." + for pfx in sorted(self.schema.prefixes.values(), key=lambda p: str(p.prefix_prefix).lower()) + ] + # return the new prefix lines joined with prefix-stripped data + return "\n".join(prefix_lines + [""] + lines) + def add_simple_data_type(func: Callable, r: ElementName) -> None: for datatype in list(ShaclDataType): @@ -873,6 +900,19 @@ def add_simple_data_type(func: Callable, r: ElementName) -> None: show_default=True, help="Use --include-annotations to include annotations of slots, types, and classes in the generated SHACL shapes.", ) +@click.option( + "--exclude-order/--include-order", + default=False, + show_default=True, + help="Use --exclude-order to exclude sh:order from being added automatically to generated SHACL property shapes. " + "Default behavior adds order. This flag has no influence on sh:order being added via annotations.", +) +@click.option( + "--include-stripped-prefixes/--exclude-stripped-prefixes", + default=False, + show_default=True, + help="Use --include-stripped-prefixes to include prefixes that are unused and hence stripped during rdflib serialization.", +) @click.option( "--exclude-imports/--include-imports", default=False, ``` This now runs without failure. Now I have `dfg-cp-survey` schema outputs for both versions of linkml. **First check: there is no difference between the two yaml files.** It is difficult to determine differences in shacl outputs, because linkml changed the way it renders the TTL (property shapes were embedded in node shapes previously, now they are only referenced) **Second check: in the exported SHACL, the prefixes are different for the `family_name` slot of the `Contributor` class: ** - linkml `1.11.1`: `dlthings:Person` -> `dlthings:family_name` - linkml `1.11.1`: `trr379cps:Contributor` -> `dlthings:family_name` - linkml `main`: `dlthings:Person` -> `dlthings:family_name` - linkml `main`: `trr379cps:Contributor` -> `trr379cps:family_name`

jsheunis commented 2026-09-25 07:23:54 +00:00

Author

Member

Copy link

@c.moench pointed out the following:

I looks as if the slot-usage declarations change the slot URI. If the following:

    slot_usage:
      family_name:
        required: true
      given_name:
        required: true

is not stated in the schema, the JSON->TTL conversion will emit dlthings:family_name and dlthings:given_name.

This cann be mitigated by declaring the slot_usage as follows:

    slot_usage:
      family_name:
        required: true
        slot_uri: dlthings:family_name
      given_name:
        required: true
        slot_uri: dlthings:given_name

I am not sure whether this is a bug or a feature in linkml.

I confirmed that applying these changes to the schema also has the desired effect in SHACL export.

Looking into the LinkML code:

• in the shacl generator, it is coming from Generator.schemaview.class_induced_slots()
• Generator.schemaview comes from SchemaView, which is imported from linkml_runtime
• This looks like the origin: https://github.com/linkml/linkml/blame/main/packages/linkml_runtime/src/linkml_runtime/utils/schemaview.py#L1762

@c.moench pointed out the following: > I looks as if the slot-usage declarations change the slot URI. If the following: ```yaml slot_usage: family_name: required: true given_name: required: true ``` > is not stated in the schema, the JSON->TTL conversion will emit `dlthings:family_name` and `dlthings:given_name`. > This cann be mitigated by declaring the `slot_usage` as follows: ```yaml slot_usage: family_name: required: true slot_uri: dlthings:family_name given_name: required: true slot_uri: dlthings:given_name ``` > I am not sure whether this is a bug or a feature in linkml. I confirmed that applying these changes to the schema also has the desired effect in SHACL export. Looking into the LinkML code: - in the shacl generator, it is coming from `Generator.schemaview.class_induced_slots()` - `Generator.schemaview` comes from `SchemaView`, which is imported from `linkml_runtime` - This looks like the origin: https://github.com/linkml/linkml/blame/main/packages/linkml_runtime/src/linkml_runtime/utils/schemaview.py#L1762

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

dfg-cps-2026

shaclpatch

No results found.

Labels

Clear labels   

bug

Something is not working

  

duplicate

This issue or pull request already exists

  

enhancement

New feature

  

help wanted

Need some help

  

invalid

Something is wrong

  

question

More information is needed

  

wontfix

This won't be fixed

No labels

bug

duplicate

enhancement

help wanted

invalid

question

wontfix

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

q02/concepts.trr379.de#10

No description provided.