ShEx

Testing data with the Shape Expressions Language

SWAT4HCLS-2017
28 October 2017

http://shexspec.github.io/talks/2017/12-04-swat4hcls-ericp/

Eric Prud'hommeaux, <eric@w3.org>
- Sanitation Engineer, W3C
Jose Emilio Labra Gayo, <jelabra@gmail.com>
- Associate Professor, University of Oviedo
Tom Baker, <tom@tombaker.org>
- Assistant Director, DCMI
Andra Waagmeester, <andra@micelio.be>
- GeneWiki and Micelio
Kat Thornton, <katherine.thornton@yale.edu>
- Yale University
Lucas Werkmeister, <lucas.werkmeister@wikimedia.de>
- Wikimedia Deutschland e. V.
Greg Stupp, <gstupp@scripps.edu>
- The Scripps Research Institute

Shape Expressions (ShEx)

a language for describing RDF data ("graphs")
can be used for:
- Model development – parallel development of a schema and examples/test cases makes the proposal | testing | refinement process much more concrete. This applies both to initial development and subsequent model refinement
- Documentation – terse, human-readable representation helping contributors and maintainers quickly grok the model
- Legacy review – develop punch lists for existing data issues that need fixing
- Client pre-submission – submitters test their data before submission to make sure they’re saying what they want to say and that the receiving schema can accommodate all of their data
- Server pre-ingestion – submission process checks data as it comes in and either rejects or warns about non-conformant data
- Generating user interfaces
- Transforming RDF data into other data formats

RDF Data as graph and as text

<samples9298996>
  rdf:type bf:Text ;
  rdf:type bf:Work ;
  bf:title "Oliver Twist." ;
  bf:class <id.loc.gov/…/PZ3> ;
  bf:creator [
    rdf:type bf:Person ;
    bf:label "Dickens, Charles, 1812-1870." ;
  ] .

<id.loc.gov/…/PZ3>
  rdf:type bf:LCC ;
  bf:label "PZ3.D55O165PR4567" .

ShEx Model: RDF Data is tested

RDF Data ("graph") is tested
against a ShEx Schema
to produce a Validation Result

<samples9298996>
  rdf:type bf:Text ;
  rdf:type bf:Work ;
  bf:title "Oliver Twist." ;
  bf:class <id.loc.gov/…/PZ3> ;
  bf:creator [
    rdf:type bf:Person ;
    bf:label "Dickens, Charles, 1812-1870." ;
  ] .
<id.loc.gov/…/PZ3>
  rdf:type bf:LCC ;
  bf:label "PZ3.D55O165PR4567" .

ShEx Model: ...against a ShEx Schema

<Work> EXTRA rdf:type {
  rdf:type [bf:Work] ? ;
  bf:title LITERAL ;
  bf:class @<Classification> * ;
  bf:creator @<Person> OR @<Organization> + ;
  bf:derivedFrom IRI * ;
}

<Classification>
  [<http://id.loc.gov/…/>~]
AND
  EXTRA rdf:type {
    rdf:type [bf:LCC] ? ;
    bf:label LITERAL ;
  }

RDF Data ("graph") is tested
against a ShEx Schema
- which prescribes conditions that RDF graphs must meet:
  - which subjects, predicates, and objects may appear
  - ...in what combinations
  - ...with what cardinalities and datatypes
  - ...in order to produce:
a Validation Result

ShEx Model: ...to produce a Validation Result

RDF Data ("graph") is tested
against a ShEx Schema
to produce a Validation Result
- ...which flags parts of the data that do not "conform"

validating samples9298996bad as Work:
   validating http://...oliverTwist:
      Error validating http://...oliverTwist
      as nodeKind literal:
        iri found when literal expected

ShEx Model: ...on the basis of a Shape Map

RDF Data ("graph") is tested
against a ShEx Schema
to produce a Validation Result
on the basis of a Shape Map
- which designates nodes in RDF Data to be tested against ShEx Schema
- and may be generated on the basis of a query.

inst:Alice @ school:Enrollee,
inst:Bob @ school:Enrollee,
inst:Claire @ school:Enrollee,
inst:Don @ school:Enrollee

{FOCUS, foaf:age, _} @ school:Enrollee

Problem Statement

Useful data needs consistent structure:

What's there?
How's it spelled?
How's it linked?

Detect and correct errors:

missing properties,
missing/bad type arcs,
missing referents,
inconsistent state (e.g. dates),
value set violations.

@prefix : <http://www.w3.org/2012/12/rdf-val/SOTA-ex#> .
@prefix foaf: <http://xmlns.com/foaf/0.1/'> .
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .

<issue7> a :Issue , :SecurityIssue ;
    :state :unassigned ;
    :reportedBy <user6> , <user2> ; # cardinality 1
    :reportedOn "2012-12-31T23:57:00"^^xsd:dateTime ;
    :assignedTo <user2>, <user1> ;
    :assignedOn "2012-11-31T23:57:00"^^xsd:dateTime ;
                       # reproduced before being reported
    :related <issue4>, <issue3>, <issue2> .
                       # referenced issues not included

<issue4> # a ???         missing type arc
    :state :unsinged ; # misspelled
    # :reportedBy ??? -  missing
    :reportedOn "2012-12-31T23:57:00"^^xsd:dateTime .

<user2> a foaf:Person ;
    foaf:givenName "Alice" ;
    foaf:familyName "Smith" ;
    foaf:phone <tel:+1.555.222.2222> ;
    foaf:mbox <mailto:alice@example.com> .

<user6> a foaf:Agent ; # should be foaf:Person
    foaf:givenName "Bob" ; # foaf:familyName "???" - missing
    foaf:phone <tel:+.555.222.2222> ; # malformed tel: URL
    foaf:mbox <mailto:alice@example.com> .

A ShEx Schema prescribes the "shape" of RDF Data

<Work> EXTRA rdf:type {
  rdf:type [bf:Work] ? ;
  bf:title LITERAL ;
  bf:class @<Classification> * ;
  bf:creator @<Person> OR @<Organization> + ;
  bf:derivedFrom IRI * ;
}

<Classification>
  [<http://id.loc.gov/…/>~]
AND
  EXTRA rdf:type {
    rdf:type [bf:LCC] ? ;
    bf:label LITERAL ;
  }

A resource in the RDF data matching the "Work" shape:

has zero or one "rdf:type bf:Work" statements
optionally has an extra "rdf:type" statement
has exactly one "bf:title" statement with a literal value
has zero or more "bf:class" statements taking objects that match the "Classification" shape
has one or more "bf:creator" statements taking objects that match either the "Person" or the "Organization" shape
and zero or more "bf:derivedFrom" statements taking an IRI as object

A ShEx Schema prescribes the "shape" of RDF Data

<Work> EXTRA rdf:type {
  rdf:type [bf:Work] ? ;
  bf:title LITERAL ;
  bf:class @<Classification> * ;
  bf:creator @<Person> OR @<Organization> + ;
  bf:derivedFrom IRI * ;
}

<Classification>
  [<http://id.loc.gov/…/>~]
AND
  EXTRA rdf:type {
    rdf:type [bf:LCC] ? ;
    bf:label LITERAL ;
  }

A resource matching the "Classification" shape:

is identified with an IRI starting with "http://id.loc.gov..."
has zero or one "rdf:type bf:LOC" statements
optionally has an additional "rdf:type" statement
has exactly one "bf:label" statement with a literal value."

A ShEx Schema is tested against RDF Data

<Work> EXTRA rdf:type {
  rdf:type [bf:Work] ? ;
  bf:title LITERAL ;
  bf:class @<Classification> * ;
  bf:creator @<Person> OR @<Organization> + ;
  bf:derivedFrom IRI * ;
}

<Classification>
  [<http://id.loc.gov/…/>~]
AND
  EXTRA rdf:type {
    rdf:type [bf:LCC] ? ;
    bf:label LITERAL ;
  }

<samples9298996>
  rdf:type bf:Text ;
  rdf:type bf:Work ;
  bf:title "Oliver Twist." ;
  bf:class <id.loc.gov/…/PZ3> ;
  bf:creator [
    rdf:type bf:Person ;
    bf:label "Dickens, Charles, 1812-1870." ;
  ] .

<id.loc.gov/…/PZ3>
  rdf:type bf:LCC ;
  bf:label "PZ3.D55O165PR4567" .

Shapes can specify use of properties

A <Work> must have exactly one bf:title with value LITERAL:

<Work> {
  bf:title LITERAL ;
}

<Work> { … } defines a "shape".
bf:title LITERAL expects an bf:title property with a literal value.
The default cardinality is min: 1, max: 1.

<samples9298996>
  bf:title "Oliver Twist." .

<samples9298996bad>                 
  bf:title <http://...oliverTwist> .

try it

✓<samples9298996>@<Work>

✗<samples9298996bad>@!<Work>

validating samples9298996bad as Work:
   validating http://...oliverTwist:
      Error validating http://...oliverTwist
      as nodeKind literal:
        iri found when literal expected

one passed, one failed.

Shapes can specify types

Permit some data to have a type arc identifying it as a <Work>:

<Work> {
  rdf:type [bf:Work] ? ;
  bf:label LITERAL ;
}

[]s denote a value set.
This value set as only one permissible value: bf:Work.
The '?' code means min: 0, max: 1 (like regular expressions).

<samples9298996> a bf:Work ;
  bf:title "Oliver Twist." .

<samples9298996b> a bf:Work ;
  bf:title "Oliver Twist." .

<samples9298996bad> a bf:Krow ;
  bf:title "Oliver Twist" .

try it

✓<samples9298996>@<Work>

✓<samples9298996b>@<Work>

✗<samples9298996bad>@!<Work>

validating samples9298996bad as Work:
    validating http://bibframe.org/vocab/Krow:
      Error validating http://bibframe.org/vocab/Krow
      as values [<http://bibframe.org/vocab/Work>]:
        value http://bibframe.org/vocab/Krow not found in set
        [<http://bibframe.org/vocab/Work>]

Shapes can reference other shapes

Add a reference to another shape:

<Work> {
  rdf:type [bf:Work] ? ;
  bf:label LITERAL ;
  bf:class @<Classification> * ;
}

<Classification> {
  rdf:type [bf:LCC] ? ;
  bf:label LITERAL ;
}

@<Classification> references a different shape.
The '*' code means min: 0, max: * (like regular expressions).

<samples9298996>
  bf:title "Oliver Twist." ;
  bf:class <http://id.loc.gov/…/PZ3> .
<http://id.loc.gov/…/PZ3> a bf:LCC ;
  bf:label "PZ3.D55O165PR4567" .

<samples9298996b>
  bf:title "Oliver Twist." ;
  bf:class [ bf:label "PZ3.D55O165PR4567" ].

<samples9298996bad>                                    
  bf:title "Oliver Twist." ;                           
  bf:class [ a bf:LCD ; bf:label "PZ3.D55O165PR4567" ].

try it

✓<samples9298996>@<Work>

✓<samples9298996b>@<Work>

✗<samples9298996bad>@!<Work>

validating samples9298996bad as Work:
    validating _:b1:
      validating http://bibframe.org/vocab/LCD:,
      Error validating http://bibframe.org/vocab/LCD
      as values: [<http://bibframe.org/vocab/LCC>]}:
       value <http://bibframe.org/vocab/LCD> not found
       in set [<http://bibframe.org/vocab/LCC>]

Shapes can specify value sets

Or a reference to an authority:

<Work> {
  rdf:type [bf:Work] ? ;
  bf:label LITERAL ;
  bf:class @<Classification> * ;
}

<Classification>
  [<http://id.loc.gov/…/>~]

[]s denote a value set.
This value set includes every IRI starting with <http://id.loc.gov/…/>.

<samples9298996>
  bf:title "Oliver Twist." ;
  bf:class <http://id.loc.gov/…/PZ3> .

<samples9298996bad>                             
  bf:title "Oliver Twist." ;                    
  bf:class <http://id.loc.gov/authorities/PZ3> .
                                                
<samples9298996badb>                            
  bf:title "Oliver Twist." ;                    
  bf:class [ bf:label "PZ3.D55O165PR4567" ].

try it

✓<samples9298996>@<Work>

✗<samples9298996bad>@!<Work>

validating samples9298996bad as Work:
    validating <http://id.loc.gov/authorities/PZ3>:
      NodeConstraintError: expected to match
      [<http://id.loc.gov/authorities/classification/>~]

✗<samples9298996badb>@!<Work>

validating samples9298996bad as Work:
    validating _:b0:
      NodeConstraintError: expected to match
      [<http://id.loc.gov/authorities/classification/>~]

Shapes can combine constraints

Or a reference to a structure identifed by and authority:

<Work> {
  rdf:type [bf:Work] ? ;
  bf:label LITERAL ;
  bf:class @<Classification> * ;
}

<Classification>
  [<http://id.loc.gov/…/>~]
  AND {
    rdf:type [bf:LCC] ? ;
    bf:label LITERAL ;
  }

Constraints can be combined with AND, OR, NOT and ()s.

<samples9298996>
  bf:title "Oliver Twist." ;
  bf:class <http://id.loc.gov/…/PZ3> .
<http://id.loc.gov/…/PZ3> a bf:LCC ;
  bf:label "PZ3.D55O165PR4567" .

<samples9298996bad>                             
  bf:title "Oliver Twist." ;                    
  bf:class <http://id.loc.gov/authorities/PZ3> .
<http://id.loc.gov/authorities/PZ3> a bf:LCC ;  
  bf:label "PZ3.D55O165PR4567" .                
                                                
<samples9298996badb>                            
  bf:title "Oliver Twist." ;                    
  bf:class <http://id.loc.gov/…/PZ3999> .

try it

✓<samples9298996>@<Work>

✗<samples9298996bad>@!<Work>

validating samples9298996bad as Work:
    validating >http://id.loc.gov/authorities/PZ3>:
      validating <http://id.loc.gov/authorities/PZ3> as Classification:
      NodeConstraintError: expected to match [<http://id.loc.gov/…/>~]

✗<samples9298996badb>@!<Work>

validating samples9298996bad as Work:
    validating <http://id.loc.gov/…/PZ3999>:
      validating <http://id.loc.gov/…/PZ3999> as Classification:
      Missing property: <http://bibframe.org/vocab/label>

Shapes can specify choices

You may accept multiple forms of creator.

<Work> {
  rdf:type [bf:Work] ? ;
  bf:label LITERAL ;
  bf:class . * ;
  bf:creator @<Person> OR @<Organization> + ;
}

<Person> {
  rdf:type [bf:Person] ? ;
  bf:label LITERAL ;
}

<Organization> { EXTRA rdf:type
  rdf:type [bf:Organization] ;
  bf:label LITERAL ;
  org:member @<Person> OR @<Organization> *
}

<samples9298996>
  bf:title "Oliver Twist." ;
  bf:creator [ rdf:type bf:Person ;
    bf:label "Dickens, Charles, 1812-1870."
  ] .

<wp-55-45-1>
  bf:title "Nixon blasts 'false charges'" ;
  bf:creator <WP> .

<WP> rdf:type bf:Organization , bf:Newpaper ;
  bf:label "Washington Post" ;
  org:member <BobWoodward>, <CarlBernstein> .

<BobWoodward> bf:label "Bob Woodward" .
<CarlBernstein> bf:label "Carl Bernstein" .

try it

✓<samples9298996>@<Work>

✓<wp-55-45-1>@<Work>

(DRY)

see the slides: https://www.slideshare.net/jelabra/shex-by-example/7

load the examples:

shex.js: http://rawgit.com/shexSpec/shex.js/wikidata/doc/shex-simple.html?examples=
http://localhost/shexSpec/talks/2017/12-04-swat4hcls-ericp/examples.json

ShEx 2.1

IMPORT
Inheritance:
- restriction
- extension
Severity

IMPORT

IMPORT <User.shex>

:Employee {
  &:name ;
  schema:worksFor @:Company
}
 
:Company {
  schema:employee @:Employee ;
  schema:founder  @:Person ;
}

PREFIX : <http://a.example/ns#>
PREFIX schema: <http://schema.org/>

:Person { 
   $:name ( schema:name . 
          | schema:givenName . ; schema:familyName .
          ) ;
   schema:email .               
}

import examples: http://rawgit.com/shexSpec/shex.js/wikidata/doc/shex-simple.html?examples=http://shex.io/examples/import/UserEmpCo-examples.json

Inheritance

Start with a general rule about a structure:

<ObservationShape> {
  …
  :component {
    :code . ;
    :value .
  } *
}

An observation contains N components, each with a code and a value.

Inheritance

Use that structure in a derivative shape:

<ObservationShape> {
  …
  :component {
    :code . ;
    :value .
  } *
}

<BP> -<ObservationShape> {
  …
  :component {
    :code [ "systolic" ] ;
    :value .
  };
  :component {
    :code [ "diastolic" ] ;
    :value .
  }
}

An BP observation contains 2 components with codes for systolic and diastolic.

Inheritance

Use that structure in turn in an extended shape:

<ObservationShape> {
  …
  :component {
    :code . ;
    :value .
  } *
}

<BP> -<ObservationShape> {
  …
  :component {
    :code [ "systolic" ] ;
    :value .
  };
  :component {
    :code [ "diastolic" ] ;
    :value .
  }
}

<PostureBP> &<BP> {
  …
  :component {
    :code [ "posture" ] ;
    :value .
  };
}

An PostureBP observation has an additional components with a code for posture.

Severity

HIGH { value: 1, low: 1, high: 1 }
LOW { value: 1, low: 0, high: 1 }

<EventShape> {
  HIGH schema:startDate @<Date> OR @<DateTime> ;
  LOW  schema:endDate @<Date> OR @<DateTime> ;
  LOW  schema:actor @<Person>
}

<Person> {
  LOW schema:name xsd:string
}

Diseases in Wikidata

# Shape Expression for Diseases in Wikidata
PREFIX wd: <http://www.wikidata.org/entity/>
PREFIX p: <http://www.wikidata.org/prop/>
PREFIX ps: <http://www.wikidata.org/prop/statement/>
PREFIX pq: <http://www.wikidata.org/prop/qualifier/>
PREFIX pr:  <http://www.wikidata.org/prop/reference/>
PREFIX prov: <http://www.w3.org/ns/prov#>
PREFIX xsd: <http://www.w3.org/2001/XMLSchema#>
PREFIX do: <http://purl.obolibrary.org/obo/DOID_>

start = @<wikidata-disease>

<wikidata-disease> {
  p:P31 { # instance of disease
    ps:P31 [ wd:Q12136 ]; # instance of disease
    $<has-do-reference> prov:wasDerivedFrom @<do-reference>;
  } ;
  p:P279 { # subclass of
    ps:P279 @<wikidata-disease>;
    &<has-do-reference>
  } * ;
  p:P2888 EXTRA prov:wasDerivedFrom { # exact match
    ps:P2888 [ do:~ ];
    prov:wasDerivedFrom @<do-reference> ?
  } + ;
}

<do-reference> {
  # stated in
  pr:P248 @<version-disease-ontology> ;
  # retrieved
  pr:P813 xsd:dateTime ;
  # Disease ontology ID
  pr:P699 @<disease-ontology-id> ;
}

<disease-ontology-id> LITERAL /^DOID:[0-9]+$/

<version-disease-ontology> {
  # edition or translation of Disease Ontology
  p:P629 { ps:P629 [ wd:Q5282129 ] } ;
}

Wikidata items on Cancer should have an NCI Thesaurus ID

PREFIX xsd: <http://www.w3.org/2001/XMLSchema#>
PREFIX prov: <http://www.w3.org/ns/prov#>
PREFIX p: <http://www.wikidata.org/prop/>
PREFIX pr: <http://www.wikidata.org/prop/reference/>
PREFIX ps: <http://www.wikidata.org/prop/statement/>

start = @<wikidata_item>

<wikidata_item> {
  p:P1748 {
    ps:P1748 LITERAL ;
    prov:wasDerivedFrom @<reference>
  }+
}

<reference> {
  pr:P248  IRI ;
  pr:P813  xsd:dateTime ;
  pr:P699  LITERAL
}

Endpoint: https://query.wikidata.org/bigdata/namespace/wdq/sparql

Query: SELECT ?item ?itemLabel
WHERE
{ ?item wdt:P279* wd:Q12078 .
  SERVICE wikibase:label { bd:serviceParam wikibase:language "en" }
} LIMIT 10

Try it!
Click “Wikidata item on Cancer should have a NCI Thesaurus ID” on the left,
then “Get all Wikidata items on Cancers (SPARQL)” on the right.

FHIR MedicationRequest

<MedicationRequest> {
    :status @<code> AND
        {fhir:value ["active" "on-hold" "cancelled" "completed"]}?;
    :intent @<code> AND
        {fhir:value ["proposal" "plan" "order" "instance-order"]};
    ( :medicationCodeableConcept @<CodeableConcept>  |
      :medicationReference @<MedicationReference> );
    :subject ( @<PatientReference> OR @<GroupReference> );
    :requester @<MedicationRequest.requester>;
    :reasonCode @<CodeableConcept>*;
    :reasonReference
    ( { fhir:link @<Condition> } OR
      { fhir:link @<Observation> } )*;
    :dispenseRequest {
        :dispenseRequest.numberOfRepeatsAllowed @<positiveInt>?;
        :dispenseRequest.quantity @<SimpleQuantity>?;
        :dispenseRequest.performer @<Reference>? }
    :substitution @<MedicationRequest.substitution>?;
}

# Any restrictions on medication substitution
<MedicationRequest.substitution> CLOSED {
    fhir:Element.id @<string>?;             # xml:id (or equivalent in JSON)
    fhir:Element.extension @<Extension>*;   # Additional Content defined by 
                                            # implementations 
    fhir:BackboneElement.modifierExtension @<Extension>*;  # Extensions that cannot be ignored
    :substitution.allowed @<boolean>;  # Whether substitution is allowed or 
                                            # not 
    :substitution.reason @<CodeableConcept>?;  # Why should (not) substitution be 
                                            # made 
    fhir:index xsd:integer?                 # Relative position in a list
}

# Who/What requested the Request
<MedicationRequest.requester> CLOSED {
    fhir:Element.id @<string>?;             # xml:id (or equivalent in JSON)
    fhir:Element.extension @<Extension>*;   # Additional Content defined by 
                                            # implementations 
    fhir:BackboneElement.modifierExtension @<Extension>*;  # Extensions that cannot be ignored
    :requester.agent  # Who ordered the initial 
                                            # medication(s) 
    (   @<PractitionerReference> OR
        @<OrganizationReference> OR
        @<PatientReference> OR
        @<RelatedPersonReference> OR
        @<DeviceReference>
    );
    :requester.onBehalfOf @<Reference>?;  # Organization agent is acting for
    fhir:index xsd:integer?                 # Relative position in a list
}

Three interchangeable concrete syntaxes

ShExC (used in this talk)
- a compact syntax meant for human eyes and fingers
ShExJ
- a JSON-LD Javascript syntax meant for machine processing
ShExR
- the RDF interpretation of ShExJ expressed in RDF Turtle syntax

Open-source implementations

Javascript - ShEx.js (mature)
Source: https://github.com/shexSpec/shex.js
Scala - Shaclex (mature)
Source: http://labra.github.io/shaclex/
Ruby - RDF.rb (mature)
Source: https://ruby-rdf.github.io/shex/
Java - Шекс (incomplete)
Source: https://gforge.inria.fr/projects/shex-impl/
Python - shexpy (ShEx1)
Source: https://github.com/hsolbrig/shexypy
Also:
- Extensive test suite (2000+)
- Sublime Text plugin
- Jupyter Notebook kernel (in the pipeline)
- Python (ShEx2) under development
Related work: https://www.w3.org/2001/sw/wiki/ShEx

On-line Tools

ShEx.js - https://rawgit.com/shexSpec/shex.js/wikidata/doc/shex-simple.html
Shaclex - http://shaclex.herokuapp.com/
RDF.rb - rdf.greggkellogg.net/distiller?validate=true&command=shex
ShExValidata - (ShEx1) https://www.w3.org/2015/03/ShExValidata/

Shapes Constraint Language (SHACL)

similar to ShEx.
both have logical expressions (AND, OR, NOT).
uses a SPARQL rules language for constructing new tests.
implemented in a commercial IDE (TopBraid Composer).

ShEx and SHACL compared

ShEx is like XML Schema. SHACL is like Schematron.
SHACL evaluates expressions individually.
ShEx examines shape expressions as a whole for:
- increased expressivity - cardinality on groups.
  (data:sourceURL IRI; data:contentType xsd:string)?
- consistent interpretation of OR, CLOSED.
  foaf:name LITERAL | foaf:givenName LITERAL; foaf:familyName LITERAL
- support for recursive shapes and data.
  <Person> { foaf:knows @<Person> }
- simple constraints on repeated properties.
  fhir:component { fhir:code [sct:systolic]; fhir:value xsd:integer }; fhir:component { fhir:code [sct:diastolic]; fhir:value xsd:integer };
detailed comparison in book "Validating RDF Data" (http://a.co/fD3UbCw)
- See examples at https://github.com/labra/validatingRDFBookExamples/

expressivity

compare to

W3C XML Schema
regular expressions
RelaxNG

User example

A User with a name and mbox

PREFIX foaf: <http://xmlns.com/foaf/>

<UserShape> {
    (foaf:name LITERAL
     | foaf:givenName LITERAL+,
       foaf:familyName LITERAL),
    foaf:mbox IRI
}

either a foaf:name
or one or more foaf:givenNames
and a foaf:familyName
plus an foaf:mbox

User example

A User with a name and mbox

PREFIX foaf: <http://xmlns.com/foaf/>

<UserShape> {
    (foaf:name LITERAL
     | foaf:givenName LITERAL+,
       foaf:familyName LITERAL),
    foaf:mbox IRI
}

either a foaf:name
or one or more foaf:givenNames
and a foaf:familyName
plus an foaf:mbox

User example

A User with a name and mbox

PREFIX foaf: <http://xmlns.com/foaf/>

<UserShape> {
    (foaf:name LITERAL
     | foaf:givenName LITERAL+,
       foaf:familyName LITERAL),
    foaf:mbox IRI
}

either a foaf:name
or one or more foaf:givenNames
and a foaf:familyName
plus an foaf:mbox

compare with other schema languages...

RelaxNG Compact Syntax

    (element foaf:name { xsd:string }
     | (element foaf:givenName { xsd:string }+,
        element foaf:familyName { xsd:string })),
    element foaf:mbox { xsd:anyURI }

Regex

(N|(G+F))M

NM

GFM

GGGFM

W3C XML Schema

  <xs:complexType name="UserContent">
    <xs:sequence>
      <xs:choice>
        <xs:element name="name" type="xs:string"/>
        <xs:sequence>
          <xs:element maxOccurs="unbounded" name="givenName" type="xs:string"/>
          <xs:element name="familyName" type="xs:string"/>
        </xs:sequence>
      </xs:choice>
      <xs:element name="mbox" type="xs:anyURI"/>
    </xs:sequence>
  </xs:complexType>

Existing data

Objects sometimes strings, sometimes BNodes or IRIs.
- dc:creator a string or a foaf:Person in the same data.
- likewise wikipathways:complex or metacyc:…
Value set inclusion rules vary:
- enumerated set
- dereferenced enumerated set
- too large to dereference
- inclusion by lexical constraints (e.g. IRI stem)
- product of OWL union/intersection
schema.org:
- {domain,range}Includes
- ObjectProperties can be literals
- datatype "equivalences": "schema/true", "true"^^xsd:boolean, "True"
- Role member rules

future work

visualization/editing

User-oriented views of ShEx schemas?

FHIR MedicationRequest (massaged)

 <MedicationRequest> {
   :status                                @<code>;
   :intent                                @<code>;
   (
     :medicationCodeableConcept           @<CodeableConcept>  |
     :medicationReference                 @<MedicationRef> );
   :subject                               ( @<PatientRef> OR @<GroupRef> );
   :requester {
     :requester.agent                     ( @<PractitionerRef> OR @<PatientRef> );
     :requester.onBehalfOf                @<Ref>?;
   :reasonCode                            @<CodeableConcept>*;
   :reasonReference                       ( { fhir:link @<Condition> } OR { fhir:link @<Observation> } )*;
   :dispenseRequest {                    # backbone element
     :dispenseRequest.numberOfRepeatsAllowed @<positiveInt>?;
     :dispenseRequest.quantity           @<SimpleQuantity>?;
     :dispenseRequest.performer          @<Reference>? };
}

Join the ShEx Community!

W3C Shape Expressions Community Group

ShEx homepage — http://shex.io
community group page — https://www.w3.org/community/shex
ShEx Primer — http://shex.io/shex-primer/
ShEx Specification — http://shex.io/shex-semantics/
ShapeMap Specification — shex.io/shape-map/
Gitter discussions — https://gitter.im/shapeExpressions/Lobby

ShEx

Testing data with the Shape Expressions Language

SWAT4HCLS-201728 October 2017

http://shexspec.github.io/talks/2017/12-04-swat4hcls-ericp/

Shape Expressions (ShEx)

RDF Data as graph and as text

ShEx Model: RDF Data is tested

ShEx Model: ...against a ShEx Schema

ShEx Model: ...to produce a Validation Result

ShEx Model: ...on the basis of a Shape Map

Problem Statement

A ShEx Schema prescribes the "shape" of RDF Data

A ShEx Schema prescribes the "shape" of RDF Data

A ShEx Schema is tested against RDF Data

Shapes can specify use of properties

Shapes can specify types

Shapes can reference other shapes

Shapes can specify value sets

Shapes can combine constraints

Shapes can specify choices

(DRY)

ShEx 2.1

IMPORT

Inheritance

Inheritance

Inheritance

Severity

Diseases in Wikidata

Wikidata items on Cancer should have an NCI Thesaurus ID

FHIR MedicationRequest

Three interchangeable concrete syntaxes

Open-source implementations

On-line Tools

Shapes Constraint Language (SHACL)

ShEx and SHACL compared

expressivity

User example

User example

User example

RelaxNG Compact Syntax

Regex

W3C XML Schema

Existing data

future work

User-oriented views of ShEx schemas?

FHIR MedicationRequest (massaged)

Join the ShEx Community!

SWAT4HCLS-2017
28 October 2017