Consistent RDF permalinks with content negotiation #158

MarkRobbo · 2017-08-21T15:13:06Z

This pull request tracks progress on RDF permalinking

Implement controller methods proposed in Consistent RDF permalinks with content negotiation #146
RDF tidying with permalinks
Do not wipe old commit ID version from the database when the branch expires, workaround for Separate Git branch references from the Workflow concept #164
Support for packed workflows

Closes #146

It's not really RESTful

Only what is currently stored in the SPARQL endpoint

# Conflicts: # src/main/java/org/commonwl/view/git/GitDetails.java

Previously regenerated using cwltool, which will take longer and is impossible for generic git additions

# Conflicts: # src/main/java/org/commonwl/view/cwl/CWLToolRunner.java # src/main/java/org/commonwl/view/workflow/WorkflowService.java

Workaround for common-workflow-language/cwltool#427

stain · 2017-08-24T14:30:04Z

src/main/java/org/commonwl/view/researchobject/ROBundleService.java

            // Tool attribution in createdBy
            manifest.setCreatedBy(appAgent);

            // Retrieval Info
            // TODO: Make this importedBy/On/From
            manifest.setRetrievedBy(appAgent);
            manifest.setRetrievedOn(manifest.getCreatedOn());
-            manifest.setRetrievedFrom(new URI(gitInfo.getUrl()));
+            manifest.setRetrievedFrom(new URI(id + "?format=ro"));


id might include an #anchor, but ?query must go before anchors.

stain · 2017-08-24T14:40:48Z

src/main/java/org/commonwl/view/researchobject/ROBundleService.java

@@ -159,12 +164,14 @@ public Bundle createBundle(Workflow workflow, GitDetails gitInfo) throws IOExcep
            File png = graphVizService.getGraph(workflow.getID() + ".png", workflow.getVisualisationDot(), "png");
            Files.copy(png.toPath(), bundleRoot.resolve("visualisation.png"));
            PathMetadata pngAggr = bundle.getManifest().getAggregation(bundleRoot.resolve("visualisation.png"));
-            pngAggr.setRetrievedFrom(new URI(appAgent.getUri() + workflow.getVisualisationPng()));
+            pngAggr.setRetrievedFrom(new URI("https://w3id.org/cwl/v/git" + workflow.getLastCommit()


This breaks for packed nested workflows as we would (if you look at two ROs) falsely say pav:retrievedFrom <thesame> for two different workflows, when actually none of those would be able to download (as it now throw 303).

Perhaps if we are anyway going for ?query parameters we could have &part=main to select the packed part? Trouble is that ordering of parameters in the URL matters semantically - so blah.cwl?part=main&format=png or blah.cwl?format=png&part=main?

(alternatively blah.cwl%23main?format=png as in HTML, or a new style like blah.cwl;?format=png ??)

stain · 2017-08-24T14:41:32Z

src/main/java/org/commonwl/view/researchobject/ROBundleService.java


            File svg = graphVizService.getGraph(workflow.getID() + ".svg", workflow.getVisualisationDot(), "svg");
            Files.copy(svg.toPath(), bundleRoot.resolve("visualisation.svg"));
            PathMetadata svgAggr = bundle.getManifest().getAggregation(bundleRoot.resolve("visualisation.svg"));
-            svgAggr.setRetrievedFrom(new URI(appAgent.getUri() + workflow.getVisualisationSvg()));
+            svgAggr.setRetrievedFrom(new URI("https://w3id.org/cwl/v/git" + workflow.getLastCommit()


... you said "missing slash after /git" :)

stain · 2017-08-24T14:46:28Z

src/main/java/org/commonwl/view/researchobject/ROBundleService.java

-                    logger.error("Could not get RDF for workflow from raw URL", ex.getMessage());
-                }
+            String rdfUrl = wfDetails.getUrl(workflow.getLastCommit()).replace("https://", "");
+            if (rdfService.graphExists(rdfUrl)) {


rdfService should no longer use http://localhost:3030/sparql or similar as base URL, but should use our new permalinks so they match 1:1 with the RDF resource we are responding for here (needs to include the ?part or #part - but NOT ?format) - e.g. https://w3id.org/cwl/v/git/deadbeef1337/packedworkflow.cwl?part=main

stain · 2017-08-24T14:50:18Z

src/main/java/org/commonwl/view/workflow/WorkflowController.java

-        return getGraph("svg", gitDetails, response);
+        response.setHeader("Content-Disposition", "inline; filename=\"graph.svg\"");
+        FileSystemResource test = workflowService.getWorkflowGraph("svg", gitDetails);
+        return test;


Unnecessary test variable, return directly

stain · 2017-08-24T14:56:43Z

src/main/java/org/commonwl/view/workflow/WorkflowPermalinkController.java

+    public byte[] getRdfAsTurtle(@PathVariable("commitid") String commitId,
+                                 HttpServletRequest request) {
+        Workflow workflow = getWorkflow(commitId, request);
+        String rdfUrl = workflow.getRetrievedFrom().getUrl(commitId).replace("https://", "");


No need to fetch workflow first as rdfUrl is (soon) directly creatable from commitId and request path.

stain · 2017-08-24T14:58:06Z

src/main/java/org/commonwl/view/workflow/WorkflowPermalinkController.java

+     * @param request The HttpServletRequest from the controller to extract path
+     * @throws WorkflowNotFoundException If workflow could not be found (404)
+     */
+    private Workflow getWorkflow(String commitId, HttpServletRequest request) throws WorkflowNotFoundException {


Need a similar getWorkflowURI() that reconstructs the https://w3id.org/cwl/v/... graph URI

stain · 2017-08-24T15:00:14Z

src/main/java/org/commonwl/view/workflow/WorkflowService.java

+                    // Add the old commit for the purposes of permalinks
+                    // TODO: Separate concept of commit from branch ref, see #164
+                    GitDetails byOldCommitId = workflow.getRetrievedFrom();
+                    byOldCommitId.setBranch(workflow.getLastCommit());


Does this still work OK if a Workflow already exists with that commit explicitly set?

stain · 2017-08-24T15:02:52Z

src/test/java/org/commonwl/view/researchobject/ROBundleServiceTest.java

@@ -136,7 +145,7 @@ public void generateAndSaveROBundle() throws Exception {
        // Check cwl aggregation information
        PathMetadata cwlAggregate = manifest.getAggregation(
                bundleRoot.resolve("lobSTR-workflow.cwl"));
-        assertEquals("https://raw.githubusercontent.com/common-workflow-language/workflows/933bf2a1a1cce32d88f88f136275535da9df0954/lobstr-draft3/lobSTR-workflow.cwl",
+        assertEquals("https://w3id.org/cwl/v/git/null/lobstr-draft3/lobSTR-workflow.cwl?format=raw",


Could this accidentally be null for other reasons? Rather mock a fake value like deadbeef1337

Avoids conflicts with CWL spec versions like https://w3id.org/cwl/v1.0

MarkRobbo · 2017-08-25T08:57:28Z

Changes are now finished with the exception of the use of query parameters - currently # parameters are used to distinguish packed workflows

Missed one

MarkRobbo and others added 26 commits August 21, 2017 11:08

Rename WorkflowRESTController -> WorkflowJSONController

0e6d8ba

It's not really RESTful

Add permalink controller method stubs

be1ac59

Correct mime type for Research Object Bundles

cd5fdb3

Add simple redirects to viewer and raw URL

f4e327d

Refactor and addition of RO bundle content negotiation

3410e79

Add image viewing through content negotiation

258514b

Add basic RDF through content negotiation

4f7b740

Only what is currently stored in the SPARQL endpoint

Merge branch 'packed-workflows' into rdf-permalinks

25ff4d2

# Conflicts: # src/main/java/org/commonwl/view/git/GitDetails.java

RO bundle gets RDF model from local store

46683c7

Previously regenerated using cwltool, which will take longer and is impossible for generic git additions

Refactor tests

64f5b51

Refactor to use Path instead of File objects

3e756d3

Merge remote-tracking branch 'origin/master' into rdf-permalinks

ffd8f26

# Conflicts: # src/main/java/org/commonwl/view/cwl/CWLToolRunner.java # src/main/java/org/commonwl/view/workflow/WorkflowService.java

Content negotiation via query parameters and use these permalinks in RO

ba8bd9f

Fix resolving of static resources

2dc657a

Update RDFS namespace

6f0250b

Workaround for common-workflow-language/cwltool#427

Add correct permalinks to overall retrievedFrom and ID

7d27475

Keep HTTPServletResponse object within controller

03903b5

RO bundle permalink testing

0e9e008

Create a version added by commit ID when branches move

a5cbfe2

Test coverage for PermalinkController

7bfc307

Require jsonld-java 0.9.0 for writing using jena

7bc2daf

Test coverage for WorkflowController generic git

fb914f4

Improve error handling

e9880e5

Add packed ID to workflow permalink

67ea196

Merge branch 'master' into rdf-permalinks

74d43db

Fix merge conflict

a90cbdd

MarkRobbo changed the title ~~WIP: Consistent RDF permalinks with content negotiation~~ Consistent RDF permalinks with content negotiation Aug 24, 2017

Add missing slashes to visualisation permalink

1f69623

stain requested changes Aug 24, 2017

View reviewed changes

Fix ordering of format parameter in workflow RO ID

5f81d9a

MarkRobbo added 4 commits August 24, 2017 16:30

Check if workflow already exists before creating during cache update

adb6b8c

Permalinks for graph names in SPARQL store

92a7d06

Change prefix from /v/git to /view/git

acebcde

Avoids conflicts with CWL spec versions like https://w3id.org/cwl/v1.0

Test for permalinks in RO and RDF

0aeec6a

MarkRobbo added 4 commits August 25, 2017 10:00

Fix use of null in tests

a062122

Missed one

No format URL when empty format is given

bebabcf

Simplify SPARQL queries

5afde1e

Fix commit ID used in test RDF file

ac24cee

stain approved these changes Aug 25, 2017

View reviewed changes

stain merged commit 6196d34 into master Aug 25, 2017

stain deleted the rdf-permalinks branch August 25, 2017 10:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Consistent RDF permalinks with content negotiation #158

Consistent RDF permalinks with content negotiation #158

MarkRobbo commented Aug 21, 2017 •

edited

Loading

stain Aug 24, 2017

stain Aug 24, 2017

stain Aug 24, 2017

stain Aug 24, 2017

stain Aug 24, 2017

stain Aug 24, 2017

stain Aug 24, 2017

stain Aug 24, 2017

stain Aug 24, 2017

MarkRobbo commented Aug 25, 2017

Consistent RDF permalinks with content negotiation #158

Consistent RDF permalinks with content negotiation #158

Conversation

MarkRobbo commented Aug 21, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MarkRobbo commented Aug 25, 2017

MarkRobbo commented Aug 21, 2017 •

edited

Loading