[saxon] xsl:use-package name resolution

Discussion:

Christophe Marchand

2017-05-05 19:22:07 UTC

Hello !
I'm looking for a way to map Xslt 3.0 packages to java naming
conventions, and to our project definitions.

Today, we only have XSLT 2.0, very very few XSLT3, but we are trying to
move to XSLT3.0.

Our projects are maven projects, and to avoid code duplication in Xsl,
we use maven dependencies.

Inside each project we have :
src
main
java
...
xsl
package
directory
tree
file1.xsl
otherTree
file2.xsl
test
java
...
xspec
package
directory
tree
file1.xspec
otherTree
file2.xspec

If file2.xsl needs file1.xsl, it has <xsl:include href="../tree/file1.xsl"/>
If file1 needs a xsl from another project, then it has
<xsl:include href="otherProjectArtifactId:/path/to/other.xsl"/>
And in pom we have a dependency to otherProject, and a maven-plugin
generates a catalog, with a rewrite-uri :

<rewriteURI uriStartString="http://domain.name/groupId/artifactId"
rewritePrefix="src/main/xsl" />
for use-package in the same maven project
and
<rewriteURI uriStartString="otherProjectArtifactId:/"
rewritePrefix="file:/home/xxx/.m2/repository/.../artifactId.jar" />
for other maven projects.

This allow to locate
http://domain.name/groupId/artifactId/package/tree/package2.xsl in
src/main/xsl/package/tree/package2.xsl

This works perfectly, without package-version.

For package-version, I can do some tricky things with artifact version,
but I'm not quite clear on what can be done, or not...

I have some questions on specification and Saxon implementation :
- do many xsl files belong to the same package ?
- How does Saxon 9.7EE resolve package location ?
- Must I compile packages before they are being used ?
- Does package-name resolution is based on URIResolver ? May we still
use catalogs for package-name location ?

Any help, suggestions, comments are welcome.

Best regards,
Christophe

Michael Kay

2017-05-09 22:14:05 UTC

Permalink

Post by Christophe Marchand
Hello !
I'm looking for a way to map Xslt 3.0 packages to java naming
conventions, and to our project definitions.
....
- do many xsl files belong to the same package ?

Yes. An XSLT package is a set of XSLT modules. Modules within a package are related via xsl:include or xsl:import; packages are related via xsl:use-package.

There's a difference of emphasis in the use of URIs: xsl:include and xsl:import have a presumption that the href attribute gives the location of the XSLT source of a module, unless redirected using catalogs or URIResolvers. xsl:use-package has a presumption that the package name is an abstract identifier, independent of location. In addition (and related) XSLT 3.0 assumes that packages can be deployed in multiple locations, in either source or compiled format, without the package name changing.

Post by Christophe Marchand
- How does Saxon 9.7EE resolve package location ?

There's a number of mechanisms, and the whole area should be regarded as work in progress. At the s9api level, you're expected to compile packages using XsltCompiler.compilePackage() which is given the actual resource (file or stream) containing the top-level module of the package; you can also import a compiled package; the XsltCompiler then builds up a library of known packages, which is searched when resolving an xsl:use-package declaration. There's also a very primitive "package catalog" concept but this will be replaced in 9.8.

Post by Christophe Marchand
- Must I compile packages before they are being used ?

You need to distinguish compilation (which produces a data structure in memory), export (which serializes that data structure to an XML document typically in filestore), import (which reconstructs the data structure from the export file), and bytecode generation (which produces in-memory bytecode from the in-memory data structure).

The "compilation" process is essential. Export/Import is needed only if you want to distribute a compiled package, or to speed up package loading. Bytecode generation is optional.

Post by Christophe Marchand
- Does package-name resolution is based on URIResolver ? May we still
use catalogs for package-name location ?

URIResolvers and (oasis) catalogs are not used for the process of getting a package location from a package name and version. They might be used for the process of resolving the package location - I would need to check and it may depend on which particular API you are using.

Hope this helps. Please bear in mind that this whole area is very underdeveloped in Saxon 9.7 and will evolve in future releases.

One of the issues I am still thinking about is that the relationship between source packages and compiled packages is not one-to-one: when compiling a package you bind values for its static parameters, so you can for example create two compiled packages, one compiled with streamable=yes and one with streamable=no. This means that when xsl:use-package references a package by name and version, that's not necessarily enough to tell us which of these compiled packages we should pick up.

Michael Kay
Saxonica

Post by Christophe Marchand
Any help, suggestions, comments are welcome.
Best regards,
Christophe
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
saxon-help mailing list archived at http://saxon.markmail.org/
https://lists.sourceforge.net/lists/listinfo/saxon-help

c***@oxiane.com

2017-05-10 08:19:35 UTC

Permalink

Thanks a lot, Michael. Your responses open various other questions, and
I'm gonna start new threads for these questions, quoting this one when
necessary.

Best regards,
Christophe

Post by Michael Kay

Post by Christophe Marchand
Hello !
I'm looking for a way to map Xslt 3.0 packages to java naming
conventions, and to our project definitions.
....
- do many xsl files belong to the same package ?

Yes. An XSLT package is a set of XSLT modules. Modules within a
package are related via xsl:include or xsl:import; packages are
related via xsl:use-package.
There's a difference of emphasis in the use of URIs: xsl:include and
xsl:import have a presumption that the href attribute gives the
location of the XSLT source of a module, unless redirected using
catalogs or URIResolvers. xsl:use-package has a presumption that the
package name is an abstract identifier, independent of location. In
addition (and related) XSLT 3.0 assumes that packages can be deployed
in multiple locations, in either source or compiled format, without
the package name changing.

Post by Christophe Marchand
- How does Saxon 9.7EE resolve package location ?

There's a number of mechanisms, and the whole area should be regarded
as work in progress. At the s9api level, you're expected to compile
packages using XsltCompiler.compilePackage() which is given the actual
resource (file or stream) containing the top-level module of the
package; you can also import a compiled package; the XsltCompiler then
builds up a library of known packages, which is searched when
resolving an xsl:use-package declaration. There's also a very
primitive "package catalog" concept but this will be replaced in 9.8.

Post by Christophe Marchand
- Must I compile packages before they are being used ?

You need to distinguish compilation (which produces a data structure
in memory), export (which serializes that data structure to an XML
document typically in filestore), import (which reconstructs the data
structure from the export file), and bytecode generation (which
produces in-memory bytecode from the in-memory data structure).
The "compilation" process is essential. Export/Import is needed only
if you want to distribute a compiled package, or to speed up package
loading. Bytecode generation is optional.

Post by Christophe Marchand
- Does package-name resolution is based on URIResolver ? May we still
use catalogs for package-name location ?

URIResolvers and (oasis) catalogs are not used for the process of
getting a package location from a package name and version. They might
be used for the process of resolving the package location - I would
need to check and it may depend on which particular API you are using.
Hope this helps. Please bear in mind that this whole area is very
underdeveloped in Saxon 9.7 and will evolve in future releases.
One of the issues I am still thinking about is that the relationship
between source packages and compiled packages is not one-to-one: when
compiling a package you bind values for its static parameters, so you
can for example create two compiled packages, one compiled with
streamable=yes and one with streamable=no. This means that when
xsl:use-package references a package by name and version, that's not
necessarily enough to tell us which of these compiled packages we
should pick up.
Michael Kay
Saxonica

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
saxon-help mailing list archived at http://saxon.markmail.org/
https://lists.sourceforge.net/lists/listinfo/saxon-help

c***@oxiane.com

2017-05-10 09:43:18 UTC

Permalink

Post by Michael Kay
Yes. An XSLT package is a set of XSLT modules. Modules within a
package are related via xsl:include or xsl:import; packages are
related via xsl:use-package.

Suppose I have 3 modules :
lib1.xsl xsl:package/@name=http://mydomain.com/xsl/lib
lib2.xsl xsl:package/@name=http://mydomain.com/xsl/lib
program.xsl xsl:package/@name=http://mydomain.com/xsl/undefined

program.xsl has a <xsl:use-package name="http://mydomain.com/xsl/lib" />

Does that mean I can call all visible templates, functions, variables
from lib1.xsl and from lib2.xsl ?

Post by Michael Kay
xslCompiler.importPackage(xslCompiler.compilePackage(new
StreamSource("lib1.xsl")));
xslCompiler.importPackage(xslCompiler.compilePackage(new
StreamSource("lib2.xsl")));
XslExecutable program = xslCompiler.compile(new
StreamSource("program.xsl"));
XsltTransformer tr = program.load();
tr.setSource(input);
tr.transform();

Or must I get the XsltExecutable from compiler.compilePackage(new
StreamSource("program.xsl")).link(); ?

Must I use the same XslCompiler for the 3 modules ?
Must I compile lib1.xsl and lib2.xsl BEFORE compiling program.xsl ?

Is there one PackageLibrary per XsltCompiler ?

XslCompiler.compilePackage() may return an incomplete package ; well,
XsltPackage seems to be the compiled form of a XSL module. When we
import a package (into a XsltCompiler), does the underlying
PackageLibrary is updated ?

I think there is a typing issue in
http://www.saxonica.com/html/documentation/javadoc/net/sf/saxon/s9api/XsltPackage.html#link()
[form/from]

Best regards,
Christophe

Michael Kay

2017-05-10 10:30:39 UTC

Permalink

Post by c***@oxiane.com

Post by Michael Kay
Yes. An XSLT package is a set of XSLT modules. Modules within a
package are related via xsl:include or xsl:import; packages are
related via xsl:use-package.

I think you mean 3 single-module packages?
You can't have two different packages with the same name. Or at any rate, strange things may happen if you do. Certainly xsl:use-package is only going to pick up one of them.

Post by c***@oxiane.com

Yes, that should work.

Post by c***@oxiane.com
Or must I get the XsltExecutable from compiler.compilePackage(new
StreamSource("program.xsl")).link(); ?
Must I use the same XslCompiler for the 3 modules ?

An XsltPackage created by one XsltCompiler can be imported into (the package library maintained by) another XsltCompiler. When you compile a package, xsl:use-package requests will only be satisfied by packages known to the originating XsltCompiler, but they might have originally been compiled elsewhere.

Post by c***@oxiane.com
Must I compile lib1.xsl and lib2.xsl BEFORE compiling program.xsl ?

There is an API that allows you to submit multiple packages in a single call (XsltCompiler.compilePackages()), and which attempts to sort out the dependencies between packages, but I don't think this is particularly robust or stable in 9.7 and it's probably best avoided.

Post by c***@oxiane.com
Is there one PackageLibrary per XsltCompiler ?

Yes.

Post by c***@oxiane.com
XslCompiler.compilePackage() may return an incomplete package ; well,
XsltPackage seems to be the compiled form of a XSL module. When we
import a package (into a XsltCompiler), does the underlying
PackageLibrary is updated ?

Yes.

Post by c***@oxiane.com
I think there is a typing issue in
http://www.saxonica.com/html/documentation/javadoc/net/sf/saxon/s9api/XsltPackage.html#link()
[form/from]

I don't think so. It says "to form an executable stylesheet" and that's what it means to say. Some punctuation would help.

Michael Kay
Saxonica

c***@oxiane.com

2017-05-10 12:47:29 UTC

Permalink

I certainly misunderstand the "package" term. Could you answer by true /
false to the following propositions ?

- Only one file can define a given URI for /xsl:package/@name

- A package may use other packages through xsl:use-package

- A package may include / import other xsl files through "old"
xsl:include / xsl:import. So, many files can compose a whole package ;
but a package identified by a URI is defined by one and only one
xsl:package file.

- The package file may contain xsl code, but exists mainly to define the
interface that can be use when this package is used. The code may be in
included / imported modules (is module the right term, here ?)

Best,
Christophe

Post by Michael Kay

Post by c***@oxiane.com

Post by Michael Kay
Yes. An XSLT package is a set of XSLT modules. Modules within a
package are related via xsl:include or xsl:import; packages are
related via xsl:use-package.

I think you mean 3 single-module packages?
You can't have two different packages with the same name. Or at any
rate, strange things may happen if you do. Certainly xsl:use-package
is only going to pick up one of them.

Post by c***@oxiane.com

Yes, that should work.

Post by c***@oxiane.com
Or must I get the XsltExecutable from compiler.compilePackage(new
StreamSource("program.xsl")).link(); ?
Must I use the same XslCompiler for the 3 modules ?

An XsltPackage created by one XsltCompiler can be imported into (the
package library maintained by) another XsltCompiler. When you compile
a package, xsl:use-package requests will only be satisfied by packages
known to the originating XsltCompiler, but they might have originally
been compiled elsewhere.

Post by c***@oxiane.com
Must I compile lib1.xsl and lib2.xsl BEFORE compiling program.xsl ?

There is an API that allows you to submit multiple packages in a
single call (XsltCompiler.compilePackages()), and which attempts to
sort out the dependencies between packages, but I don't think this is
particularly robust or stable in 9.7 and it's probably best avoided.

Post by c***@oxiane.com
Is there one PackageLibrary per XsltCompiler ?

Yes.

Post by c***@oxiane.com
I think there is a typing issue in
http://www.saxonica.com/html/documentation/javadoc/net/sf/saxon/s9api/XsltPackage.html#link()
[form/from]

I don't think so. It says "to form an executable stylesheet" and
that's what it means to say. Some punctuation would help.
Michael Kay
Saxonica
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
saxon-help mailing list archived at http://saxon.markmail.org/
https://lists.sourceforge.net/lists/listinfo/saxon-help

Michael Kay

2017-05-10 20:08:23 UTC

Permalink

For starters, have you read the overview at

https://www.w3.org/TR/xslt-30/#packages-and-modules

?

Post by c***@oxiane.com
I certainly misunderstand the "package" term. Could you answer by true /
false to the following propositions ?

Sorry, I don't know what it means for a file to define a URI.

Post by c***@oxiane.com
- A package may use other packages through xsl:use-package

Correct.

Post by c***@oxiane.com
- A package may include / import other xsl files through "old"
xsl:include / xsl:import. So, many files can compose a whole package ;
but a package identified by a URI is defined by one and only one
xsl:package file.

No. A package is a set of modules (which are XML documents, and therefore typically but not necessarily files).

One of these is the top-level module also known as the "package manifest" (though the term reflects an earlier design in which the manifest only held interface details). The other modules of the package are all modules reachable by xsl:include and xsl:import, transitively.

So the package doesn't include/import files. The package contains modules, and modules may import/include other modules, which thereby become part of the package.

Post by c***@oxiane.com
- The package file may contain xsl code, but exists mainly to define the
interface that can be use when this package is used. The code may be in
included / imported modules (is module the right term, here ?)

The package definitely contains XSLT code components (in the extreme case it might only contain interface definitions, but I think that would be unusual). But importantly it also defines the visibility of the code components outside the package.

Michael Kay
Saxonica

c***@oxiane.com

2017-05-10 09:03:05 UTC

Permalink

Post by Michael Kay
You need to distinguish compilation (which produces a data structure
in memory), export (which serializes that data structure to an XML
document typically in filestore), import (which reconstructs the data
structure from the export file), and bytecode generation (which
produces in-memory bytecode from the in-memory data structure).
The "compilation" process is essential. Export/Import is needed only
if you want to distribute a compiled package, or to speed up package
loading. Bytecode generation is optional.

Here are what I think correct ; I may be wrong, please correct me if
necessary.

When I want to distribute compiled modules, I have to :

Compile the source file

Post by Michael Kay
XsltExecutable xslExec = xslCompiler.compile(source);

Produce the serialized form of the compiled data

Post by Michael Kay
xslExec.export(outputStream);

Then, in my application,
Indicate if I want or not to produce bytecode

Post by Michael Kay
xslCompiler.setGenerateByteCode(boolean);

I have to import the serialized form

Post by Michael Kay
XsltExecutable xslExec = xslCompiler.compile(sefSource); // this

will just import serialized module, not compile it
Execute the module with XSLT2 API

Post by Michael Kay
for(Source source:sources) {
XsltTransformer tr = xslExec.load();
tr.setSource(source);
tr.setDestination(getDestination(source));
tr.transform();
}

or with XSLT3 API

Post by Michael Kay
Xslt30Tranformer tr = xslExec.load30();
for(Source source:sources) {
tr.applyTemplates(source, getDestination(source));
}

This means, as a Java developper, I must know for each module how I have
to run it. I need inputs from the XSL developper on how to use each
module.

But I still can transform documents with Xslt30 modules and 2.0 API ;
but I won't be able to call functions directly, or to start transform
with a specific mode.

I hope I'm not too far from truth...

Best regards,
Christophe

Michael Kay

2017-05-10 09:44:52 UTC

Permalink

Here are what I think correct ; I may be wrong, please correct me if necessary.
Compile the source file

Post by Michael Kay
XsltExecutable xslExec = xslCompiler.compile(source);

Produce the serialized form of the compiled data

Post by Michael Kay
xslExec.export(outputStream);

Let's try to avoid the term modules - a module is a part of a package and a module can't be separately compiled.

You can compile and export a single package using

XsltPackage pack = xsltCompiler.compilePackage(source);
pack.save(...)

Or if the stylesheet consists of a single package you can do what you suggested:

XsltExecutable exec = xsltCompiler.compile(source);
exec.export()

Or it the stylesheet consists of multiple packages you can compile them separately, link them, and export the assembled executable:

XsltPackage lib= xsltCompiler.compilePackage(source);
xsltCompiler.import(lib);
XsltPackage top = xsltCompiler.compilePackage(source);
xsltCompiler.import(top);
XsltExecutable exec = top.link();
exec.export(...)

Then, in my application,
Indicate if I want or not to produce bytecode

Post by Michael Kay
xslCompiler.setGenerateByteCode(boolean);

I have to import the serialized form

Post by Michael Kay
XsltExecutable xslExec = xslCompiler.compile(sefSource); // this will just import serialized module, not compile it

I think that will work, but most of our tests use

XsltExecutable exec = xsltCompiler.loadExecutablePackage(URI location)

Execute the module with XSLT2 API

Post by Michael Kay
for(Source source:sources) {
XsltTransformer tr = xslExec.load();
tr.setSource(source);
tr.setDestination(getDestination(source));
tr.transform();
}

or with XSLT3 API

Post by Michael Kay
Xslt30Tranformer tr = xslExec.load30();
for(Source source:sources) {
tr.applyTemplates(source, getDestination(source));
}

This means, as a Java developper, I must know for each module how I have to run it. I need inputs from the XSL developper on how to use each module.
But I still can transform documents with Xslt30 modules and 2.0 API ; but I won't be able to call functions directly, or to start transform with a specific mode.

Both XsltTransformer and Xslt30Transformer can happily execute stylesheets the old way by applying templates to the root of a source document, or by entry at a named template, and then wrapping the results as an XDM document tree, with optional serialization. Xslt30Transformer gives new capabilities, e.g. supplying parameters to the initial template, entry at a named function, separation of the global context item from the initial match selection (relevant mainly when streaming), or returning "raw" results such as a sequence of integers without wrapping as a result tree. There's no harm in always using Xslt30Transformer for everything - that's what the Transform command line does. By contrast, the JAXP glue layer uses XsltTransformer for everything, because JAXP never needs the new ways of doing things.

Michael Kay
Saxonica