Disclaimer: so I started this draft months ago, and apparently for months now, my main job has been taking all my time, so instead of keeping it as a draft, I thought I would just release this. Keep in mind it might not be perfect, but hopefully it can still help others. I’ve been using kind of the same process for months on production and our plugins work all fine.
Here is a long due article: how to compress/shrink and build production-ready plug-ins. The documentation is quite sparse about that, I found this forum post and of course there is the official Dojo documentation, but although it will help you build a product including what you need from Dojo, this doesn’t really explain how to build your plug-in so it does not include Dojo at all, since in our case it’s already part of ICN.
In this article, I will explain how to automate all the process using Gradle build, unlike the forum post I linked above which is providing an ant build to do this. Both are perfectly valid but as you can see in my open source plug-ins hosted in the IBM ECM GitHub, I’m fond of Gradle 🙂
Contents
- 0. Introduction
- 1. Create the package.json file
- 2. Configure the plug-in main Java class
- 3. Download The Dojo sources
- 4. Unzip Dojo
- 5. Download the ICN resources
- 6. Copy your plug-in resources
- 7. Include all classes in the main script file
- 8. Run the Dojo build tool
- 9. Compress the main script file
- 10. Enable the compress process when asked
0. Introduction
Before detailing step by step the process, I would like to clarify a few things important to understand what we are going. The shrinking process for Dojo is not a simple minify/compress process like we can have with other frameworks. Since the convention in Dojo is to have one JS file per class, you’ll probably end up with a lot of files, which without optimization will be loaded separately by your browser. We do have in my projects plug-ins with several hundreds of files, and for remote sites, it could take minutes to load…
Therefore, the Dojo build process does more than minifying/compressing your files, it will also merge all of them in layers, which will greatly decrease the load time. In order to do this, it will need all the used resources to understand them and include them in your final build, except if you include them, which we will do for what’s is already brought by ICN. And even going further, it will run your javascript to make sure it’s valid, which means we are using a JavaScript engine in the process. This being said, let’s begin.
1. Create the package.json file
Although not mandatory, it’s good practice to have a package.json file describing your dojo package.
{ "name": "myPluginDojo", "version":"@VERSION@", "directories": { "lib": "." }, "main":"main", "dependencies": { "dojo":"@DOJOVERSION@" }, "description": "" }
I like to use variable in my resources file so I don’t have to update the version manually when we release new versions. If you do so, you’ll have to filter your resources in your Gradle build:
// Do not filter binary files like images or that will corrupt them processResources { // You can do this in one statement in Gradle 3.1 or newer // filesMatching(['**/*.js', '**/*.json']) { // filter ReplaceTokens, tokens: [ // VERSION: "${version} - ${new Date().format('M/d/yyyy')}".toString(), // DOJOVERSION: project.dojoVersion, // ] // } def tokens = [ VERSION: "${version} - ${new Date().format('M/d/yyyy')}".toString(), DOJOVERSION: project.dojoVersion ] filesMatching('**/*.js') { filter ReplaceTokens, tokens: tokens } filesMatching('**/*.json') { filter ReplaceTokens, tokens: tokens }
2. Configure the plug-in main Java class
When you build/optimize your plug-in, you’ll need to reference the jgz files instead of the JS and style files in the plug-in main class, but keeping the normal js and css file as debug files so you can fall back to use unshrinked version when you want to by using the ?debug=true or ?logLevel=n as URL parameter.
I usually let the Gradle build to that so I can have an option to compress or not the plugin. That means the script and style files should stay the uncompressed if I’m not compressing the plug-in. To do this, first edit your plug-in Java main class with the following changes:
public String getVersion() { return "@VERSION@"; } public String getCopyright() { return "@COPYRIGHT@"; } public String getScript() { return "@MAINJSSCRIPT@"; } public String getDebugScript() { return "MyPlugin.js"; } public String getCSSFileName() { return "@MAINCSS@"; } public String getDebugCSSFileName() { return "MyPlugin.css"; }
And edit the gradle build to filter your sources and used the generated version.
// Automated version injection task generateSources(type: Copy) { from sourceSets.main.java into "$buildDir/generated-src" filter(ReplaceTokens, tokens: [ VERSION: "${version} - ${new Date().format('M/d/yyyy')}".toString(), DOJOVERSION: project.dojoVersion, COPYRIGHT: "Copyright 2017 IBM Corporation", MAINJSSCRIPT: project.hasProperty('compress') ? "${project.mainScriptName}.js.jgz".toString() : "${project.mainScriptName}.js".toString(), MAINCSS: project.hasProperty('compress') ? "${project.mainScriptName}.css.jgz".toString() : "${project.mainScriptName}.css".toString() ]) } compileJava.doFirst { source = "$buildDir/generated-src" } compileJava.dependsOn generateSources
This will create the correct class dependending if you want to compress (compress property) or not.
3. Download The Dojo sources
Ad said in the introduction, The build engine will need Dojo since we are using it in our plug-ins. So let’s download it automatically to make sure it will be available where ever you run your build.
task dlDojo << { def f = new File("${project.buildDir}/dojo.zip") if (!f.exists()) { new URL("${project.dojo_src_zip}").withInputStream{ i -> f.withOutputStream{ it << i }} } }
To make this more configurable, I’ve extracted a few things in variables.
project.ext { // Dojo version to build against dojoVersion = "1.10.4" // Url to download Dojo dojo_src_zip = "http://download.dojotoolkit.org/release-${dojoVersion}/dojo-release-${dojoVersion}-src.zip" }
4. Unzip Dojo
Now that we downloaded Dojo, we have to unzip it. We’ll make sure that this task depends on the dlDojo one to make sure it’s there. Since the src zip from the Dojo website is containing and unnecessary root folder, I’m truncating it at unzip time.
task unzipDojo(type: Copy, dependsOn: 'dlDojo') { from zipTree(file("${project.buildDir}/dojo.zip")) into "${buildDir}/shrink/dojo" eachFile { FileCopyDetails fcp -> // Truncating first folder fcp.relativePath = new RelativePath(!fcp.file.isDirectory(), fcp.relativePath.segments[1..-1] as String[]) } includeEmptyDirs = false }
5. Download the ICN resources
As for Dojo, since we are using ecm.* classes in our plug-ins, and possibly idx.*, gridx.*, pvd.* and pvr.*, we need to make sure they are available to the build engine. We could also use the dojo, dojox and dijit packages from ICN but I like to choose the version I want to build against. So you’ll have to zip the following folder from your ICN application in a zip and make it available to your build either as a local path or URL (I hosted the zip in our Nexus server which allow building from different build servers without caring about this file). You can find these resources in the ear files you’re deploying, or if you are using WebSphere, they are directly available in the following exploded location: /profiles//installedApps//navigator.ear/navigator.war where was_root is /opt/IBM/WebSphere/AppServer with a default CPIT installation, as well as profile_name is AppSrv02 and cell_name P8Node02Cell.
task unzipIcnJs { doLast { def localPath = project.icn_src if (project.icn_src.startsWith("http")) { localPath = "${buildDir}/navigator_js.zip" def f = new File(localPath) if (!f.exists()) { new URL(project.icn_src).withInputStream{ i -> f.withOutputStream{ it << i }} } } copy { from zipTree(localPath) into "${buildDir}/shrink/dojo" exclude 'ecm/nls/ecm_*.jgz', '**/FormSigningContainer.js', '*/DocumentFormContainer.js', '**/PropertiesControllerRuntime.js' } } }
And you will need this variable as well to add to your existing project.ext block of course:
project.ext { icn_src = "navigator_js.zip" // Or you can use a URL, the unzipIcnJs task can handle both // icn_src = "http://nexushost/nexus/service/local/repositories/myrepo/content/com/ibm/icn/js/resources/2.0.3-FP8/resources-2.0.3-FP8.zip" }
6. Copy your plug-in resources
Still for the same purpose, the build engine needs your plug-in classes, so let’s copy them in the shrink/dojo folder as well:
task copyPluginSources(type: Copy) { from "${project.webContentDir}/${project.dojoPackage}" into "${buildDir}/shrink/dojo/${project.dojoPackage}" }
7. Include all classes in the main script file
In order for the Dojo shrinker to include Dojo classes in your output layer, they need to be used somewhere. Usually in ICNm entry points are actions and features, which are not used directly in the main script file but then configured in the desktop. That means the Dojo build system won’t put them in your layer, which defeats the purpose of gathering your files in one layer. You could import your actions and features in your main script file, and tag your templates as declarative to make sure you follow classes in templates if you’re using implicit imports, but that doesn’t always work for me so I prefer to include everything at build time in the main script to make sure I gather everything in the layer. Here is a Gradle task to do that. This task includes all js files in your plug-in’s dojo package, except your profile file, in your main script if they are not already imported.
task includeAllJs(dependsOn: 'copyPluginSources') { doLast { def m = Pattern.compile("require\\(\\s*\\[\\s*(.*?)\\s*\\]", Pattern.DOTALL).matcher(file("${project.webContentDir}/${project.mainScriptName}.js").text) if (m.find()) { def alreadyImported = m.group(1).findAll(/([\w\/]+)/) def res = [], len = project.webContentDir.length() + 1 fileTree("${project.webContentDir}/${project.dojoPackage}").filter { it.isFile() && it.name.endsWith(".js") && !it.name.endsWith("profile.js") }.each { f -> def mod = f.path[len..-4].replace("\\", "/") if (!alreadyImported.contains(mod)) { res << "\"" + mod + "\"" } else { println "${mod} already imported in ${project.mainScriptName}.js, skipping it." } } file("${buildDir}/shrink/dojo/${project.dojoPackage}/${project.mainScriptName}.js").append( m.replaceAll("require([\n" + m.group(1) + (m.group(1).contains("/") ? ", " : "") + res.join(',') + "\n]"), "UTF-8"); } } }
8. Run the Dojo build tool
At this time you should have task doing the following to prepare everything for the Dojo Build tool:
- Download/unzip the Dojo sources
- Download/unzip the ICN JS sources
- Copy your plug-in Dojo package
- Automatically created all import in your main script
We can now run the Dojo build tool. You’ll need a profile file to tell Dojo what to do, what to exclude and what layer to build. You can create a file profiles/profile.js in your project with the following content:
var profile = (function(){ return { basePath: "../build/shrink/dojo", releaseDir: "../build/release", action: "release", layerOptimize: "comments", optimize: "comments", stripConsole: "warn", localeList: "en,fr", packages: [ { name: "dojo", location: "./dojo" }, { name: "dijit", location: "./dijit" }, { name: "dojox", location: "./dojox" }, { name: "gridx", location: "./gridx" }, { name: "idx", location: "./idx" }, { name: "ecm", location: "./ecm" }, { name: "pvr", location: "./pvr" }, { name: "pvd", location: "./pvd" }, { name: "myPluginDojo", location: "./myPluginDojo" } ], layers: { "myPluginDojo/MyPlugin": { include: [ "myPluginDojo/MyPlugin" ], exclude: [ "dojo/dojo", "dijit/dijit", "ecm/ecm" ] } } }; })();
The two important parts you will have to adapt to your plugin are:
- In the packages array, add an element for your plug-in as
{ name: "myPluginDojo", location: "./myPluginDojo" }
- In the layers array, make sure you created a layer named as your main script (in this example myPluginDojo/MyPlugin) and including your main script, which is enough since we imported everything in it, and exluce the dojo/dijit and ecm packes from it since they are already part of ICN and compress in one jgz file, so we don’t need to include them in our jgz file as well.
Now that you have this profiles/profile.js file, you can configure run the Dojo build and use it. The dojo build is a JS engine so we will use the Mozilla JS engine to run it, which is included in the Dojo sources.
task runShrinksafe(type:JavaExec, dependsOn: ['includeAllJs', 'unzipIcnJs', 'unzipDojo']) { main = 'org.mozilla.javascript.tools.shell.Main' maxHeapSize "2048m" // We have to ignore the errors since ecm/widget/eforms/DocumentFormContainer is generating a dependency error we can't avoid ignoreExitValue = true workingDir = "${buildDir}/shrink/dojo/util/buildscripts" classpath = files("${buildDir}/shrink/dojo/util/shrinksafe/js.jar", "${buildDir}/shrink/dojo/util/shrinksafe/shrinksafe.jar", "${buildDir}/shrink/dojo/util/closureCompiler/compiler.jar") args "${buildDir}/shrink/dojo/dojo/dojo.js" args "baseUrl=${buildDir}/shrink/dojo/dojo" args "load=build" args "--profile" args "${projectDir}/profiles/profile.js" }
9. Compress the main script file
The previous step creates a unique shrinked file with all your classes as /build/shrink/build/release//. We now want to compress that as one jgz file.
// Utility function to gzip a single file def gzip(String f, String dest) { file(f).withInputStream{ stream -> file(dest).withOutputStream{ out -> def gOut = new GZIPOutputStream(out) gOut << stream gOut.close(); }} } // Compress the main script result of the build, the css file and copy over the i18n files. task gzipJsCss(dependsOn: 'runShrinksafe') { doLast { gzip("${buildDir}/shrink/build/release/${project.dojoPackage}/${project.mainScriptName}.js", "${project.webContentDir}/${project.mainScriptName}.js.jgz") copy { from("${buildDir}/shrink/build/release/${project.dojoPackage}/nls"){ include { it.name ==~ /${project.mainScriptName}_\w+\.js/ } } into "${project.webContentDir}/${project.dojoPackage}/nls" } gzip("${project.webContentDir}/${project.mainScriptName}.css", "${project.webContentDir}/${project.mainScriptName}.css.jgz") } }
10. Enable the compress process when asked
At the beginning, we configured the filtering process to generate the plug-in main class according to the compress property, in order to be able to enable the dojo build process or not. We also need to make sure all the tasks we just created are called when enabling it. They are all chained, so we just need to call the last one when using the compress property:
if (project.hasProperty('compress')) { println "compress detected" jar.dependsOn gzipJsCss }
Which means you will just have to add -Pcompress to your Gradle call to activate the Dojo build system:
gradle build -Pcompress
Here is a full Gradle file as an example, fully documented to help you understand how everything works together. This is actually one of my open source project that can be found on the official IBM ECM GitHub. You can refer to the osurcres in GitHub to see the profile and package.json files
import org.apache.tools.ant.filters.ReplaceTokens import java.util.zip.GZIPOutputStream import java.util.regex.Pattern apply plugin: 'java' apply plugin: 'eclipse' version = '0.1.0' group = 'com.ibm.icn' project.ext { // Main package where the java class and dojo package are mainPackage = "com.ibm.icn.extensions.diffmerge" // name of the dojo package dojoPackage = "diffMergePluginDojo" // Name of the main Javascript and Css scripts (without extension) mainScriptName = "DiffMergePlugin" // Dojo version to build against (for shrinking purpose) dojoVersion = "1.10.4" // Url to download Dojo (for shrinking purpose) dojo_src_zip = "http://download.dojotoolkit.org/release-${dojoVersion}/dojo-release-${dojoVersion}-src.zip" // Path of the zip containing the ICN resources (ecm/gridx/idx/pcr/pvd, dojo/dojox/dijit are downloaded), this can also be an URL if you decide to host that in Nexus or somewhere else online icn_src = "navigator_js.zip" // No need to change that, this is just to avoid repeating it webContentDir = "${sourceSets.main.output.resourcesDir}/" + project.mainPackage.replaceAll("\\.", "/") + "/WebContent" } // Keeping Java 6 for a bit longer, until WAS 9 becomes predominant sourceCompatibility = 1.6 // To use local jar, this is not needed if you have your own Nexus server // to which you deployed the j2ee, navigator and Filenet jars repositories { flatDir { dirs 'lib' } } // Filtering sources to inject versions, copyright, and the correct main scrip/style files depending if we're using the Dojo build system or not task generateSources(type: Copy) { from sourceSets.main.java into "$buildDir/generated-src" filter(ReplaceTokens, tokens: [ VERSION: "${version} - ${new Date().format('M/d/yyyy')}".toString(), DOJOVERSION: project.dojoVersion, COPYRIGHT: "Copyright 2017 IBM Corporation", MAINJSSCRIPT: project.hasProperty('compress') ? "${project.mainScriptName}.js.jgz".toString() : "${project.mainScriptName}.js".toString(), MAINCSS: project.hasProperty('compress') ? "${project.mainScriptName}.css.jgz".toString() : "${project.mainScriptName}.css".toString() ]) } // Use the generated sources instead of the ones in the src/mina/java compileJava.doFirst { source = "$buildDir/generated-src" } // Make sure we generate before compiling compileJava.dependsOn generateSources // Filter the resources to inject versions, we filter (match) what files we want to filter because filtering binary files like images will corrupt them processResources { // You can use this in one statement in Gradle 3.1 or newer // filesMatching(['**/*.js', '**/*.js']) { // filter ReplaceTokens, tokens: [ // VERSION: "${version} - ${new Date().format('M/d/yyyy')}".toString(), // DOJOVERSION: project.dojoVersion, // ] // } // If not using Gradle 3.1 yet, filesMatching with variable argument number does not exist yet def tokens = [ VERSION: "${version} - ${new Date().format('M/d/yyyy')}".toString(), DOJOVERSION: project.dojoVersion ] filesMatching('**/*.js') { filter ReplaceTokens, tokens: tokens } filesMatching('**/*.json') { filter ReplaceTokens, tokens: tokens } } // Convenient task to gather classes and resources in a single folder // to be used in ICN as class file path while developing task copyResourcesToClasses(type: Copy, dependsOn: processResources) { from sourceSets.main.output.resourcesDir from sourceSets.main.output.classesDir into "$buildDir/all" } classes.dependsOn copyResourcesToClasses dependencies { // Use the following lines to use local jars, located in the lib folder compile name: 'jace' compile name: 'j2ee' compile name: 'navigator' // Use the following lines if you have the jars deployed in your Nexus repository // compile 'com.ibm.filenet:jace:5.2.1' // compile 'com.ibm.javax:j2ee:1.4.1' // compile 'com.ibm.icn:navigatorAPI-plugin:2.0.3' compile 'log4j:log4j:1.2.16' } // Include the Plug-in main Java class in the jar's Manifest jar { manifest { attributes 'Plugin-Class': "${project.mainPackage}.DiffMergePlugin" } } // Download the Dojo sources task dlDojo << { def f = new File("${project.buildDir}/dojo.zip") if (!f.exists()) { new URL("${project.dojo_src_zip}").withInputStream{ i -> f.withOutputStream{ it << i }} } } // Unzip the Dojo sources task unzipDojo(type: Copy, dependsOn: 'dlDojo') { from zipTree(file("${project.buildDir}/dojo.zip")) into "${buildDir}/shrink/dojo" eachFile { FileCopyDetails fcp -> // Truncating first folder fcp.relativePath = new RelativePath(!fcp.file.isDirectory(), fcp.relativePath.segments[1..-1] as String[]) } includeEmptyDirs = false } // Unzip the ICN JavaScript resources task unzipIcnJs(type: Copy) { from zipTree("${project.icn_src}") into "${buildDir}/shrink/dojo" exclude 'ecm/nls/ecm_*.jgz', '**/FormSigningContainer.js', '*/DocumentFormContainer.js', '**/PropertiesControllerRuntime.js' } // Copy our plug-in Dojo package task copyPluginSources(type: Copy) { from "${project.webContentDir}/${project.dojoPackage}" from "profiles/profile.js" into "${buildDir}/shrink/dojo/${project.dojoPackage}" } // Automatically include all our Dojo classes to the plug-in main script task includeAllJs(dependsOn: 'copyPluginSources') { doLast { def m = Pattern.compile("require\\(\\s*\\[\\s*(.*?)\\s*\\]", Pattern.DOTALL).matcher(file("${project.webContentDir}/${project.mainScriptName}.js").text) if (m.find()) { def alreadyImported = m.group(1).findAll(/([\w\/]+)/) def res = [], len = project.webContentDir.length() + 1 fileTree("${project.webContentDir}/${project.dojoPackage}").filter { it.isFile() && it.name.endsWith(".js") && !it.name.endsWith("profile.js") }.each { f -> def mod = f.path[len..-4].replace("\\", "/") if (!alreadyImported.contains(mod)) { res << "\"" + mod + "\"" } else { println "${mod} already imported in ${project.mainScriptName}.js, skipping it." } } file("${buildDir}/shrink/dojo/${project.dojoPackage}/${project.mainScriptName}.js").append( m.replaceAll("require([\n" + m.group(1) + (m.group(1).contains("/") ? ", " : "") + res.join(',') + "\n]"), "UTF-8"); } } } // Run the Dojo build system task runShrinksafe(type:JavaExec, dependsOn: ['includeAllJs', 'unzipIcnJs', 'unzipDojo']) { main = 'org.mozilla.javascript.tools.shell.Main' maxHeapSize "2048m" // We have to ignore the errors since ecm/widget/eforms/DocumentFormContainer is generating a dependency error we can't avoid ignoreExitValue = true workingDir = "${buildDir}/shrink/dojo/util/buildscripts" classpath = files("${buildDir}/shrink/dojo/util/shrinksafe/js.jar", "${buildDir}/shrink/dojo/util/shrinksafe/shrinksafe.jar", "${buildDir}/shrink/dojo/util/closureCompiler/compiler.jar") args "${buildDir}/shrink/dojo/dojo/dojo.js" args "baseUrl=${buildDir}/shrink/dojo/dojo" args "load=build" args "--profile" args "${projectDir}/profiles/profile.js" } // Utility function to gzip a single file def gzip(String f, String dest) { file(f).withInputStream{ stream -> file(dest).withOutputStream{ out -> def gOut = new GZIPOutputStream(out) gOut << stream gOut.close(); }} } // Gzip the main script and css file task gzipJsCss(dependsOn: 'runShrinksafe') { doLast { gzip("${buildDir}/shrink/build/release/${project.dojoPackage}/${project.mainScriptName}.js", "${project.webContentDir}/${project.mainScriptName}.js.jgz") //fileTree("${buildDir}/shrink/build/release/${project.dojoPackage}/nls").filter{ // it.isFile() && it.name ==~ /DiffMergePlugin_\w+\.js/ //}.files.each{ f -> // gzip(f.path, "${project.webContentDir}/${project.dojoPackage}/nls/${f.name}.jgz") //} copy { from("${buildDir}/shrink/build/release/${project.dojoPackage}/nls"){ include { it.name ==~ /${project.mainScriptName}_\w+\.js/ } } into "${project.webContentDir}/${project.dojoPackage}/nls" } gzip("${project.webContentDir}/${project.mainScriptName}.css", "${project.webContentDir}/${project.mainScriptName}.css.jgz") } } if (project.hasProperty('compress')) { println "compress detected" jar.dependsOn gzipJsCss } // Distribute a jar without version suffix assemble { doLast { copy { from "$buildDir/libs" into "$buildDir/dist" rename { String jarName -> jarName.replace("-$version", '') } } } }
Great article, your posts are always a great read and the first place I steer new developers when they are first getting started with icn customizations. Question – you mentioned your open source projects hosted on ibm ecm github, all i see are the ‘public’ ones from ibm contributors, none of your repos, is there a process / maintainer you would recommend I reach out to to be added to the organization?
Thanks again for all the great posts,
DP
Hi David, I was talking about this and this. The one from this article, which is a plugin giving inline editing with syntax highlighting and merging between versions features never made it there for the same reason this post took months to be published, main job… However, those 2 plugins are so small I didn’t compress them. I’ll try to go back to the validation process and have the DiffMerge plugin published as well. I would ask internally the ICN/ECM team who we are supposed to contact if you want to contribute. I’d say it’s me if it’s one of my plugin, but if you want to add new repo for new plugins I don’t know yet.
Guillaume