Skip to content

Extracting config files from resources in jars on the classpath #217

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
muryoh opened this issue Mar 6, 2018 · 9 comments
Closed

Extracting config files from resources in jars on the classpath #217

muryoh opened this issue Mar 6, 2018 · 9 comments

Comments

@muryoh
Copy link

muryoh commented Mar 6, 2018

Hey, thanks for the plugin 👍

I like the idea, however I am unable to achieve what I would expect to be achievable.
Typically, my company uses a global import order configuration and code formatter configuration.

What we usually do is try to use opensource plugins over which we apply "standard" (as in company standard) configuration.

So typically, I would like to be able to create a plugin that applies and configures this very plugin. However, the configuration can only be done using File instances. But when you're in a JAR, you don't have a File

A possible workaround would have been to do something like
spotlessApply.doFirst {
...
}

And, there, create the files before the tasks run. However:

  • the tasks are only created in an afterEvaluate{} block which forces me to put that doFirst {} in another afterEvaluate {} block
  • tasks are not configured lazily: the files provided in the configuration are loaded as soon as the tasks are created

Do you have any pointers as to how you'd do that?

Cheers

@nedtwigg
Copy link
Member

nedtwigg commented Mar 6, 2018

the tasks are only created in an afterEvaluate{} block which forces me to put that doFirst {} in another afterEvaluate {} block

Yeah, ideally the tasks would be created immediately. I think we're likely to break lots of things if we change that now, though.

the files provided in the configuration are loaded as soon as the tasks are created

Everything that goes into a FormatterStep's state (e.g. config files) are loaded and parsed lazily. Usually, the first time they are loaded is when gradle is doing its up-to-date check to see if the task needs to rerun:

return FormatterStep.createLazy(NAME,
() -> new State(JarState.from(MAVEN_COORDINATE + version, provisioner), settingsFiles),
State::createFormat);

/**
* This function is guaranteed to be called at most once.
* If the state is never required, then it will never be called at all.
*
* Throws exception because it's likely that there will be some IO going on.
*/
protected abstract T calculateState() throws Exception;
/** Returns the underlying state, possibly triggering a call to {{@link #calculateState()}. */
protected final T state() {
// double-checked locking for lazy evaluation of calculateState
if (state == null) {
synchronized (this) {
if (state == null) {
try {
state = calculateState();
} catch (Exception e) {
throw ThrowingEx.asRuntime(e);
}
}
}
}
return state; // will always be nonnull at this point
}

any pointers as to how you'd do that?

spotlessCheck and spotlessApply are actually just "command" tasks - they don't do the work. e.g. for java, there are three tasks: spotlessJavaCheck, spotlessJavaApply, and spotlessJava. spotlessJava is the task that actually does the work, and the checks and applys just depend on it:

// create the check and apply control tasks
Task checkTask = project.getTasks().create(taskName + CHECK);
Task applyTask = project.getTasks().create(taskName + APPLY);
// the root tasks depend on them
rootCheckTask.dependsOn(checkTask);
rootApplyTask.dependsOn(applyTask);
// and they depend on the work task
checkTask.dependsOn(spotlessTask);
applyTask.dependsOn(spotlessTask);
// when the task graph is ready, we'll configure the spotlessTask appropriately
project.getGradle().getTaskGraph().whenReady(new Closure(null) {
private static final long serialVersionUID = 1L;
// called by gradle
@SuppressFBWarnings("UMAC_UNCALLABLE_METHOD_OF_ANONYMOUS_CLASS")
public Object doCall(TaskExecutionGraph graph) {
if (graph.hasTask(checkTask)) {
spotlessTask.setCheck();
}
if (graph.hasTask(applyTask)) {
spotlessTask.setApply();
}
return Closure.DONE;
}
});
});

If you do spotlessJava.doFirst(), that still won't work because the task will already have done an up-to-date check (thus loading the files) before your doFirst has run.

I would do this:

task extractConfigDeps {
    // extract resources to files
}
tasks.withType(com.diffplug.gradle.spotless.SpotlessTask) { 
  it.dependsOn(extractConfigDependencies)
}

We'd love a PR for handling resources, I think your usecase is great and we're not handling it very well at the moment.

@nedtwigg nedtwigg changed the title Specifying Eclipse Java configuration from a JAR resource Extracting config files from resources in jars on the classpath Mar 6, 2018
@muryoh
Copy link
Author

muryoh commented Mar 7, 2018

Hey, thanks for the detailed answer :-)

Everything that goes into a FormatterStep's state (e.g. config files) are loaded and parsed lazily. Usually, the first time they are loaded is when gradle is doing its up-to-date check to see if the task needs to rerun:

Hm, this does not seem to fit with the behavior I'm seeing. Using this sample gradle script:

plugins {
  id "com.diffplug.gradle.spotless" version "3.10.0"
}

apply plugin: 'java'

spotless {
    java {
        importOrderFile file('nonexisting')
    }
}

file('nonexisting').createNewFile()

I get the following stack trace which indicates the file is actually checked for existence right when I'm calling the extension in my script:

Caused by: java.io.UncheckedIOException: java.nio.file.NoSuchFileException: /tmp/spotless-example/nonexisting
        at com.diffplug.spotless.java.ImportOrderStep.getImportOrder(ImportOrderStep.java:76)
        at com.diffplug.spotless.java.ImportOrderStep.createFromFile(ImportOrderStep.java:58)
        at com.diffplug.gradle.spotless.JavaExtension.importOrderFile(JavaExtension.java:75)
        at org.gradle.internal.metaobject.BeanDynamicObject$MetaClassAdapter.invokeMethod(BeanDynamicObject.java:479)
        at org.gradle.internal.metaobject.BeanDynamicObject.tryInvokeMethod(BeanDynamicObject.java:191)
        at org.gradle.internal.metaobject.ConfigureDelegate.invokeMethod(ConfigureDelegate.java:57)
        at build_eo04ezjjnv70ww5krj9lo2mdh$_run_closure1$_closure2.doCall(/tmp/spotless-example/build.gradle:9)
        at org.gradle.api.internal.ClosureBackedAction.execute(ClosureBackedAction.java:71)
        at org.gradle.util.ConfigureUtil.configureTarget(ConfigureUtil.java:160)
        at org.gradle.util.ConfigureUtil.configure(ConfigureUtil.java:106)
        at org.gradle.util.ConfigureUtil$1.execute(ConfigureUtil.java:123)
        at com.diffplug.gradle.spotless.SpotlessExtension.configure(SpotlessExtension.java:156)
        at com.diffplug.gradle.spotless.SpotlessExtension.java(SpotlessExtension.java:79)
        at com.diffplug.gradle.spotless.SpotlessExtension_Decorated.java(Unknown Source)
        at org.gradle.internal.metaobject.BeanDynamicObject$MetaClassAdapter.invokeMethod(BeanDynamicObject.java:479)
        at org.gradle.internal.metaobject.BeanDynamicObject.tryInvokeMethod(BeanDynamicObject.java:191)
        at org.gradle.internal.metaobject.CompositeDynamicObject.tryInvokeMethod(CompositeDynamicObject.java:98)
        at org.gradle.internal.metaobject.MixInClosurePropertiesAsMethodsDynamicObject.tryInvokeMethod(MixInClosurePropertiesAsMethodsDynamicObject.java:30)
        at org.gradle.internal.metaobject.ConfigureDelegate.invokeMethod(ConfigureDelegate.java:57)
        at build_eo04ezjjnv70ww5krj9lo2mdh$_run_closure1.doCall(/tmp/spotless-example/build.gradle:8)
        at org.gradle.api.internal.ClosureBackedAction.execute(ClosureBackedAction.java:71)
        at org.gradle.util.ConfigureUtil.configureTarget(ConfigureUtil.java:160)
        at org.gradle.util.ConfigureUtil.configure(ConfigureUtil.java:106)
        at org.gradle.util.ConfigureUtil$1.execute(ConfigureUtil.java:123)
        at org.gradle.api.internal.plugins.ExtensionsStorage$ExtensionHolder.configure(ExtensionsStorage.java:183)
        at org.gradle.api.internal.plugins.ExtensionsStorage.configureExtension(ExtensionsStorage.java:67)
        at org.gradle.api.internal.plugins.DefaultConvention.configureExtension(DefaultConvention.java:399)
        at org.gradle.api.internal.plugins.DefaultConvention.access$500(DefaultConvention.java:45)
        at org.gradle.api.internal.plugins.DefaultConvention$ExtensionsDynamicObject.tryInvokeMethod(DefaultConvention.java:336)
        at org.gradle.internal.metaobject.CompositeDynamicObject.tryInvokeMethod(CompositeDynamicObject.java:98)
        at org.gradle.internal.metaobject.MixInClosurePropertiesAsMethodsDynamicObject.tryInvokeMethod(MixInClosurePropertiesAsMethodsDynamicObject.java:30)
        at org.gradle.groovy.scripts.BasicScript$ScriptDynamicObject.tryInvokeMethod(BasicScript.java:134)
        at org.gradle.internal.metaobject.AbstractDynamicObject.invokeMethod(AbstractDynamicObject.java:160)
        at org.gradle.groovy.scripts.BasicScript.invokeMethod(BasicScript.java:83)
        at build_eo04ezjjnv70ww5krj9lo2mdh.run(/tmp/spotless-example/build.gradle:7)
        at org.gradle.groovy.scripts.internal.DefaultScriptRunnerFactory$ScriptRunnerImpl.run(DefaultScriptRunnerFactory.java:90)
        ... 93 more
Caused by: java.nio.file.NoSuchFileException: /tmp/spotless-example/nonexisting
        at com.diffplug.spotless.java.ImportOrderStep.getImportOrder(ImportOrderStep.java:68)
        ... 128 more

@muryoh
Copy link
Author

muryoh commented Mar 7, 2018

Oh, my bad, I did not understand that what you referred to as a "FormatterStep" was for example
eclipse().configFile 'non-existent-file'

It does work as advertised for that 👍

@nedtwigg
Copy link
Member

Relevant: gradle/gradle#2760

@JamieMagee
Copy link

I came across this page that seems very relevant

@JamieMagee
Copy link

I managed to get this working (with zips, not jars, but what's the difference really 😁). Here's an example build.gradle for a project that contains your spotless config. In this case I have a file called eclipseformat.xml under a directory called spotless

plugins {
  id 'base'
  id 'maven-publish'
}

group = 'uk.co.jamiemagee'
version = '1.0'

task doZip(type: Zip) {
    from ('.') {
        include 'spotless/'
    }

    destinationDir = file(buildDir)
    baseName = 'spotless-config'
}

publishing {
    publications {
        spotlessconfig(MavenPublication) {
            artifact zipSpotlessConfig
            artifactId 'spotless-config'
        }
    }
}

Then to refer to the published zip artifact in another project's build.gradle file

plugins {
  id 'java'
  id 'com.diffplug.gradle.spotless' version '3.18.0'
}

repositories {
  mavenLocal()
}

configurations {
    spotlessConfig
}

dependencies {
    spotlessConfig 'uk.co.jamiemagee:spotless-config:+@zip'
}

spotless {
  java {
    eclipse().configFile resources.text.fromArchiveEntry(configurations.spotlessConfig, 'spotless/eclipseformat.xml').asFile()
  }
}

I hope this works for you @muryoh. @nedtwigg I'd like to write this up a bit better an put it in the docs if you think it'd be useful to more people?

@JLLeitschuh
Copy link
Member

@nedtwigg If this is a feature offered, you need to ensure that the config file can be supplied lazily so it isn't unpacked during the Gradle configuration phases. If you don't make it lazy, it will significantly slow down the configuration phase of the Gradle build.

@nedtwigg
Copy link
Member

nedtwigg commented Mar 9, 2019

It would be great to have this written up in the docs!

I'm guessing this is a place where lazy task configuration would really shine. We have a great abandoned PR which did all of the hard parts of this (#277). But the work mentioned in this issue is helpful regardless.

@nedtwigg
Copy link
Member

I think this is now solved well by #513. Feel free to discuss further if you think Spotless should implement something more, I'm happy to reopen.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants