From b86568a6d0396e951e7dda724bb0cf38428fb063 Mon Sep 17 00:00:00 2001 From: Johan Maasing Date: Sun, 13 Apr 2025 08:56:47 +0200 Subject: [PATCH 01/20] Add readme section about generating --- README.md | 96 +++++++++++++++++++++++++- endpoints-templates/endpoints-list.ftl | 14 ++++ 2 files changed, 109 insertions(+), 1 deletion(-) create mode 100644 endpoints-templates/endpoints-list.ftl diff --git a/README.md b/README.md index c647419..2eb6a32 100644 --- a/README.md +++ b/README.md @@ -104,6 +104,7 @@ in the freemarker template as a Map of String-keys to String-values. `/some/endpoint <- SomeType(foo:String)` is an endpoint declaration. It declares one endpoint that have a request body data type called `SomeType` that has a field called `foo` of the type `String`. +### Data types The DSL uses Scala convention of writing data types after the field name separated by a colon. Of course the DSL parser does not know anything about java or scala types, as far as it is concerned these are 2 strings and the first one is just named field-name and the other string is named field-type. @@ -111,6 +112,8 @@ just named field-name and the other string is named field-type. `Embedded(foo:Bar)` is a `namedTypeDeclaration` which is parsed the same way as the request type above. But isn't tied to a specific endpoint. +### Automatically named data types + `/some/other/endpoint <- (bar:Seq[Embedded])` is another endpoint declaration. However this time the request body is not named in the DSL. But all datatypes must have a name so it will simply name it after the last path segment and tack on the string 'Request' at the end. So the AST till contain a datatype named `endpointRequest` with a field named @@ -122,9 +125,100 @@ decide to generate in the templates. The only 'semantic' validation the parser performs is to check that not two types have the same name. +### Reponse data types + +It is possible to have an optional response data type declared like so: + +`/some/other/endpoint <- (bar:Seq[Embedded]) -> ResponseType(foo: Bar)` + +The right pointing arrow `->` denotes a response type, it can be an anonymous data type in which case the parser till +name it from the last path segment and add 'Response' to the end of the data type name. + ### DSL config The only key in the config block the generator looks at is called `ending`, this will be used as the file ending for the resulting file of applying the freemarker template. -## Generating \ No newline at end of file +## Generating + +If the parser is successful it will hold the following data in the AST + +```java +public record DocumentNode( + Map config, + List typeDefinitions, + List endpoints) { +} +``` + +This will be passed to the freemarker engine as the 'root' data object, meaning you have access to the parts in your freemarker template like this: + +```injectedfreemarker +<#list typeDefinitions as type> + This is the datat type name: ${type.name?cap_first} with the first letter capitalized. + +``` + +That is, you can directly reference `typeDefinitions`, `endpoints` or `config`. + +### Config + +The config object is simply a String-map with the keys and values unfiltered from the input file. Here is an example +that writes the value for a config key called 'package'. + +`package ${config.package}` + +### Data types + +These are all the data types the parser have collected, either from explicit declarations, request payloads and response +bodies. + +```java +public record TypeNode(String name, List fields) { } +public record FieldNode(String name, String type) { } +``` + +Here is an example template that writes the data types as Scala case classes +```injectedfreemarker +object Protocol: +<#list typeDefinitions?sort as type> + case class ${type.name?cap_first}( + <#list type.fields as field> + ${field.name} : ${field.type}, + + ) + +``` + +### Endpoints + +The parser will collect the following data for endpoint declarations + +```java +public record EndpointNode( + PathsNode paths, + String inputType, + Optional outputType) {} + +public record PathsNode(List paths) {} +``` + +This is an example that will write out the endpoints with the path first, then the Input data type, then the optional +Output data type. + +```injectedfreemarker +<#list endpoints as endpoint> + <#list endpoint.paths.paths> + <#items as segment>/${segment} + Input: + ${endpoint.inputType?cap_first} + Output: + <#if endpoint.outputType.isPresent()> + ${endpoint.outputType.get()?cap_first} + <#else> + Not specified + + + + +``` \ No newline at end of file diff --git a/endpoints-templates/endpoints-list.ftl b/endpoints-templates/endpoints-list.ftl new file mode 100644 index 0000000..9a779fb --- /dev/null +++ b/endpoints-templates/endpoints-list.ftl @@ -0,0 +1,14 @@ +<#list endpoints as endpoint> + <#list endpoint.paths.paths> + <#items as segment>/${segment} + Input: + ${endpoint.inputType?cap_first} + Output: + <#if endpoint.outputType.isPresent()> + ${endpoint.outputType.get()?cap_first} + <#else> + Not specified + + + + \ No newline at end of file From 6f1b026dd57026966b9e48de0e31d1db6986042f Mon Sep 17 00:00:00 2001 From: Johan Maasing Date: Sun, 13 Apr 2025 14:11:03 +0200 Subject: [PATCH 02/20] Output files are named as the templates with the ending stripped. --- README.md | 37 +++++++------------ .../{Codecs.ftl => Codecs.scala.ftl} | 0 .../{Endpoints.ftl => Endpoints.scala.ftl} | 0 .../{Protocol.ftl => Protocol.scala.ftl} | 0 .../{endpoints-list.ftl => endpoints.txt.ftl} | 0 .../nu/zoom/dsl/freemarker/Generator.java | 27 ++++++++------ 6 files changed, 30 insertions(+), 34 deletions(-) rename endpoints-templates/{Codecs.ftl => Codecs.scala.ftl} (100%) rename endpoints-templates/{Endpoints.ftl => Endpoints.scala.ftl} (100%) rename endpoints-templates/{Protocol.ftl => Protocol.scala.ftl} (100%) rename endpoints-templates/{endpoints-list.ftl => endpoints.txt.ftl} (100%) diff --git a/README.md b/README.md index 2eb6a32..8cf8fcb 100644 --- a/README.md +++ b/README.md @@ -4,7 +4,6 @@ This is a converter tool that reads a DSL and generates output files. Endgen is Open Source Software using the [Apache Software License v2.0](http://www.apache.org/licenses/LICENSE-2.0) ## Motivation - The motivation behind this tool is that I wanted to generate boilerplate code for handling HTTP Endpoints (hence the endgen name). @@ -23,25 +22,24 @@ parser and a code generator using [freemarker](https://freemarker.apache.org). | Endgen | ------------------------------------------------- ----------- ||--------| |-----------| |------------|| -| endpoint | || Parser | | In-Memory | | Freemarker || --------------- -| file | --> || | --> | AST | --> | engine || --> | Output file | -\__________\ ||--------| |-----------| |------------|| \_____________\ - ------------------------------------------------- - +| endpoint | || Parser | | In-Memory | | Freemarker || ------------------ +| file | --> || | --> | AST | --> | engine || --> | Output file | +\__________\ ||--------| |-----------| |------------|| | mytemplate.xxx | + -------------------------------------------------- \________________\ ^ | - ---------------- - | Template.ftl | - \_______________\ + ---------------------- + | mytemplate.xxx.ftl | + \____________________\ ``` ## How to Run - You need a Java 24 runtime and java in the path. A very convenient way to install a java runtime is [SdkMan](https://sdkman.io). Unpack the archive, run the provided shellscript file. ### Usage ``` -Usage: EndpointsCLI [-hvV] [-o=] [-t=] +Usage: run.sh [-hvV] [-o=] [-t=] Generate source code from an endpoints specification file. The source endpoints DSL file. -h, --help Show this help message and exit. @@ -55,28 +53,28 @@ Generate source code from an endpoints specification file. ``` ## DSL example - In the simplest form the DSL looks like this ``` /some/endpoint <- SomeType(foo:String) ``` -This gets parsed into an [AST](https://en.wikipedia.org/wiki/Abstract_syntax_tree) which is this case holds a list of +This gets parsed into an [AST](https://en.wikipedia.org/wiki/Abstract_syntax_tree) which in this case holds a list of Path segments and a data strucutre representing the input/body type. ## Code generation example - When the parser is done reading the DSL it will look in a directory for [freemarker](https://freemarker.apache.org) templates. For each template it finds it sends in the AST. The resulting file (per template) is written to an output directory. +The templates must have the file ending `.ftl` - this ending is stripped when generating the output file. So a template +called `types.java.ftl` will generate a file called `types.java`. + The idea being that you can take these files and probably adapt them before checking them into your project. Endgen does not aim to be a roundtrip tool (i.e. reading the generated source, or being smart in updating them etc). It is also a very limited DSL, you can for example not express what type of HTTP Verb to use or declare response codes. There are no plans to extend the DSL to do that either. ## DSL - This is the ANTLR grammar for the root of the DSL ```antlrv4 @@ -89,7 +87,7 @@ Here is an example: ``` { package: se.rutdev.senash, - ending: .scala + mykey: myvalue } /some/endpoint <- SomeType(foo:String) @@ -98,7 +96,7 @@ Embedded(foo:Bar) /some/other/endpoint <- (bar:Seq[Embedded]) ``` -This consists of a config block with 2 items, the 'package' and the 'ending' deinfition. These are available to be used +This consists of a config block with 2 items, the 'package' and the 'mykey' definition. These are available to be used in the freemarker template as a Map of String-keys to String-values. `/some/endpoint <- SomeType(foo:String)` is an endpoint declaration. It declares one endpoint that have a request body @@ -113,7 +111,6 @@ just named field-name and the other string is named field-type. to a specific endpoint. ### Automatically named data types - `/some/other/endpoint <- (bar:Seq[Embedded])` is another endpoint declaration. However this time the request body is not named in the DSL. But all datatypes must have a name so it will simply name it after the last path segment and tack on the string 'Request' at the end. So the AST till contain a datatype named `endpointRequest` with a field named @@ -126,7 +123,6 @@ decide to generate in the templates. The only 'semantic' validation the parser performs is to check that not two types have the same name. ### Reponse data types - It is possible to have an optional response data type declared like so: `/some/other/endpoint <- (bar:Seq[Embedded]) -> ResponseType(foo: Bar)` @@ -135,12 +131,10 @@ The right pointing arrow `->` denotes a response type, it can be an anonymous da name it from the last path segment and add 'Response' to the end of the data type name. ### DSL config - The only key in the config block the generator looks at is called `ending`, this will be used as the file ending for the resulting file of applying the freemarker template. ## Generating - If the parser is successful it will hold the following data in the AST ```java @@ -162,14 +156,12 @@ This will be passed to the freemarker engine as the 'root' data object, meaning That is, you can directly reference `typeDefinitions`, `endpoints` or `config`. ### Config - The config object is simply a String-map with the keys and values unfiltered from the input file. Here is an example that writes the value for a config key called 'package'. `package ${config.package}` ### Data types - These are all the data types the parser have collected, either from explicit declarations, request payloads and response bodies. @@ -191,7 +183,6 @@ object Protocol: ``` ### Endpoints - The parser will collect the following data for endpoint declarations ```java diff --git a/endpoints-templates/Codecs.ftl b/endpoints-templates/Codecs.scala.ftl similarity index 100% rename from endpoints-templates/Codecs.ftl rename to endpoints-templates/Codecs.scala.ftl diff --git a/endpoints-templates/Endpoints.ftl b/endpoints-templates/Endpoints.scala.ftl similarity index 100% rename from endpoints-templates/Endpoints.ftl rename to endpoints-templates/Endpoints.scala.ftl diff --git a/endpoints-templates/Protocol.ftl b/endpoints-templates/Protocol.scala.ftl similarity index 100% rename from endpoints-templates/Protocol.ftl rename to endpoints-templates/Protocol.scala.ftl diff --git a/endpoints-templates/endpoints-list.ftl b/endpoints-templates/endpoints.txt.ftl similarity index 100% rename from endpoints-templates/endpoints-list.ftl rename to endpoints-templates/endpoints.txt.ftl diff --git a/parser/src/main/java/nu/zoom/dsl/freemarker/Generator.java b/parser/src/main/java/nu/zoom/dsl/freemarker/Generator.java index bb1a29a..dcfa86a 100644 --- a/parser/src/main/java/nu/zoom/dsl/freemarker/Generator.java +++ b/parser/src/main/java/nu/zoom/dsl/freemarker/Generator.java @@ -29,10 +29,12 @@ import java.util.Objects; import java.util.stream.Stream; public class Generator { - private final Path templatesDir ; + private final Path templatesDir; private final DocumentNode data; - private final Path outputDir ; + private final Path outputDir; private final Configuration cfg; + private final String TEMPLATE_EXTENSION = ".ftl"; + private final int TEMPLATE_EXTENSION_LENGTH = TEMPLATE_EXTENSION.length(); public Generator(Path templatesDir, DocumentNode data, Path outputDir) throws IOException { this.templatesDir = Objects.requireNonNull(templatesDir); @@ -49,23 +51,26 @@ public class Generator { public List generate() throws IOException, TemplateException { try (Stream files = Files.list(templatesDir)) { - List templates = files.filter(p -> p.toString().endsWith(".ftl")).toList() ; + List templates = files + .map(Path::getFileName) + .map(Path::toString) + .filter(p -> p.length() > TEMPLATE_EXTENSION_LENGTH && p.endsWith(TEMPLATE_EXTENSION) + ) + .toList(); ArrayList out = new ArrayList<>(); - for (Path template : templates) { - String configEnding = this.data.config().get("ending") ; - String ending = configEnding != null ? configEnding.trim() : ""; - Path outpath = outputDir.resolve(outputFilenameFromTemplate(template.getFileName(), ending)); - Template ftl = this.cfg.getTemplate(template.getFileName().toString()) ; + for (String template : templates) { + Path outpath = outputDir.resolve(outputFilenameFromTemplate(template)); + Template ftl = this.cfg.getTemplate(template); try (var outw = Files.newBufferedWriter(outpath, StandardCharsets.UTF_8)) { ftl.process(this.data, outw); out.add(outpath); } } - return out ; + return out; } } - private String outputFilenameFromTemplate(Path template, String ending) { - return template.getFileName().toString().replace(".ftl", ending); + private String outputFilenameFromTemplate(String template) { + return template.substring(0, template.length() - TEMPLATE_EXTENSION_LENGTH); } } From fa67bdf41bc4d59a62c8c2f5823bddd944a57c31 Mon Sep 17 00:00:00 2001 From: Johan Maasing Date: Sun, 13 Apr 2025 15:41:33 +0200 Subject: [PATCH 03/20] Add Jenkinsfile --- Jenkinsfile | 38 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+) create mode 100644 Jenkinsfile diff --git a/Jenkinsfile b/Jenkinsfile new file mode 100644 index 0000000..08ef6d2 --- /dev/null +++ b/Jenkinsfile @@ -0,0 +1,38 @@ +def version = '0.0.0-SNAPSHOT' +pipeline { + agent { + label 'x86' + } + tools { + maven 'Maven-3.9.9' + jdk 'Zulu-24' + } + + stages { + stage('Checkout') { + steps { + checkout scmGit( + branches: [[name: '*/master']], + userRemoteConfigs: [[ + credentialsId: 'forgejo-user-accesstoken', + url : 'https://vcs.zoom.nu/zoom/endgen.git' + ]] + ) + } + } + stage('Maven package & deploy') { + steps { + withCredentials([file(credentialsId: 'jenkins-settings.xml', variable: 'SETTINGS_XML')]) { + script { + version = sh(returnStdout: true, script: 'mvn --global-settings ${SETTINGS_XML} help:evaluate -Dexpression=project.version -q -DforceStdout') + currentBuild.description = "cluster-admin:$version" + } + echo "Building version ${version}" + sh """ + mvn --global-settings \${SETTINGS_XML} clean package deploy + """ + } + } + } + } +} \ No newline at end of file From a61eb8ac47c2a54d098c4e17e86813a220ed9242 Mon Sep 17 00:00:00 2001 From: Johan Maasing Date: Sun, 13 Apr 2025 15:55:38 +0200 Subject: [PATCH 04/20] Set correct branch name --- Jenkinsfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Jenkinsfile b/Jenkinsfile index 08ef6d2..75dcf6e 100644 --- a/Jenkinsfile +++ b/Jenkinsfile @@ -12,7 +12,7 @@ pipeline { stage('Checkout') { steps { checkout scmGit( - branches: [[name: '*/master']], + branches: [[name: '*/main']], userRemoteConfigs: [[ credentialsId: 'forgejo-user-accesstoken', url : 'https://vcs.zoom.nu/zoom/endgen.git' From 527e22c6d89ff820948250b2a0235e94d87774a7 Mon Sep 17 00:00:00 2001 From: Johan Maasing Date: Sun, 13 Apr 2025 15:58:14 +0200 Subject: [PATCH 05/20] [maven-release-plugin] prepare release endgen-1.0 --- endgen-dist/pom.xml | 6 ++---- parser/pom.xml | 6 ++---- pom.xml | 8 +++----- 3 files changed, 7 insertions(+), 13 deletions(-) diff --git a/endgen-dist/pom.xml b/endgen-dist/pom.xml index 9d2a59c..e670ada 100644 --- a/endgen-dist/pom.xml +++ b/endgen-dist/pom.xml @@ -1,13 +1,11 @@ - + 4.0.0 nu.zoom.dsl endgen - 1.0-SNAPSHOT + 1.0 endgen-dist diff --git a/parser/pom.xml b/parser/pom.xml index 5e312e1..0da3aae 100644 --- a/parser/pom.xml +++ b/parser/pom.xml @@ -14,15 +14,13 @@ // See the License for the specific language governing permissions and // limitations under the License. --> - + 4.0.0 nu.zoom.dsl endgen - 1.0-SNAPSHOT + 1.0 parser diff --git a/pom.xml b/pom.xml index 8625d2e..38f41b6 100644 --- a/pom.xml +++ b/pom.xml @@ -14,14 +14,12 @@ // See the License for the specific language governing permissions and // limitations under the License. --> - + 4.0.0 nu.zoom.dsl endgen - 1.0-SNAPSHOT + 1.0 pom @@ -61,7 +59,7 @@ scm:git:https://codeberg.org/darkstar/endgen.git scm:git:ssh://git@vcs.zoom.nu:1122/zoom/endgen.git - admin-parent-1.1 + endgen-1.0 From 8efef2de97e643572b2c9ca3c195cc3cf9ad833d Mon Sep 17 00:00:00 2001 From: Johan Maasing Date: Sun, 13 Apr 2025 15:58:16 +0200 Subject: [PATCH 06/20] [maven-release-plugin] prepare for next development iteration --- endgen-dist/pom.xml | 2 +- parser/pom.xml | 2 +- pom.xml | 4 ++-- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/endgen-dist/pom.xml b/endgen-dist/pom.xml index e670ada..cfe5ff3 100644 --- a/endgen-dist/pom.xml +++ b/endgen-dist/pom.xml @@ -5,7 +5,7 @@ nu.zoom.dsl endgen - 1.0 + 1.1-SNAPSHOT endgen-dist diff --git a/parser/pom.xml b/parser/pom.xml index 0da3aae..8dad406 100644 --- a/parser/pom.xml +++ b/parser/pom.xml @@ -20,7 +20,7 @@ nu.zoom.dsl endgen - 1.0 + 1.1-SNAPSHOT parser diff --git a/pom.xml b/pom.xml index 38f41b6..0ce154e 100644 --- a/pom.xml +++ b/pom.xml @@ -19,7 +19,7 @@ nu.zoom.dsl endgen - 1.0 + 1.1-SNAPSHOT pom @@ -59,7 +59,7 @@ scm:git:https://codeberg.org/darkstar/endgen.git scm:git:ssh://git@vcs.zoom.nu:1122/zoom/endgen.git - endgen-1.0 + admin-parent-1.1 From 98529dd3bd1b2f571bc403f63da40e8304677a2e Mon Sep 17 00:00:00 2001 From: Johan Maasing Date: Tue, 15 Apr 2025 14:17:08 +0200 Subject: [PATCH 07/20] Lower java requirement to 21 from 24 --- parser/pom.xml | 6 +++--- pom.xml | 6 +++--- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/parser/pom.xml b/parser/pom.xml index 8dad406..c2f56aa 100644 --- a/parser/pom.xml +++ b/parser/pom.xml @@ -50,8 +50,8 @@ maven-compiler-plugin 3.13.0 - 24 - 24 + ${maven.compiler.source} + ${maven.compiler.source} @@ -91,4 +91,4 @@ - \ No newline at end of file + diff --git a/pom.xml b/pom.xml index 0ce154e..b234fa1 100644 --- a/pom.xml +++ b/pom.xml @@ -23,8 +23,8 @@ pom - 24 - 24 + 21 + 21 UTF-8 @@ -67,4 +67,4 @@ endgen-dist - \ No newline at end of file + From 35ef968ed1b9b549bd9b3eb97e25567e33a23145 Mon Sep 17 00:00:00 2001 From: Johan Maasing Date: Tue, 15 Apr 2025 16:57:14 +0200 Subject: [PATCH 08/20] Add verbose output while running, fix arguments to run.sh --- README.md | 2 +- endgen-dist/src/main/resources/run.sh | 2 +- parser/pom.xml | 4 ++-- .../src/main/java/nu/zoom/dsl/cli/EndpointsCLI.java | 11 +++++++++++ test01.endpoints | 3 +-- 5 files changed, 16 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index 8cf8fcb..2293eef 100644 --- a/README.md +++ b/README.md @@ -33,7 +33,7 @@ parser and a code generator using [freemarker](https://freemarker.apache.org). \____________________\ ``` ## How to Run -You need a Java 24 runtime and java in the path. A very convenient way to install a java runtime is [SdkMan](https://sdkman.io). +You need a Java 21 (or later) runtime and java in the path. A very convenient way to install a java runtime is [SdkMan](https://sdkman.io). Unpack the archive, run the provided shellscript file. diff --git a/endgen-dist/src/main/resources/run.sh b/endgen-dist/src/main/resources/run.sh index 720ffe7..75894b3 100755 --- a/endgen-dist/src/main/resources/run.sh +++ b/endgen-dist/src/main/resources/run.sh @@ -1,4 +1,4 @@ #! /bin/sh SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd ) cd "${SCRIPT_DIR}" -java -jar parser-${artifact.baseVersion}.jar \ No newline at end of file +java -jar parser-${artifact.baseVersion}.jar "$@" \ No newline at end of file diff --git a/parser/pom.xml b/parser/pom.xml index c2f56aa..02c7c9c 100644 --- a/parser/pom.xml +++ b/parser/pom.xml @@ -51,7 +51,7 @@ 3.13.0 ${maven.compiler.source} - ${maven.compiler.source} + ${maven.compiler.target} @@ -91,4 +91,4 @@ - + \ No newline at end of file diff --git a/parser/src/main/java/nu/zoom/dsl/cli/EndpointsCLI.java b/parser/src/main/java/nu/zoom/dsl/cli/EndpointsCLI.java index 57bf2bc..b26cb9c 100644 --- a/parser/src/main/java/nu/zoom/dsl/cli/EndpointsCLI.java +++ b/parser/src/main/java/nu/zoom/dsl/cli/EndpointsCLI.java @@ -61,11 +61,16 @@ public class EndpointsCLI implements Callable { validateTemplateDirectory(); validateInputFile(); validateOutputDirectory(); + verbose("Parsing " + file.toAbsolutePath()); DocumentNode rootNode = ParserWrapper.parse(file); + verbose("AST: " + rootNode); + verbose("Generating from templates in: " + templateDir.toAbsolutePath()); Generator generator = new Generator(templateDir, rootNode, outputDir); List generatedPaths = generator.generate(); if (generatedPaths.isEmpty()) { System.out.println("No generated paths found."); + } else { + generatedPaths.forEach(p -> verbose("Generated: " + p.toAbsolutePath())); } return 0; } catch (Exception e) { @@ -74,6 +79,12 @@ public class EndpointsCLI implements Callable { } } + private void verbose(String message) { + if (this.verbose) { + System.out.println(message); + } + } + private void validateOutputDirectory() throws IOException { if (Files.notExists(this.outputDir)) { Files.createDirectories(this.outputDir); diff --git a/test01.endpoints b/test01.endpoints index e2db8ab..6f212ed 100644 --- a/test01.endpoints +++ b/test01.endpoints @@ -17,8 +17,7 @@ { some: configvalue, someother: value, - package: se.rutdev.senash, - ending: .scala + package: se.rutdev.senash } /some/endpoint <- SomeType(foo:String) From f5ff4e06bcfa5c55ff92d259bc9a5d7d843e5452 Mon Sep 17 00:00:00 2001 From: Johan Maasing Date: Sun, 20 Apr 2025 11:46:55 +0200 Subject: [PATCH 09/20] Add states parser --- .gitignore | 4 +- README.md | 106 ++++++++++++-- parser/pom.xml | 1 + parser/src/main/antlr4/imports/Common.g4 | 41 ++++++ .../antlr4/nu/zoom/dsl/parser/Endpoints.g4 | 32 +---- .../main/antlr4/nu/zoom/dsl/parser/States.g4 | 22 +++ .../java/nu/zoom/dsl/ast/DocumentNode.java | 6 +- .../dsl/ast/EndpointsVisitorTransformer.java | 9 +- .../java/nu/zoom/dsl/ast/ParserWrapper.java | 28 +++- .../main/java/nu/zoom/dsl/ast/StateNode.java | 6 + .../dsl/ast/StatesVisitorTransformer.java | 132 ++++++++++++++++++ .../java/nu/zoom/dsl/ast/TransitionNode.java | 4 + .../java/nu/zoom/dsl/cli/EndpointsCLI.java | 31 +++- states-templates/Codecs.scala.ftl | 19 +++ states-templates/Types.scala.ftl | 23 +++ states-templates/nodes.md.ftl | 29 ++++ test01.states | 21 +++ 17 files changed, 454 insertions(+), 60 deletions(-) create mode 100644 parser/src/main/antlr4/imports/Common.g4 create mode 100644 parser/src/main/antlr4/nu/zoom/dsl/parser/States.g4 create mode 100644 parser/src/main/java/nu/zoom/dsl/ast/StateNode.java create mode 100644 parser/src/main/java/nu/zoom/dsl/ast/StatesVisitorTransformer.java create mode 100644 parser/src/main/java/nu/zoom/dsl/ast/TransitionNode.java create mode 100644 states-templates/Codecs.scala.ftl create mode 100644 states-templates/Types.scala.ftl create mode 100644 states-templates/nodes.md.ftl create mode 100644 test01.states diff --git a/.gitignore b/.gitignore index 48010bb..7acc024 100644 --- a/.gitignore +++ b/.gitignore @@ -32,4 +32,6 @@ build/ .vscode/ ### Mac OS ### -.DS_Store \ No newline at end of file +.DS_Store + +/endpoints-output/** \ No newline at end of file diff --git a/README.md b/README.md index 2293eef..2e73f92 100644 --- a/README.md +++ b/README.md @@ -32,6 +32,17 @@ parser and a code generator using [freemarker](https://freemarker.apache.org). | mytemplate.xxx.ftl | \____________________\ ``` + +Endgen currently contains two separate parsers: +* endpoint - A DSL for expressing HTTP endpoints. +* state - A DSL for expressing state and transitions. + +Which parser that is used to read the input file is determined by the file name ending +'.endpoints' or '.states' or by a command line argument. + +The endpoint DSL and the state DSL share the grammar for expressing configuration and data types +,see below for details. + ## How to Run You need a Java 21 (or later) runtime and java in the path. A very convenient way to install a java runtime is [SdkMan](https://sdkman.io). @@ -39,20 +50,23 @@ Unpack the archive, run the provided shellscript file. ### Usage ``` -Usage: run.sh [-hvV] [-o=] [-t=] +sage: run.sh [-hvV] [-o=] [-p=] [-t=] Generate source code from an endpoints specification file. The source endpoints DSL file. -h, --help Show this help message and exit. -o, --output= The directory to write the generated code to. - Default is ~/endpoints-output + Default is endpoints-output + -p, --parser= Force use of a specific parser instead of + determining from filename. Valid values: + Endpoints, States. -t, --template= The template directory. Default is - ~/endpoints-templates + endpoints-template -v, --verbose Print verbose debug messages. -V, --version Print version information and exit. ``` -## DSL example +## Endpoint DSL example In the simplest form the DSL looks like this ``` /some/endpoint <- SomeType(foo:String) @@ -80,6 +94,8 @@ This is the ANTLR grammar for the root of the DSL ```antlrv4 document : generatorconfig? (namedTypeDeclaration|endpoint)* ; ``` + +### Configuration block Meaning that the DSL file has an optional `generatorconfig` block at the top. Then you can write either; a type definition, or an endpoint declaration, as many times as you like. @@ -99,7 +115,9 @@ Embedded(foo:Bar) This consists of a config block with 2 items, the 'package' and the 'mykey' definition. These are available to be used in the freemarker template as a Map of String-keys to String-values. -`/some/endpoint <- SomeType(foo:String)` is an endpoint declaration. It declares one endpoint that have a request body +### Endpoint definition + +`/some/endpoint <- SomeType(foo:String)` is an endpoint definition. It declares one endpoint that have a request body data type called `SomeType` that has a field called `foo` of the type `String`. ### Data types @@ -110,7 +128,7 @@ just named field-name and the other string is named field-type. `Embedded(foo:Bar)` is a `namedTypeDeclaration` which is parsed the same way as the request type above. But isn't tied to a specific endpoint. -### Automatically named data types +### Automatically named endpoint data types `/some/other/endpoint <- (bar:Seq[Embedded])` is another endpoint declaration. However this time the request body is not named in the DSL. But all datatypes must have a name so it will simply name it after the last path segment and tack on the string 'Request' at the end. So the AST till contain a datatype named `endpointRequest` with a field named @@ -122,7 +140,7 @@ decide to generate in the templates. The only 'semantic' validation the parser performs is to check that not two types have the same name. -### Reponse data types +### Endpoint reponse data type It is possible to have an optional response data type declared like so: `/some/other/endpoint <- (bar:Seq[Embedded]) -> ResponseType(foo: Bar)` @@ -130,9 +148,45 @@ It is possible to have an optional response data type declared like so: The right pointing arrow `->` denotes a response type, it can be an anonymous data type in which case the parser till name it from the last path segment and add 'Response' to the end of the data type name. -### DSL config -The only key in the config block the generator looks at is called `ending`, this will be used as the file ending for -the resulting file of applying the freemarker template. +### State grammar + +This is an example of a state file: +``` +start -> middle: message, +middle -> middle: selfmessage, +middle -> end: endmessage +``` + +It contains 3 state definitions `start`, `middle` and `end`. A state definition will be parsed as a data type with +the name of the state as the type name. + +It also contains 3 message definitions `message`, `selfmessage` and `endmessage`. Message definitions will also be +parsed as data types. + +Since the parser will extract datatypes it is possible to define the fields of the data types. This is a slightly more +complicated example: + +``` +start(foo:Foo) -> middle: message(a: String), +middle(bar:Bar) -> middle: selfmessage, +middle -> end: endmessage +``` +Where for example the data type for `middle` will have the field declaration with the name `bar` and the type `Bar`. + +Fields for the same state data type, or message data type, will be merged. Here is a complex example: + +``` +start(s:S) -> middle(foo:foo): message(foo:foo), +middle -> middle(bar:bar): selfmessage(bar:bar), +middle -> end: message(bar:baz) +``` + +Not that we can declare fields on both the `from` and `to` state declarations. The `middle` datat type will have field +definitons for `foo` and `bar`. + +The data type for `message` will have fields for `foo` and `bar`. + +One restriction is that states and messages may not have the same name, i.e. be parsed as the same data type. ## Generating If the parser is successful it will hold the following data in the AST @@ -140,12 +194,17 @@ If the parser is successful it will hold the following data in the AST ```java public record DocumentNode( Map config, - List typeDefinitions, - List endpoints) { + Set typeDefinitions, + List endpoints, + Set states) { } ``` -This will be passed to the freemarker engine as the 'root' data object, meaning you have access to the parts in your freemarker template like this: +Depending on the parser used the endpoints or the states will be null but config and typeDefinitions are populated the +same for both parsers. + +This will be passed to the freemarker engine as the 'root' data object, meaning you have access to the parts in your +freemarker template like this: ```injectedfreemarker <#list typeDefinitions as type> @@ -153,7 +212,7 @@ This will be passed to the freemarker engine as the 'root' data object, meaning ``` -That is, you can directly reference `typeDefinitions`, `endpoints` or `config`. +That is, you can directly reference `typeDefinitions`, `endpoints`, `states` or `config`. ### Config The config object is simply a String-map with the keys and values unfiltered from the input file. Here is an example @@ -212,4 +271,21 @@ Output data type. -``` \ No newline at end of file +``` + +### States + +The set of states will hold items of this shape: + +```injectedfreemarker +public record StateNode(String name, String data, Set transitions) { +} +``` + +and the transitions has this structure: + +```injectedfreemarker +public record TransitionNode(String message, String toState) { +} +``` + diff --git a/parser/pom.xml b/parser/pom.xml index 02c7c9c..a81607b 100644 --- a/parser/pom.xml +++ b/parser/pom.xml @@ -57,6 +57,7 @@ org.apache.maven.plugins maven-jar-plugin + 3.4.2 diff --git a/parser/src/main/antlr4/imports/Common.g4 b/parser/src/main/antlr4/imports/Common.g4 new file mode 100644 index 0000000..8844343 --- /dev/null +++ b/parser/src/main/antlr4/imports/Common.g4 @@ -0,0 +1,41 @@ +// Copyright 2025 "Johan Maasing" +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +grammar Common; +generatorconfig : '{' (configitem)? (',' configitem)* '}'; +configitem : configkey ':' configvalue ; +configkey : IDENTIFIER ; +configvalue : (IDENTIFIER|VALUE) ; +namedTypeDeclaration : typeName typeDeclaration ; +typeName : IDENTIFIER ; +typeDeclaration : '(' typeField (',' typeField)* ')' ; +typeField : fieldName COLON fieldType ; +fieldName : IDENTIFIER ; +fieldType : IDENTIFIER ; + +fragment LOWERCASE : [a-z] ; +fragment UPPERCASE : [A-Z] ; +fragment GENERICS : '['|']'|'<'|'>' ; +fragment DOT : '.' ; +fragment COMMENT_BEGIN : '/*' ; +fragment COMMENT_END : '*/' ; +fragment DIGIT : [0-9] ; + +WS : [ \t\n\r]+ -> skip; +COMMENT : COMMENT_BEGIN .*? COMMENT_END -> skip; +LEFT_ARROW : '<-' ; +RIGHT_ARROW : '->' ; +IDENTIFIER : (LOWERCASE | UPPERCASE) (LOWERCASE | UPPERCASE | DIGIT | GENERICS | DOT)* ; +VALUE : ~[ ,{}:()/="#';*\n\r\t]+ ; +SLASH : '/' ; +COLON : ':' ; \ No newline at end of file diff --git a/parser/src/main/antlr4/nu/zoom/dsl/parser/Endpoints.g4 b/parser/src/main/antlr4/nu/zoom/dsl/parser/Endpoints.g4 index 4f47bf8..1a74607 100644 --- a/parser/src/main/antlr4/nu/zoom/dsl/parser/Endpoints.g4 +++ b/parser/src/main/antlr4/nu/zoom/dsl/parser/Endpoints.g4 @@ -12,35 +12,11 @@ // See the License for the specific language governing permissions and // limitations under the License. grammar Endpoints; +import Common; + document : generatorconfig? (namedTypeDeclaration|endpoint)* ; -generatorconfig : '{' (configitem)? (',' configitem)* '}'; -configitem : configkey ':' configvalue ; -configkey : IDENTIFIER ; -configvalue : (IDENTIFIER|VALUE) ; -namedTypeDeclaration : typeName typeDeclaration ; -typeName : IDENTIFIER ; -typeDeclaration : '(' typeField (',' typeField)* ')' ; -typeField : fieldName ':' fieldType ; -fieldName : IDENTIFIER ; -fieldType : IDENTIFIER ; -requestBody : REQUEST_PREFIX (namedTypeDeclaration | typeDeclaration | IDENTIFIER) ; -responseBody : RESPONSE_PREFIX (namedTypeDeclaration | typeDeclaration | IDENTIFIER) ; +requestBody : LEFT_ARROW (namedTypeDeclaration | typeDeclaration | IDENTIFIER) ; +responseBody : RIGHT_ARROW (namedTypeDeclaration | typeDeclaration | IDENTIFIER) ; endpoint : path requestBody responseBody?; path : (pathSegment)+ ; pathSegment : SLASH (IDENTIFIER|VALUE) ; - - -fragment DIGIT : [0-9] ; -fragment LOWERCASE : [a-z] ; -fragment UPPERCASE : [A-Z] ; -fragment GENERICS : '['|']'|'<'|'>' ; -fragment DOT : '.' ; -fragment COMMENT_BEGIN : '/*' ; -fragment COMMENT_END : '*/' ; -WS : [ \t\n\r]+ -> skip; -COMMENT : COMMENT_BEGIN .*? COMMENT_END -> skip; -REQUEST_PREFIX : '<-' ; -RESPONSE_PREFIX : '->' ; -SLASH : '/' ; -IDENTIFIER : (LOWERCASE | UPPERCASE) (LOWERCASE | UPPERCASE | DIGIT | GENERICS | DOT)* ; -VALUE : ~[ ,{}:()/="#';*\n\r\t]+ ; diff --git a/parser/src/main/antlr4/nu/zoom/dsl/parser/States.g4 b/parser/src/main/antlr4/nu/zoom/dsl/parser/States.g4 new file mode 100644 index 0000000..dbfe628 --- /dev/null +++ b/parser/src/main/antlr4/nu/zoom/dsl/parser/States.g4 @@ -0,0 +1,22 @@ +// Copyright 2025 "Johan Maasing" +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +grammar States; +import Common; + +document : generatorconfig? transition (',' transition)* ; +transition : from RIGHT_ARROW to COLON message ; +from : state ; +to : state ; +message : typeName typeDeclaration? ; +state : typeName typeDeclaration? ; \ No newline at end of file diff --git a/parser/src/main/java/nu/zoom/dsl/ast/DocumentNode.java b/parser/src/main/java/nu/zoom/dsl/ast/DocumentNode.java index 24ac244..fcb8621 100644 --- a/parser/src/main/java/nu/zoom/dsl/ast/DocumentNode.java +++ b/parser/src/main/java/nu/zoom/dsl/ast/DocumentNode.java @@ -15,9 +15,11 @@ package nu.zoom.dsl.ast; import java.util.List; import java.util.Map; +import java.util.Set; public record DocumentNode( Map config, - List typeDefinitions, - List endpoints) { + Set typeDefinitions, + List endpoints, + Set states) { } diff --git a/parser/src/main/java/nu/zoom/dsl/ast/EndpointsVisitorTransformer.java b/parser/src/main/java/nu/zoom/dsl/ast/EndpointsVisitorTransformer.java index 5d846d6..e530a80 100644 --- a/parser/src/main/java/nu/zoom/dsl/ast/EndpointsVisitorTransformer.java +++ b/parser/src/main/java/nu/zoom/dsl/ast/EndpointsVisitorTransformer.java @@ -19,7 +19,8 @@ import org.antlr.v4.runtime.tree.TerminalNode; import java.util.*; -public class EndpointsVisitorTransformer extends EndpointsBaseVisitor { +public class EndpointsVisitorTransformer + extends EndpointsBaseVisitor { private final ArrayList endpoints = new ArrayList<>(); private final HashMap config = new HashMap<>(); private final HashSet dataTypes = new HashSet<>(); @@ -35,8 +36,8 @@ public class EndpointsVisitorTransformer extends EndpointsBaseVisitor getDataTypes() { - return List.copyOf(dataTypes); + public Set getDataTypes() { + return Set.copyOf(dataTypes); } @Override @@ -130,7 +131,7 @@ public class EndpointsVisitorTransformer extends EndpointsBaseVisitor transitions) { +} diff --git a/parser/src/main/java/nu/zoom/dsl/ast/StatesVisitorTransformer.java b/parser/src/main/java/nu/zoom/dsl/ast/StatesVisitorTransformer.java new file mode 100644 index 0000000..cd4bef0 --- /dev/null +++ b/parser/src/main/java/nu/zoom/dsl/ast/StatesVisitorTransformer.java @@ -0,0 +1,132 @@ +package nu.zoom.dsl.ast; + +import nu.zoom.dsl.parser.StatesBaseVisitor; +import nu.zoom.dsl.parser.StatesParser; +import org.antlr.v4.runtime.tree.TerminalNode; + +import java.util.*; +import java.util.stream.Stream; + +public class StatesVisitorTransformer extends StatesBaseVisitor { + private final HashMap config = new HashMap<>(); + private final HashSet nodeTypes = new HashSet<>(); + private final HashSet messageTypes = new HashSet<>(); + // from -> + private final HashMap> transitions = new HashMap<>(); + + @Override + public StatesParser.DocumentContext visitTransition(StatesParser.TransitionContext ctx) { + String from = ctx.from().state().typeName().IDENTIFIER().getText() ; + String to = ctx.to().state().typeName().IDENTIFIER().getText() ; + String message = ctx.message().typeName().IDENTIFIER().getText() ; + this.transitions.computeIfAbsent(from, k -> new HashMap<>()).put(to, message); + return super.visitTransition(ctx); + } + + @Override + public StatesParser.DocumentContext visitState(StatesParser.StateContext ctx) { + String stateName = ctx.typeName().IDENTIFIER().getText() ; + List fields = extractFields(ctx.typeDeclaration()) ; + this.nodeTypes.add(new TypeNode(stateName, fields)); + return super.visitState(ctx); + } + + @Override + public StatesParser.DocumentContext visitMessage(StatesParser.MessageContext ctx) { + String messageName = ctx.typeName().IDENTIFIER().getText() ; + List fields = extractFields(ctx.typeDeclaration()) ; + this.messageTypes.add(new TypeNode(messageName, fields)); + return super.visitMessage(ctx); + } + + @Override + public StatesParser.DocumentContext visitConfigitem(StatesParser.ConfigitemContext ctx) { + String configKey = ctx.configkey().IDENTIFIER().getText(); + String configValue = getText(ctx.configvalue().IDENTIFIER(), ctx.configvalue().VALUE()); + this.config.put(configKey, configValue); + return super.visitConfigitem(ctx); + } + + public Set getStates() { + HashSet states = new HashSet<>(); + this.transitions.forEach((state,v)->{ + HashSet transitionNodes = new HashSet<>(); + v.forEach((to, message) -> transitionNodes.add(new TransitionNode(message, to))); + states.add(new StateNode(state, "", transitionNodes)) ; + }) ; + return states ; + } + + public Map getConfig() { + return Map.copyOf(config); + } + + public Set getTypes() { + final HashMap stateTypeNodes = new HashMap<>(); + this.nodeTypes.forEach(typeNode -> { + if (stateTypeNodes.containsKey(typeNode.name())) { + TypeNode mergedNode = mergeTypeFields(typeNode, stateTypeNodes.get(typeNode.name())); + stateTypeNodes.put(typeNode.name(), mergedNode); + } else { + stateTypeNodes.put(typeNode.name(), typeNode); + } + }) ; + final HashMap messageTypeNodes = new HashMap<>(); + this.messageTypes.forEach(typeNode -> { + if (stateTypeNodes.containsKey(typeNode.name())) { + throw new ParseException("Message " + typeNode.name() + " conflicts with state with the same type name"); + } + if (messageTypeNodes.containsKey(typeNode.name())) { + TypeNode mergedNode = mergeTypeFields(typeNode, messageTypeNodes.get(typeNode.name())); + messageTypeNodes.put(typeNode.name(), mergedNode); + } else { + messageTypeNodes.put(typeNode.name(), typeNode); + } + }) ; + HashSet allTypeNodes = new HashSet<>(stateTypeNodes.values()); + allTypeNodes.addAll(messageTypeNodes.values()); + return allTypeNodes ; + } + + private TypeNode mergeTypeFields(TypeNode t1, TypeNode t2) { + List t1Fields = (t1 != null) ? t1.fields() : List.of() ; + List t2Fields = (t2 != null) ? t2.fields() : List.of() ; + HashMap mergedFields = new HashMap<>(); + t1Fields.forEach(field -> { + if (mergedFields.containsKey(field.name())) { + throw new ParseException("Duplicate field name: " + field.name()); + } + mergedFields.put(field.name(), field); + }); + t2Fields.forEach(field -> { + if (mergedFields.containsKey(field.name())) { + throw new ParseException("Duplicate field name: " + field.name()); + } + mergedFields.put(field.name(), field); + }); + return new TypeNode(t1.name(), mergedFields.values().stream().toList()) ; + } + + private List extractFields(StatesParser.TypeDeclarationContext declaration) { + if (declaration == null) { + return Collections.emptyList(); + } + return declaration + .typeField() + .stream() + .map( + ctx -> + new FieldNode(ctx.fieldName().getText(), ctx.fieldType().getText()) + ) + .toList(); + } + + // Concatenate the text from two terminal nodes. Useful for contexts that are either an identifier or a value, + // and you just want the text from whichever is not null. + private String getText(TerminalNode identifier, TerminalNode value) { + return + ((identifier != null) ? identifier.getText() : "") + + ((value != null) ? value.getText() : ""); + } + +} diff --git a/parser/src/main/java/nu/zoom/dsl/ast/TransitionNode.java b/parser/src/main/java/nu/zoom/dsl/ast/TransitionNode.java new file mode 100644 index 0000000..90ebd3e --- /dev/null +++ b/parser/src/main/java/nu/zoom/dsl/ast/TransitionNode.java @@ -0,0 +1,4 @@ +package nu.zoom.dsl.ast; + +public record TransitionNode(String message, String toState) { +} diff --git a/parser/src/main/java/nu/zoom/dsl/cli/EndpointsCLI.java b/parser/src/main/java/nu/zoom/dsl/cli/EndpointsCLI.java index b26cb9c..e3d5794 100644 --- a/parser/src/main/java/nu/zoom/dsl/cli/EndpointsCLI.java +++ b/parser/src/main/java/nu/zoom/dsl/cli/EndpointsCLI.java @@ -34,22 +34,29 @@ import java.util.concurrent.Callable; description = "Generate source code from an endpoints specification file." ) public class EndpointsCLI implements Callable { + public enum ParserType { + Endpoints, + States + } @SuppressWarnings("unused") @Parameters(index = "0", description = "The source endpoints DSL file.") private Path file; @SuppressWarnings("CanBeFinal") - @Option(names = {"-t", "--template"}, description = "The template directory. Default is ~/endpoints-templates") - private Path templateDir = Paths.get(System.getProperty("user.dir"), "endpoints-templates"); + @Option(names = {"-t", "--template"}, defaultValue = "endpoints-template", description = "The template directory. Default is ${DEFAULT-VALUE}") + private Path templateDir ; @SuppressWarnings("CanBeFinal") - @Option(names = {"-o", "--output"}, description = "The directory to write the generated code to. Default is ~/endpoints-output") - private Path outputDir = Paths.get(System.getProperty("user.dir"), "endpoints-output"); + @Option(names = {"-o", "--output"}, defaultValue = "endpoints-output", description = "The directory to write the generated code to. Default is ${DEFAULT-VALUE}") + private Path outputDir ; @SuppressWarnings("unused") @Option(names = {"-v", "--verbose"}, description = "Print verbose debug messages.") private Boolean verbose = false; + @Option(names = {"-p", "--parser"}, description = "Force use of a specific parser instead of determining from filename. Valid values: ${COMPLETION-CANDIDATES}.") + private ParserType parser = null; + public static void main(String[] args) { int exitCode = new CommandLine(new EndpointsCLI()).execute(args); System.exit(exitCode); @@ -61,8 +68,20 @@ public class EndpointsCLI implements Callable { validateTemplateDirectory(); validateInputFile(); validateOutputDirectory(); - verbose("Parsing " + file.toAbsolutePath()); - DocumentNode rootNode = ParserWrapper.parse(file); + verbose("Parsing: " + file.toAbsolutePath()); + if (parser == null) { + if (file.getFileName().toString().endsWith(".states")) { + parser = ParserType.States; + } + } + final DocumentNode rootNode ; + if (parser == ParserType.States) { + verbose("using state grammar.") ; + rootNode = ParserWrapper.parseStates(file); + } else { + verbose("using endpoints grammar.") ; + rootNode = ParserWrapper.parseEndpoints(file); + } verbose("AST: " + rootNode); verbose("Generating from templates in: " + templateDir.toAbsolutePath()); Generator generator = new Generator(templateDir, rootNode, outputDir); diff --git a/states-templates/Codecs.scala.ftl b/states-templates/Codecs.scala.ftl new file mode 100644 index 0000000..f11070f --- /dev/null +++ b/states-templates/Codecs.scala.ftl @@ -0,0 +1,19 @@ +// Copyright 2025 "Johan Maasing" +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +package ${config.package} + +object Codecs: +<#list typeDefinitions as type> + given Codec[${type.name?cap_first}] = deriveCodec + diff --git a/states-templates/Types.scala.ftl b/states-templates/Types.scala.ftl new file mode 100644 index 0000000..f2322b6 --- /dev/null +++ b/states-templates/Types.scala.ftl @@ -0,0 +1,23 @@ +// Copyright 2025 "Johan Maasing" +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +package ${config.package} + +object StateAndMessageTypes: +<#list typeDefinitions?sort as type> + case class ${type.name?cap_first}( + <#list type.fields as field> + ${field.name} : ${field.type}, + + ) + diff --git a/states-templates/nodes.md.ftl b/states-templates/nodes.md.ftl new file mode 100644 index 0000000..89a5603 --- /dev/null +++ b/states-templates/nodes.md.ftl @@ -0,0 +1,29 @@ +``` +Copyright 2025 "Johan Maasing" + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +``` + +# The nodes + +```mermaid +--- +title: ${config.title} +--- +stateDiagram-v2 +<#list states as state> + <#list state.transitions as transition> + ${state.name} --> ${transition.toState} : ${transition.message} + + +``` \ No newline at end of file diff --git a/test01.states b/test01.states new file mode 100644 index 0000000..bcb304f --- /dev/null +++ b/test01.states @@ -0,0 +1,21 @@ +/* + Copyright 2025 "Johan Maasing" + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +*/ + +{ title: SomeNodes, package: nu.zoom.dsl.states } + +start(s:S) -> middle(foo:foo): message(foo:foo), +middle -> middle(bar:bar): selfmessage(bar:bar), +middle -> end: message(bar:baz) From ac98e7948f9d12b269005b19e4efff9ecc84b59a Mon Sep 17 00:00:00 2001 From: Johan Maasing Date: Sun, 20 Apr 2025 11:48:25 +0200 Subject: [PATCH 10/20] [maven-release-plugin] prepare release v1.1 --- endgen-dist/pom.xml | 2 +- parser/pom.xml | 2 +- pom.xml | 4 ++-- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/endgen-dist/pom.xml b/endgen-dist/pom.xml index cfe5ff3..729e41a 100644 --- a/endgen-dist/pom.xml +++ b/endgen-dist/pom.xml @@ -5,7 +5,7 @@ nu.zoom.dsl endgen - 1.1-SNAPSHOT + 1.1 endgen-dist diff --git a/parser/pom.xml b/parser/pom.xml index a81607b..7444c5f 100644 --- a/parser/pom.xml +++ b/parser/pom.xml @@ -20,7 +20,7 @@ nu.zoom.dsl endgen - 1.1-SNAPSHOT + 1.1 parser diff --git a/pom.xml b/pom.xml index b234fa1..f53798f 100644 --- a/pom.xml +++ b/pom.xml @@ -19,7 +19,7 @@ nu.zoom.dsl endgen - 1.1-SNAPSHOT + 1.1 pom @@ -59,7 +59,7 @@ scm:git:https://codeberg.org/darkstar/endgen.git scm:git:ssh://git@vcs.zoom.nu:1122/zoom/endgen.git - admin-parent-1.1 + v1.1 From e87599708ea7f14d4092abe02aab8ba8b36fada8 Mon Sep 17 00:00:00 2001 From: Johan Maasing Date: Sun, 20 Apr 2025 11:48:27 +0200 Subject: [PATCH 11/20] [maven-release-plugin] prepare for next development iteration --- endgen-dist/pom.xml | 2 +- parser/pom.xml | 2 +- pom.xml | 4 ++-- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/endgen-dist/pom.xml b/endgen-dist/pom.xml index 729e41a..dda6922 100644 --- a/endgen-dist/pom.xml +++ b/endgen-dist/pom.xml @@ -5,7 +5,7 @@ nu.zoom.dsl endgen - 1.1 + 1.2-SNAPSHOT endgen-dist diff --git a/parser/pom.xml b/parser/pom.xml index 7444c5f..ba9d182 100644 --- a/parser/pom.xml +++ b/parser/pom.xml @@ -20,7 +20,7 @@ nu.zoom.dsl endgen - 1.1 + 1.2-SNAPSHOT parser diff --git a/pom.xml b/pom.xml index f53798f..86744cc 100644 --- a/pom.xml +++ b/pom.xml @@ -19,7 +19,7 @@ nu.zoom.dsl endgen - 1.1 + 1.2-SNAPSHOT pom @@ -59,7 +59,7 @@ scm:git:https://codeberg.org/darkstar/endgen.git scm:git:ssh://git@vcs.zoom.nu:1122/zoom/endgen.git - v1.1 + admin-parent-1.1 From bc458299dc85e1a0d45f2194ec1e16ab1ec98372 Mon Sep 17 00:00:00 2001 From: Johan Maasing Date: Sun, 20 Apr 2025 12:15:42 +0200 Subject: [PATCH 12/20] Fixup documentation --- README.md | 85 ++++++++++++++++++++++++++++++++----------------------- 1 file changed, 50 insertions(+), 35 deletions(-) diff --git a/README.md b/README.md index 2e73f92..d23fa85 100644 --- a/README.md +++ b/README.md @@ -37,10 +37,10 @@ Endgen currently contains two separate parsers: * endpoint - A DSL for expressing HTTP endpoints. * state - A DSL for expressing state and transitions. -Which parser that is used to read the input file is determined by the file name ending -'.endpoints' or '.states' or by a command line argument. +Only one parser will be sued when reading a file. Determined by the file name ending; +'.endpoints', or '.states', or by a command line argument. -The endpoint DSL and the state DSL share the grammar for expressing configuration and data types +The endpoint-DSL and the state-DSL share the grammar for expressing configuration and data types ,see below for details. ## How to Run @@ -89,15 +89,20 @@ a very limited DSL, you can for example not express what type of HTTP Verb to us no plans to extend the DSL to do that either. ## DSL -This is the ANTLR grammar for the root of the DSL +This is the ANTLR grammar for the root of the Endpoint-DSL ```antlrv4 document : generatorconfig? (namedTypeDeclaration|endpoint)* ; ``` +the corresponding grammar for the root of the State-DSL + +```antlrv4 +document : generatorconfig? transition (',' transition)* ; +``` + ### Configuration block -Meaning that the DSL file has an optional `generatorconfig` block at the top. Then you can write either; a type -definition, or an endpoint declaration, as many times as you like. +Both types of DSL files has an optional `generatorconfig` block at the top. Here is an example: ``` @@ -105,25 +110,32 @@ Here is an example: package: se.rutdev.senash, mykey: myvalue } +``` +This consists of a config block with 2 items, the 'package' and the 'mykey' definition. These are available to be used +in the freemarker template as a Map of String-keys to String-values. + +### Endpoint DSL +After the optional configuration block you can write either; a type definition, or an endpoint declaration, and repeat +as many times as you like. + +Here is an example: +``` /some/endpoint <- SomeType(foo:String) Embedded(foo:Bar) /some/other/endpoint <- (bar:Seq[Embedded]) ``` -This consists of a config block with 2 items, the 'package' and the 'mykey' definition. These are available to be used -in the freemarker template as a Map of String-keys to String-values. - ### Endpoint definition `/some/endpoint <- SomeType(foo:String)` is an endpoint definition. It declares one endpoint that have a request body data type called `SomeType` that has a field called `foo` of the type `String`. ### Data types -The DSL uses Scala convention of writing data types after the field name separated by a colon. Of course the DSL parser -does not know anything about java or scala types, as far as it is concerned these are 2 strings and the first one is -just named field-name and the other string is named field-type. +Both DSL-grammars use the Scala convention of writing data types after the field name separated by a colon. Of course +the parsers do not know anything about java or scala types, as far as the parser is concerned these are 2 strings and +the first one is just named: field-name and the other string is named: field-type. `Embedded(foo:Bar)` is a `namedTypeDeclaration` which is parsed the same way as the request type above. But isn't tied to a specific endpoint. @@ -148,45 +160,44 @@ It is possible to have an optional response data type declared like so: The right pointing arrow `->` denotes a response type, it can be an anonymous data type in which case the parser till name it from the last path segment and add 'Response' to the end of the data type name. -### State grammar - +### State DSL This is an example of a state file: ``` -start -> middle: message, -middle -> middle: selfmessage, -middle -> end: endmessage +start -> middle: message, +middle -> middle: selfmessage, +middle -> end: endmessage ``` +The file declares 3 transitions. The first line states: Transition from the 'start' state to the 'middle' state with +the message 'message'. -It contains 3 state definitions `start`, `middle` and `end`. A state definition will be parsed as a data type with -the name of the state as the type name. - -It also contains 3 message definitions `message`, `selfmessage` and `endmessage`. Message definitions will also be -parsed as data types. +From this we can see that the file contains 3 state definitions `start`, `middle` and `end`. +A state definition will be parsed as a data type with the name of the state as the type name. It also contains 3 +message definitions `message`, `selfmessage` and `endmessage`. Message definitions will also be parsed as data types. Since the parser will extract datatypes it is possible to define the fields of the data types. This is a slightly more complicated example: ``` -start(foo:Foo) -> middle: message(a: String), -middle(bar:Bar) -> middle: selfmessage, -middle -> end: endmessage +start -> middle: message(a: String), +middle(bar:Bar) -> middle: selfmessage, +middle -> end: endmessage ``` -Where for example the data type for `middle` will have the field declaration with the name `bar` and the type `Bar`. +The data type for `middle` will have a field declaration with the name `bar` and the type `Bar`. Fields for the same state data type, or message data type, will be merged. Here is a complex example: ``` -start(s:S) -> middle(foo:foo): message(foo:foo), -middle -> middle(bar:bar): selfmessage(bar:bar), -middle -> end: message(bar:baz) +start(s:S) -> middle(foo:foo): message(foo:foo), +middle -> middle(bar:bar): selfmessage(bar:bar), +middle -> end: message(bar:baz) ``` -Not that we can declare fields on both the `from` and `to` state declarations. The `middle` datat type will have field +Note that we can declare fields on both the `from` and `to` state declarations. The `middle` datat type will have field definitons for `foo` and `bar`. The data type for `message` will have fields for `foo` and `bar`. -One restriction is that states and messages may not have the same name, i.e. be parsed as the same data type. +One restriction is that a state and a messages may share have the same name, i.e. be parsed as the same data type. ## Generating If the parser is successful it will hold the following data in the AST @@ -221,8 +232,8 @@ that writes the value for a config key called 'package'. `package ${config.package}` ### Data types -These are all the data types the parser have collected, either from explicit declarations, request payloads and response -bodies. +These are all the data types the parser have collected, either from explicit declarations, request payloads, response +bodies, states or messages. ```java public record TypeNode(String name, List fields) { } @@ -281,11 +292,15 @@ The set of states will hold items of this shape: public record StateNode(String name, String data, Set transitions) { } ``` +* `name` is the name of the state. +* `data` is the name of the data type for the state. +* `transistions` are the outgoing arrows from the named state. -and the transitions has this structure: +Transitions have this structure: ```injectedfreemarker public record TransitionNode(String message, String toState) { } ``` - +* `name` is the message name. +* `toState` is the name of the target state. From c1c062b6cf39fb41772b6869f4a2eb8349a9e5f6 Mon Sep 17 00:00:00 2001 From: Johan Maasing Date: Tue, 22 Apr 2025 20:28:17 +0200 Subject: [PATCH 13/20] Changed grammar for states --- README.md | 18 +++++++++--------- .../main/antlr4/nu/zoom/dsl/parser/States.g4 | 2 +- test01.states | 6 +++--- 3 files changed, 13 insertions(+), 13 deletions(-) diff --git a/README.md b/README.md index d23fa85..d4306c2 100644 --- a/README.md +++ b/README.md @@ -163,9 +163,9 @@ name it from the last path segment and add 'Response' to the end of the data typ ### State DSL This is an example of a state file: ``` -start -> middle: message, -middle -> middle: selfmessage, -middle -> end: endmessage +start -> message -> middle , +middle -> selfmessage -> middle, +middle -> endmessage -> end ``` The file declares 3 transitions. The first line states: Transition from the 'start' state to the 'middle' state with the message 'message'. @@ -178,18 +178,18 @@ Since the parser will extract datatypes it is possible to define the fields of t complicated example: ``` -start -> middle: message(a: String), -middle(bar:Bar) -> middle: selfmessage, -middle -> end: endmessage +start -> message -> middle, +middle -> selfmessage -> middle(bar:bar), +middle -> message -> end ``` The data type for `middle` will have a field declaration with the name `bar` and the type `Bar`. Fields for the same state data type, or message data type, will be merged. Here is a complex example: ``` -start(s:S) -> middle(foo:foo): message(foo:foo), -middle -> middle(bar:bar): selfmessage(bar:bar), -middle -> end: message(bar:baz) +start(s:S) -> message(foo:foo) -> middle(foo:foo) , +middle -> selfmessage(bar:bar) -> middle(bar:bar), +middle -> message(bar:baz) -> end ``` Note that we can declare fields on both the `from` and `to` state declarations. The `middle` datat type will have field diff --git a/parser/src/main/antlr4/nu/zoom/dsl/parser/States.g4 b/parser/src/main/antlr4/nu/zoom/dsl/parser/States.g4 index dbfe628..d2a2b4d 100644 --- a/parser/src/main/antlr4/nu/zoom/dsl/parser/States.g4 +++ b/parser/src/main/antlr4/nu/zoom/dsl/parser/States.g4 @@ -15,7 +15,7 @@ grammar States; import Common; document : generatorconfig? transition (',' transition)* ; -transition : from RIGHT_ARROW to COLON message ; +transition : from RIGHT_ARROW message RIGHT_ARROW to ; from : state ; to : state ; message : typeName typeDeclaration? ; diff --git a/test01.states b/test01.states index bcb304f..3b9a67a 100644 --- a/test01.states +++ b/test01.states @@ -16,6 +16,6 @@ { title: SomeNodes, package: nu.zoom.dsl.states } -start(s:S) -> middle(foo:foo): message(foo:foo), -middle -> middle(bar:bar): selfmessage(bar:bar), -middle -> end: message(bar:baz) +start(s:S) -> message(foo:foo) -> middle(foo:foo) , +middle -> selfmessage(bar:bar) -> middle(bar:bar), +middle -> message(bar:baz) -> end From 68dc70c17618b0feaaaebf3d9c258d25c9b29793 Mon Sep 17 00:00:00 2001 From: Johan Maasing Date: Fri, 2 May 2025 08:44:44 +0200 Subject: [PATCH 14/20] Makes it possible to put templates in subdirectories --- .../{ => nu/zoom/dsl}/Codecs.scala.ftl | 0 .../{ => nu/zoom/dsl}/Endpoints.scala.ftl | 0 .../{ => nu/zoom/dsl}/Protocol.scala.ftl | 0 .../nu/zoom/dsl/freemarker/Generator.java | 24 +++++++++++-------- 4 files changed, 14 insertions(+), 10 deletions(-) rename endpoints-templates/{ => nu/zoom/dsl}/Codecs.scala.ftl (100%) rename endpoints-templates/{ => nu/zoom/dsl}/Endpoints.scala.ftl (100%) rename endpoints-templates/{ => nu/zoom/dsl}/Protocol.scala.ftl (100%) diff --git a/endpoints-templates/Codecs.scala.ftl b/endpoints-templates/nu/zoom/dsl/Codecs.scala.ftl similarity index 100% rename from endpoints-templates/Codecs.scala.ftl rename to endpoints-templates/nu/zoom/dsl/Codecs.scala.ftl diff --git a/endpoints-templates/Endpoints.scala.ftl b/endpoints-templates/nu/zoom/dsl/Endpoints.scala.ftl similarity index 100% rename from endpoints-templates/Endpoints.scala.ftl rename to endpoints-templates/nu/zoom/dsl/Endpoints.scala.ftl diff --git a/endpoints-templates/Protocol.scala.ftl b/endpoints-templates/nu/zoom/dsl/Protocol.scala.ftl similarity index 100% rename from endpoints-templates/Protocol.scala.ftl rename to endpoints-templates/nu/zoom/dsl/Protocol.scala.ftl diff --git a/parser/src/main/java/nu/zoom/dsl/freemarker/Generator.java b/parser/src/main/java/nu/zoom/dsl/freemarker/Generator.java index dcfa86a..1ac2cb0 100644 --- a/parser/src/main/java/nu/zoom/dsl/freemarker/Generator.java +++ b/parser/src/main/java/nu/zoom/dsl/freemarker/Generator.java @@ -21,6 +21,7 @@ import nu.zoom.dsl.ast.DocumentNode; import java.io.IOException; import java.nio.charset.StandardCharsets; +import java.nio.file.FileVisitOption; import java.nio.file.Files; import java.nio.file.Path; import java.util.ArrayList; @@ -50,17 +51,20 @@ public class Generator { } public List generate() throws IOException, TemplateException { - try (Stream files = Files.list(templatesDir)) { - List templates = files - .map(Path::getFileName) - .map(Path::toString) - .filter(p -> p.length() > TEMPLATE_EXTENSION_LENGTH && p.endsWith(TEMPLATE_EXTENSION) - ) - .toList(); + try (Stream files = Files.walk(templatesDir)) { + List templates = files + .filter(p -> { + var fname = p.getFileName().toString(); + return fname.length() > TEMPLATE_EXTENSION_LENGTH && fname.endsWith(TEMPLATE_EXTENSION) ; + } + ) + .map(p -> templatesDir.relativize(p)) + .toList(); ArrayList out = new ArrayList<>(); - for (String template : templates) { - Path outpath = outputDir.resolve(outputFilenameFromTemplate(template)); - Template ftl = this.cfg.getTemplate(template); + for (Path template : templates) { + Path outpath = outputDir.resolve(outputFilenameFromTemplate(template.toString())); + Files.createDirectories(outpath.getParent()); + Template ftl = this.cfg.getTemplate(template.toString()); try (var outw = Files.newBufferedWriter(outpath, StandardCharsets.UTF_8)) { ftl.process(this.data, outw); out.add(outpath); From 7e8aa018e95541affd65b07b1065484f287d834d Mon Sep 17 00:00:00 2001 From: Johan Maasing Date: Fri, 2 May 2025 09:25:27 +0200 Subject: [PATCH 15/20] Makes it possible to put templates in subdirectories --- README.md | 22 +++- endpoints-templates/endpoints.txt.ftl | 2 + .../nu/zoom/dsl/Codecs.scala.ftl | 2 +- .../nu/zoom/dsl/Endpoints.scala.ftl | 2 +- .../nu/zoom/dsl/Protocol.scala.ftl | 2 +- .../java/nu/zoom/dsl/ast/DocumentNode.java | 1 + .../java/nu/zoom/dsl/ast/GeneratorNode.java | 13 +++ .../src/main/java/nu/zoom/dsl/ast/Meta.java | 9 ++ .../nu/zoom/dsl/freemarker/Generator.java | 109 ++++++++++-------- 9 files changed, 109 insertions(+), 53 deletions(-) create mode 100644 parser/src/main/java/nu/zoom/dsl/ast/GeneratorNode.java create mode 100644 parser/src/main/java/nu/zoom/dsl/ast/Meta.java diff --git a/README.md b/README.md index d4306c2..7dc6cc1 100644 --- a/README.md +++ b/README.md @@ -203,11 +203,12 @@ One restriction is that a state and a messages may share have the same name, i.e If the parser is successful it will hold the following data in the AST ```java -public record DocumentNode( - Map config, - Set typeDefinitions, - List endpoints, - Set states) { +public record GeneratorNode( + Map config, + Set typeDefinitions, + List endpoints, + Set states, + Meta meta) { } ``` @@ -304,3 +305,14 @@ public record TransitionNode(String message, String toState) { ``` * `name` is the message name. * `toState` is the name of the target state. + +### Meta + +The meta container holds information about the template file used to generate the output file. +It holds the subdirectory below the given `templateDir` where the template was found. + +This is useful to generate e.g. java `package`statements like this: + +```injectedfreemarker +<#list meta.templateDirectories>package <#items as dir>${dir}<#sep>.; +``` \ No newline at end of file diff --git a/endpoints-templates/endpoints.txt.ftl b/endpoints-templates/endpoints.txt.ftl index 9a779fb..4904c00 100644 --- a/endpoints-templates/endpoints.txt.ftl +++ b/endpoints-templates/endpoints.txt.ftl @@ -1,3 +1,5 @@ +Generated from template: ${meta.templateFile} + <#list endpoints as endpoint> <#list endpoint.paths.paths> <#items as segment>/${segment} diff --git a/endpoints-templates/nu/zoom/dsl/Codecs.scala.ftl b/endpoints-templates/nu/zoom/dsl/Codecs.scala.ftl index f11070f..7ea382d 100644 --- a/endpoints-templates/nu/zoom/dsl/Codecs.scala.ftl +++ b/endpoints-templates/nu/zoom/dsl/Codecs.scala.ftl @@ -11,7 +11,7 @@ // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. -package ${config.package} +<#list meta.templateDirectories>package <#items as dir>${dir}<#sep>.; object Codecs: <#list typeDefinitions as type> diff --git a/endpoints-templates/nu/zoom/dsl/Endpoints.scala.ftl b/endpoints-templates/nu/zoom/dsl/Endpoints.scala.ftl index a68864c..fbc3538 100644 --- a/endpoints-templates/nu/zoom/dsl/Endpoints.scala.ftl +++ b/endpoints-templates/nu/zoom/dsl/Endpoints.scala.ftl @@ -11,7 +11,7 @@ // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. -package ${config.package} +<#list meta.templateDirectories>package<#items as dir>${dir}<#sep>.; class Endpoints: <#list endpoints as endpoint> diff --git a/endpoints-templates/nu/zoom/dsl/Protocol.scala.ftl b/endpoints-templates/nu/zoom/dsl/Protocol.scala.ftl index faa5d55..41e6547 100644 --- a/endpoints-templates/nu/zoom/dsl/Protocol.scala.ftl +++ b/endpoints-templates/nu/zoom/dsl/Protocol.scala.ftl @@ -11,7 +11,7 @@ // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. -package ${config.package} +<#list meta.templateDirectories>package<#items as dir>${dir}<#sep>.; object Protocol: <#list typeDefinitions?sort as type> diff --git a/parser/src/main/java/nu/zoom/dsl/ast/DocumentNode.java b/parser/src/main/java/nu/zoom/dsl/ast/DocumentNode.java index fcb8621..7edb239 100644 --- a/parser/src/main/java/nu/zoom/dsl/ast/DocumentNode.java +++ b/parser/src/main/java/nu/zoom/dsl/ast/DocumentNode.java @@ -13,6 +13,7 @@ // limitations under the License. package nu.zoom.dsl.ast; +import java.nio.file.Path; import java.util.List; import java.util.Map; import java.util.Set; diff --git a/parser/src/main/java/nu/zoom/dsl/ast/GeneratorNode.java b/parser/src/main/java/nu/zoom/dsl/ast/GeneratorNode.java new file mode 100644 index 0000000..2cea7ca --- /dev/null +++ b/parser/src/main/java/nu/zoom/dsl/ast/GeneratorNode.java @@ -0,0 +1,13 @@ +package nu.zoom.dsl.ast; + +import java.util.List; +import java.util.Map; +import java.util.Set; + +public record GeneratorNode( + Map config, + Set typeDefinitions, + List endpoints, + Set states, + Meta meta) { +} diff --git a/parser/src/main/java/nu/zoom/dsl/ast/Meta.java b/parser/src/main/java/nu/zoom/dsl/ast/Meta.java new file mode 100644 index 0000000..567f3ab --- /dev/null +++ b/parser/src/main/java/nu/zoom/dsl/ast/Meta.java @@ -0,0 +1,9 @@ +package nu.zoom.dsl.ast; + +import java.util.List; + +public record Meta( + List templateDirectories, + String templateFile +) { +} diff --git a/parser/src/main/java/nu/zoom/dsl/freemarker/Generator.java b/parser/src/main/java/nu/zoom/dsl/freemarker/Generator.java index 1ac2cb0..0458669 100644 --- a/parser/src/main/java/nu/zoom/dsl/freemarker/Generator.java +++ b/parser/src/main/java/nu/zoom/dsl/freemarker/Generator.java @@ -18,63 +18,82 @@ import freemarker.template.Template; import freemarker.template.TemplateException; import freemarker.template.TemplateExceptionHandler; import nu.zoom.dsl.ast.DocumentNode; +import nu.zoom.dsl.ast.GeneratorNode; +import nu.zoom.dsl.ast.Meta; import java.io.IOException; import java.nio.charset.StandardCharsets; -import java.nio.file.FileVisitOption; import java.nio.file.Files; import java.nio.file.Path; import java.util.ArrayList; import java.util.List; import java.util.Objects; +import java.util.stream.Collectors; import java.util.stream.Stream; public class Generator { - private final Path templatesDir; - private final DocumentNode data; - private final Path outputDir; - private final Configuration cfg; - private final String TEMPLATE_EXTENSION = ".ftl"; - private final int TEMPLATE_EXTENSION_LENGTH = TEMPLATE_EXTENSION.length(); + private final Path templatesDir; + private final DocumentNode documentNode; + private final Path outputDir; + private final Configuration cfg; + private final String TEMPLATE_EXTENSION = ".ftl"; + private final int TEMPLATE_EXTENSION_LENGTH = TEMPLATE_EXTENSION.length(); - public Generator(Path templatesDir, DocumentNode data, Path outputDir) throws IOException { - this.templatesDir = Objects.requireNonNull(templatesDir); - this.data = Objects.requireNonNull(data); - this.outputDir = Objects.requireNonNull(outputDir); - this.cfg = new Configuration(Configuration.VERSION_2_3_34); - cfg.setDirectoryForTemplateLoading(templatesDir.toFile()); - cfg.setDefaultEncoding("UTF-8"); - cfg.setTemplateExceptionHandler(TemplateExceptionHandler.RETHROW_HANDLER); - cfg.setLogTemplateExceptions(false); - cfg.setWrapUncheckedExceptions(true); - cfg.setFallbackOnNullLoopVariable(false); - } + public Generator(Path templatesDir, DocumentNode documentNode, Path outputDir) throws IOException { + this.templatesDir = Objects.requireNonNull(templatesDir); + this.documentNode = Objects.requireNonNull(documentNode); + this.outputDir = Objects.requireNonNull(outputDir); + this.cfg = new Configuration(Configuration.VERSION_2_3_34); + cfg.setDirectoryForTemplateLoading(templatesDir.toFile()); + cfg.setDefaultEncoding("UTF-8"); + cfg.setTemplateExceptionHandler(TemplateExceptionHandler.RETHROW_HANDLER); + cfg.setLogTemplateExceptions(false); + cfg.setWrapUncheckedExceptions(true); + cfg.setFallbackOnNullLoopVariable(false); + } - public List generate() throws IOException, TemplateException { - try (Stream files = Files.walk(templatesDir)) { - List templates = files - .filter(p -> { - var fname = p.getFileName().toString(); - return fname.length() > TEMPLATE_EXTENSION_LENGTH && fname.endsWith(TEMPLATE_EXTENSION) ; - } - ) - .map(p -> templatesDir.relativize(p)) - .toList(); - ArrayList out = new ArrayList<>(); - for (Path template : templates) { - Path outpath = outputDir.resolve(outputFilenameFromTemplate(template.toString())); - Files.createDirectories(outpath.getParent()); - Template ftl = this.cfg.getTemplate(template.toString()); - try (var outw = Files.newBufferedWriter(outpath, StandardCharsets.UTF_8)) { - ftl.process(this.data, outw); - out.add(outpath); - } - } - return out; - } - } + public List generate() throws IOException, TemplateException { + try (Stream files = Files.walk(templatesDir)) { + List templates = files + .filter(p -> { + var fname = p.getFileName().toString(); + return fname.length() > TEMPLATE_EXTENSION_LENGTH && fname.endsWith(TEMPLATE_EXTENSION); + } + ) + .map(p -> templatesDir.relativize(p)) + .toList(); + ArrayList out = new ArrayList<>(); + for (Path template : templates) { + Path outpath = outputDir.resolve(outputFilenameFromTemplate(template.toString())); + Files.createDirectories(outpath.getParent()); + Path templateSubdirectory = template.getParent(); + ArrayList templateDirectories= new ArrayList<>() ; + if (templateSubdirectory != null) { + var ti = templateSubdirectory.iterator(); + while (ti.hasNext()) { + templateDirectories.add(ti.next().toString()); + } + } + String templateName = template.getFileName().toString(); + Meta meta = new Meta(templateDirectories, templateName); + GeneratorNode generatorNode = new GeneratorNode( + this.documentNode.config(), + this.documentNode.typeDefinitions(), + this.documentNode.endpoints(), + this.documentNode.states(), + meta + ); + Template ftl = this.cfg.getTemplate(template.toString()); + try (var outw = Files.newBufferedWriter(outpath, StandardCharsets.UTF_8)) { + ftl.process(generatorNode, outw); + out.add(outpath); + } + } + return out; + } + } - private String outputFilenameFromTemplate(String template) { - return template.substring(0, template.length() - TEMPLATE_EXTENSION_LENGTH); - } + private String outputFilenameFromTemplate(String template) { + return template.substring(0, template.length() - TEMPLATE_EXTENSION_LENGTH); + } } From e0922d5639f1f5a5e7a17b264439a10b92bfa9f4 Mon Sep 17 00:00:00 2001 From: Johan Maasing Date: Fri, 2 May 2025 09:25:27 +0200 Subject: [PATCH 16/20] Makes it possible to put templates in subdirectories --- README.md | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 7dc6cc1..3262cf7 100644 --- a/README.md +++ b/README.md @@ -309,10 +309,18 @@ public record TransitionNode(String message, String toState) { ### Meta The meta container holds information about the template file used to generate the output file. -It holds the subdirectory below the given `templateDir` where the template was found. +```java +public record Meta( + List templateDirectories, + String templateFile +) { } +``` +* `templateDirectories` holds the subdirectory below the given `templateDir` where the template was found. This is useful to generate e.g. java `package`statements like this: ```injectedfreemarker <#list meta.templateDirectories>package <#items as dir>${dir}<#sep>.; -``` \ No newline at end of file +``` + +* `templateFile` is the filename of the template (including the .ftl-ending) used to generate the output. \ No newline at end of file From 46da5c50195aeba0ddb71cb8625f8a6be1514984 Mon Sep 17 00:00:00 2001 From: Johan Maasing Date: Sun, 4 May 2025 09:41:16 +0200 Subject: [PATCH 17/20] needs testing --- endgen-maven-plugin/README.md | 8 ++ endgen-maven-plugin/pom.xml | 69 +++++++++++++ .../java/nu/zoom/dsl/maven/EndgenMojo.java | 64 +++++++++++++ .../java/nu/zoom/dsl/maven/MavenLogger.java | 17 ++++ .../src/main/java/nu/zoom/dsl/maven/Run.java | 52 ++++++++++ .../java/nu/zoom/dsl/cli/EndpointsCLI.java | 48 +++------- .../java/nu/zoom/dsl/run/EndgenException.java | 15 +++ .../nu/zoom/dsl/run/GeneratorException.java | 15 +++ .../src/main/java/nu/zoom/dsl/run/Logger.java | 5 + .../main/java/nu/zoom/dsl/run/NullLogger.java | 8 ++ .../java/nu/zoom/dsl/run/ParserException.java | 15 +++ .../zoom/dsl/{ast => run}/ParserWrapper.java | 5 +- .../src/main/java/nu/zoom/dsl/run/Runner.java | 96 +++++++++++++++++++ .../java/nu/zoom/dsl/run/StdoutLogger.java | 9 ++ .../nu/zoom/dsl/run/ValidationException.java | 15 +++ pom.xml | 1 + 16 files changed, 406 insertions(+), 36 deletions(-) create mode 100644 endgen-maven-plugin/README.md create mode 100644 endgen-maven-plugin/pom.xml create mode 100644 endgen-maven-plugin/src/main/java/nu/zoom/dsl/maven/EndgenMojo.java create mode 100644 endgen-maven-plugin/src/main/java/nu/zoom/dsl/maven/MavenLogger.java create mode 100644 endgen-maven-plugin/src/main/java/nu/zoom/dsl/maven/Run.java create mode 100644 parser/src/main/java/nu/zoom/dsl/run/EndgenException.java create mode 100644 parser/src/main/java/nu/zoom/dsl/run/GeneratorException.java create mode 100644 parser/src/main/java/nu/zoom/dsl/run/Logger.java create mode 100644 parser/src/main/java/nu/zoom/dsl/run/NullLogger.java create mode 100644 parser/src/main/java/nu/zoom/dsl/run/ParserException.java rename parser/src/main/java/nu/zoom/dsl/{ast => run}/ParserWrapper.java (93%) create mode 100644 parser/src/main/java/nu/zoom/dsl/run/Runner.java create mode 100644 parser/src/main/java/nu/zoom/dsl/run/StdoutLogger.java create mode 100644 parser/src/main/java/nu/zoom/dsl/run/ValidationException.java diff --git a/endgen-maven-plugin/README.md b/endgen-maven-plugin/README.md new file mode 100644 index 0000000..1e02921 --- /dev/null +++ b/endgen-maven-plugin/README.md @@ -0,0 +1,8 @@ +# References +## Maven + +https://maven.apache.org/plugins/maven-compiler-plugin/compile-mojo.html#generatedsourcesdirectory + +### For executing the plugin several times +See executions +https://maven.apache.org/guides/mini/guide-configuring-plugins.html \ No newline at end of file diff --git a/endgen-maven-plugin/pom.xml b/endgen-maven-plugin/pom.xml new file mode 100644 index 0000000..e9f42c5 --- /dev/null +++ b/endgen-maven-plugin/pom.xml @@ -0,0 +1,69 @@ + + + 4.0.0 + + + nu.zoom.dsl + endgen + 1.2-SNAPSHOT + + endgen-maven-plugin + maven-plugin + + + 3.15.1 + + + + org.apache.maven + maven-plugin-api + 3.9.9 + provided + + + + org.apache.maven.plugin-tools + maven-plugin-annotations + ${maven-plugin-tools.version} + provided + + + nu.zoom.dsl + parser + ${project.parent.version} + + + + + + + org.apache.maven.plugins + maven-plugin-plugin + ${maven-plugin-tools.version} + + + help-mojo + + helpmojo + + + + + + nu.zoom.dsl + endgen-maven-plugin + 1.2-SNAPSHOT + + + + ${project.build.sourceDirectory}/main/endgen-templates + ${project.build.sourceDirectory}/generated-sources/endgen endpoints-output + ${project.basedir}/../test01.endpoints + + + + + + + + \ No newline at end of file diff --git a/endgen-maven-plugin/src/main/java/nu/zoom/dsl/maven/EndgenMojo.java b/endgen-maven-plugin/src/main/java/nu/zoom/dsl/maven/EndgenMojo.java new file mode 100644 index 0000000..8d204aa --- /dev/null +++ b/endgen-maven-plugin/src/main/java/nu/zoom/dsl/maven/EndgenMojo.java @@ -0,0 +1,64 @@ +package nu.zoom.dsl.maven; + +import nu.zoom.dsl.run.Runner; +import nu.zoom.dsl.run.ValidationException; +import org.apache.maven.plugin.AbstractMojo; +import org.apache.maven.plugin.MojoExecutionException; +import org.apache.maven.plugin.MojoFailureException; +import org.apache.maven.plugins.annotations.LifecyclePhase; +import org.apache.maven.plugins.annotations.Mojo; +import org.apache.maven.plugins.annotations.Parameter; + +import java.io.File; +import java.util.Optional; + +@Mojo( + name = "endgen", + defaultPhase = LifecyclePhase.GENERATE_SOURCES +) +public class EndgenMojo extends AbstractMojo { + @Parameter(defaultValue = "${project.build.sourceDirectory}/main/endgen-templates") + File templates; + + @Parameter(defaultValue = "${project.build.outputDirectory}/generated-sources/endgen") + File output; + + @Parameter + File dsl; + + @Parameter + String parser; + + + @Override + public void execute() throws MojoExecutionException, MojoFailureException { + getLog().info("Running endgen"); + getLog().info("Using dsl: " + dsl); + try { + Runner.run( + optional(dsl).map(File::toPath).orElseThrow(), + optional(templates).map(File::toPath), + optional(output).map(File::toPath), + getParserType(parser), + new MavenLogger(getLog()) + ); + } catch (Exception e) { + throw new MojoExecutionException(e.getMessage(), e); + } + } + + private Optional getParserType(final String type) throws ValidationException { + if (type == null) { + return Optional.empty(); + } + try { + return Optional.of(Runner.ParserType.valueOf(type)); + } catch (IllegalArgumentException e) { + throw new ValidationException(e); + } + } + + private Optional optional(T arg) { + return arg == null ? Optional.empty() : Optional.of(arg); + } +} diff --git a/endgen-maven-plugin/src/main/java/nu/zoom/dsl/maven/MavenLogger.java b/endgen-maven-plugin/src/main/java/nu/zoom/dsl/maven/MavenLogger.java new file mode 100644 index 0000000..03b98be --- /dev/null +++ b/endgen-maven-plugin/src/main/java/nu/zoom/dsl/maven/MavenLogger.java @@ -0,0 +1,17 @@ +package nu.zoom.dsl.maven; + +import nu.zoom.dsl.run.Logger; +import org.apache.maven.plugin.logging.Log; + +public class MavenLogger implements Logger { + private final Log delegate; + + public MavenLogger(Log delegate) { + this.delegate = delegate; + } + + @Override + public void println(String message) { + this.delegate.debug(message); + } +} diff --git a/endgen-maven-plugin/src/main/java/nu/zoom/dsl/maven/Run.java b/endgen-maven-plugin/src/main/java/nu/zoom/dsl/maven/Run.java new file mode 100644 index 0000000..c8a0779 --- /dev/null +++ b/endgen-maven-plugin/src/main/java/nu/zoom/dsl/maven/Run.java @@ -0,0 +1,52 @@ +package nu.zoom.dsl.maven; + +import java.io.File; + +public class Run { + private File templates; + private File output; + private File dsl; + private String parser; + + public File getTemplates() { + return templates; + } + + public void setTemplates(File templates) { + this.templates = templates; + } + + public File getOutput() { + return output; + } + + public void setOutput(File output) { + this.output = output; + } + + public File getDsl() { + return dsl; + } + + public void setDsl(File dsl) { + this.dsl = dsl; + } + + public String getParser() { + return parser; + } + + public void setParser(String parser) { + this.parser = parser; + } + + @Override + public String toString() { + return "Run{" + + "templates=" + templates + + ", output=" + output + + ", dsl=" + dsl + + ", parser='" + parser + '\'' + + '}'; + } +} diff --git a/parser/src/main/java/nu/zoom/dsl/cli/EndpointsCLI.java b/parser/src/main/java/nu/zoom/dsl/cli/EndpointsCLI.java index e3d5794..8d79757 100644 --- a/parser/src/main/java/nu/zoom/dsl/cli/EndpointsCLI.java +++ b/parser/src/main/java/nu/zoom/dsl/cli/EndpointsCLI.java @@ -14,7 +14,7 @@ package nu.zoom.dsl.cli; import nu.zoom.dsl.ast.DocumentNode; -import nu.zoom.dsl.ast.ParserWrapper; +import nu.zoom.dsl.run.*; import nu.zoom.dsl.freemarker.Generator; import picocli.CommandLine; import picocli.CommandLine.Command; @@ -24,8 +24,8 @@ import picocli.CommandLine.Parameters; import java.io.IOException; import java.nio.file.Files; import java.nio.file.Path; -import java.nio.file.Paths; import java.util.List; +import java.util.Optional; import java.util.concurrent.Callable; @Command( @@ -34,20 +34,16 @@ import java.util.concurrent.Callable; description = "Generate source code from an endpoints specification file." ) public class EndpointsCLI implements Callable { - public enum ParserType { - Endpoints, - States - } @SuppressWarnings("unused") @Parameters(index = "0", description = "The source endpoints DSL file.") private Path file; @SuppressWarnings("CanBeFinal") - @Option(names = {"-t", "--template"}, defaultValue = "endpoints-template", description = "The template directory. Default is ${DEFAULT-VALUE}") + @Option(names = {"-t", "--template"}, defaultValue = Runner.DEFAULT_TEMPLATE_DIRECTORY_NAME, description = "The template directory. Default is ${DEFAULT-VALUE}") private Path templateDir ; @SuppressWarnings("CanBeFinal") - @Option(names = {"-o", "--output"}, defaultValue = "endpoints-output", description = "The directory to write the generated code to. Default is ${DEFAULT-VALUE}") + @Option(names = {"-o", "--output"}, defaultValue = Runner.DEFAULT_OUTPUT_DIRECTORY_NAME, description = "The directory to write the generated code to. Default is ${DEFAULT-VALUE}") private Path outputDir ; @SuppressWarnings("unused") @@ -55,7 +51,7 @@ public class EndpointsCLI implements Callable { private Boolean verbose = false; @Option(names = {"-p", "--parser"}, description = "Force use of a specific parser instead of determining from filename. Valid values: ${COMPLETION-CANDIDATES}.") - private ParserType parser = null; + private Runner.ParserType parser = null; public static void main(String[] args) { int exitCode = new CommandLine(new EndpointsCLI()).execute(args); @@ -65,32 +61,14 @@ public class EndpointsCLI implements Callable { @Override public Integer call() { try { - validateTemplateDirectory(); - validateInputFile(); - validateOutputDirectory(); - verbose("Parsing: " + file.toAbsolutePath()); - if (parser == null) { - if (file.getFileName().toString().endsWith(".states")) { - parser = ParserType.States; - } - } - final DocumentNode rootNode ; - if (parser == ParserType.States) { - verbose("using state grammar.") ; - rootNode = ParserWrapper.parseStates(file); - } else { - verbose("using endpoints grammar.") ; - rootNode = ParserWrapper.parseEndpoints(file); - } - verbose("AST: " + rootNode); - verbose("Generating from templates in: " + templateDir.toAbsolutePath()); - Generator generator = new Generator(templateDir, rootNode, outputDir); - List generatedPaths = generator.generate(); - if (generatedPaths.isEmpty()) { - System.out.println("No generated paths found."); - } else { - generatedPaths.forEach(p -> verbose("Generated: " + p.toAbsolutePath())); - } + final Logger logger = this.verbose ? new StdoutLogger() : new NullLogger() ; + Runner.run( + this.file, + Optional.of(this.templateDir), + Optional.of(this.outputDir), + parser == null ? Optional.empty() : Optional.of(parser), + logger + ); return 0; } catch (Exception e) { System.err.println(e.getMessage()); diff --git a/parser/src/main/java/nu/zoom/dsl/run/EndgenException.java b/parser/src/main/java/nu/zoom/dsl/run/EndgenException.java new file mode 100644 index 0000000..7771e11 --- /dev/null +++ b/parser/src/main/java/nu/zoom/dsl/run/EndgenException.java @@ -0,0 +1,15 @@ +package nu.zoom.dsl.run; + +public abstract class EndgenException extends Exception { + public EndgenException(String message, Throwable cause) { + super(message, cause); + } + + public EndgenException(Throwable cause) { + super(cause); + } + + public EndgenException(String message) { + super(message); + } +} diff --git a/parser/src/main/java/nu/zoom/dsl/run/GeneratorException.java b/parser/src/main/java/nu/zoom/dsl/run/GeneratorException.java new file mode 100644 index 0000000..5157d97 --- /dev/null +++ b/parser/src/main/java/nu/zoom/dsl/run/GeneratorException.java @@ -0,0 +1,15 @@ +package nu.zoom.dsl.run; + +public class GeneratorException extends EndgenException { + public GeneratorException(String message) { + super(message); + } + + public GeneratorException(String message, Throwable cause) { + super(message, cause); + } + + public GeneratorException(Throwable cause) { + super(cause); + } +} diff --git a/parser/src/main/java/nu/zoom/dsl/run/Logger.java b/parser/src/main/java/nu/zoom/dsl/run/Logger.java new file mode 100644 index 0000000..80826d3 --- /dev/null +++ b/parser/src/main/java/nu/zoom/dsl/run/Logger.java @@ -0,0 +1,5 @@ +package nu.zoom.dsl.run; + +public interface Logger { + void println(String message); +} diff --git a/parser/src/main/java/nu/zoom/dsl/run/NullLogger.java b/parser/src/main/java/nu/zoom/dsl/run/NullLogger.java new file mode 100644 index 0000000..014f473 --- /dev/null +++ b/parser/src/main/java/nu/zoom/dsl/run/NullLogger.java @@ -0,0 +1,8 @@ +package nu.zoom.dsl.run; + +public class NullLogger implements Logger { + @Override + public void println(String message) { + + } +} diff --git a/parser/src/main/java/nu/zoom/dsl/run/ParserException.java b/parser/src/main/java/nu/zoom/dsl/run/ParserException.java new file mode 100644 index 0000000..b726384 --- /dev/null +++ b/parser/src/main/java/nu/zoom/dsl/run/ParserException.java @@ -0,0 +1,15 @@ +package nu.zoom.dsl.run; + +public class ParserException extends EndgenException { + public ParserException(String message) { + super(message); + } + + public ParserException(String message, Throwable cause) { + super(message, cause); + } + + public ParserException(Throwable cause) { + super(cause); + } +} diff --git a/parser/src/main/java/nu/zoom/dsl/ast/ParserWrapper.java b/parser/src/main/java/nu/zoom/dsl/run/ParserWrapper.java similarity index 93% rename from parser/src/main/java/nu/zoom/dsl/ast/ParserWrapper.java rename to parser/src/main/java/nu/zoom/dsl/run/ParserWrapper.java index 11a95fd..ce62d9c 100644 --- a/parser/src/main/java/nu/zoom/dsl/ast/ParserWrapper.java +++ b/parser/src/main/java/nu/zoom/dsl/run/ParserWrapper.java @@ -11,8 +11,11 @@ // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. -package nu.zoom.dsl.ast; +package nu.zoom.dsl.run; +import nu.zoom.dsl.ast.DocumentNode; +import nu.zoom.dsl.ast.EndpointsVisitorTransformer; +import nu.zoom.dsl.ast.StatesVisitorTransformer; import nu.zoom.dsl.parser.*; import org.antlr.v4.runtime.CharStreams; import org.antlr.v4.runtime.CommonTokenStream; diff --git a/parser/src/main/java/nu/zoom/dsl/run/Runner.java b/parser/src/main/java/nu/zoom/dsl/run/Runner.java new file mode 100644 index 0000000..ebf162b --- /dev/null +++ b/parser/src/main/java/nu/zoom/dsl/run/Runner.java @@ -0,0 +1,96 @@ +package nu.zoom.dsl.run; + +import freemarker.template.TemplateException; +import nu.zoom.dsl.ast.DocumentNode; +import nu.zoom.dsl.freemarker.Generator; + +import java.io.IOException; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.Paths; +import java.util.List; +import java.util.Optional; + +public final class Runner { + public static final String DEFAULT_TEMPLATE_DIRECTORY_NAME = "endpoints-template"; + public static final String DEFAULT_OUTPUT_DIRECTORY_NAME = "endpoints-output"; + + public static void run( + Path dsl, + Optional templates, + Optional output, + Optional maybeParser, + Logger logger + ) throws ValidationException, IOException, GeneratorException { + Path templatesDir = templates.orElse(Paths.get(DEFAULT_TEMPLATE_DIRECTORY_NAME)); + Path outputDir = output.orElse(Paths.get(DEFAULT_OUTPUT_DIRECTORY_NAME)); + + validateOutputDirectory(outputDir); + validateTemplateDirectory(templatesDir); + validateInputFile(dsl); + logger.println("Parsing: " + dsl.toAbsolutePath()); + + final ParserType parser = + maybeParser.orElseGet( + () -> { + if (dsl.getFileName().toString().endsWith(".states")) { + return ParserType.States; + } else { + return ParserType.Endpoints; + } + } + ); + + final DocumentNode rootNode; + if (parser == ParserType.States) { + logger.println("using state grammar."); + rootNode = ParserWrapper.parseStates(dsl); + } else { + logger.println("using endpoints grammar."); + rootNode = ParserWrapper.parseEndpoints(dsl); + } + logger.println("AST: " + rootNode); + logger.println("Generating from templates in: " + templatesDir.toAbsolutePath()); + Generator generator = new Generator(templatesDir, rootNode, outputDir); + List generatedPaths = null; + try { + generatedPaths = generator.generate(); + } catch (TemplateException e) { + throw new GeneratorException(e); + } + if (generatedPaths.isEmpty()) { + System.out.println("No generated paths found."); + } else { + generatedPaths.forEach(p -> logger.println("Generated: " + p.toAbsolutePath())); + } + } + + private static void validateOutputDirectory(Path outputDir) throws IOException, ValidationException { + if (Files.notExists(outputDir)) { + Files.createDirectories(outputDir); + } + if (!Files.isDirectory(outputDir)) { + throw new ValidationException("Output directory: '" + outputDir + " 'is not a directory."); + } + } + + private static void validateTemplateDirectory(Path templateDir) throws ValidationException { + if (!Files.isDirectory(templateDir)) { + throw new ValidationException("Template directory '" + templateDir + "' is not a directory."); + } + } + + private static void validateInputFile(Path file) throws ValidationException { + if (Files.notExists(file)) { + throw new ValidationException("Input file '" + file + "' does not exist."); + } + if (!Files.isReadable(file) || !Files.isRegularFile(file)) { + throw new ValidationException("Input file '" + file + "' is not a readable file."); + } + } + + public enum ParserType { + Endpoints, + States + } +} diff --git a/parser/src/main/java/nu/zoom/dsl/run/StdoutLogger.java b/parser/src/main/java/nu/zoom/dsl/run/StdoutLogger.java new file mode 100644 index 0000000..7b0b7d4 --- /dev/null +++ b/parser/src/main/java/nu/zoom/dsl/run/StdoutLogger.java @@ -0,0 +1,9 @@ +package nu.zoom.dsl.run; + +public class StdoutLogger implements Logger { + + @Override + public void println(String message) { + System.out.println(message); + } +} diff --git a/parser/src/main/java/nu/zoom/dsl/run/ValidationException.java b/parser/src/main/java/nu/zoom/dsl/run/ValidationException.java new file mode 100644 index 0000000..1627d72 --- /dev/null +++ b/parser/src/main/java/nu/zoom/dsl/run/ValidationException.java @@ -0,0 +1,15 @@ +package nu.zoom.dsl.run; + +public class ValidationException extends EndgenException { + public ValidationException(String message) { + super(message); + } + + public ValidationException(String message, Throwable cause) { + super(message, cause); + } + + public ValidationException(Throwable cause) { + super(cause); + } +} diff --git a/pom.xml b/pom.xml index 86744cc..cb060c8 100644 --- a/pom.xml +++ b/pom.xml @@ -65,6 +65,7 @@ parser endgen-dist + endgen-maven-plugin From 790be88cd763f0324f57207a065124c9684022cb Mon Sep 17 00:00:00 2001 From: Johan Maasing Date: Sun, 4 May 2025 15:04:47 +0200 Subject: [PATCH 18/20] Add sample project to test the maven plugin --- endgen-maven-plugin/README.md | 39 +++++++++-- endgen-maven-plugin/pom.xml | 36 +--------- .../java/nu/zoom/dsl/maven/EndgenMojo.java | 16 +++-- sample-maven/pom.xml | 70 +++++++++++++++++++ sample-maven/src/main/endgen/test01.endpoints | 28 ++++++++ sample-maven/src/main/endgen/test01.states | 21 ++++++ .../nu/zoom/dsl/sample/Endpoints.java.ftl | 22 ++++++ 7 files changed, 186 insertions(+), 46 deletions(-) create mode 100644 sample-maven/pom.xml create mode 100644 sample-maven/src/main/endgen/test01.endpoints create mode 100644 sample-maven/src/main/endgen/test01.states create mode 100644 sample-maven/src/main/endpoint-templates/nu/zoom/dsl/sample/Endpoints.java.ftl diff --git a/endgen-maven-plugin/README.md b/endgen-maven-plugin/README.md index 1e02921..3df592d 100644 --- a/endgen-maven-plugin/README.md +++ b/endgen-maven-plugin/README.md @@ -1,8 +1,35 @@ -# References -## Maven +# Configure -https://maven.apache.org/plugins/maven-compiler-plugin/compile-mojo.html#generatedsourcesdirectory +Add the following to your `pom.xml` -### For executing the plugin several times -See executions -https://maven.apache.org/guides/mini/guide-configuring-plugins.html \ No newline at end of file +```xml + + + + nu.zoom.dsl + endgen-maven-plugin + 1.2-SNAPSHOT + + + + endgen + + + ${project.basedir}/src/main/endpoint-templates + ${project.basedir}/src/main/endgen/test01.endpoints + + + + + + +``` + +Replace the `` with the latest published version of the endgen plugin. + +* `templates` should point to the template directory to use. +* `dsl` should be the file to generate code from. +* `output` can be used to specify the directory where the generated files are written. Default is `${project.build.directory}/generated-sources/endgen`. +* `parser` can be used to force the use of either the `Endpoints` or the `States` parser. Default is to determined by looking at the file ending of the dsl-file. + +If you have several DSL-files that you wish to generate from you can repeat the `` block with other configurations. \ No newline at end of file diff --git a/endgen-maven-plugin/pom.xml b/endgen-maven-plugin/pom.xml index e9f42c5..77bfe12 100644 --- a/endgen-maven-plugin/pom.xml +++ b/endgen-maven-plugin/pom.xml @@ -1,5 +1,6 @@ - + 4.0.0 @@ -33,37 +34,4 @@ ${project.parent.version} - - - - - org.apache.maven.plugins - maven-plugin-plugin - ${maven-plugin-tools.version} - - - help-mojo - - helpmojo - - - - - - nu.zoom.dsl - endgen-maven-plugin - 1.2-SNAPSHOT - - - - ${project.build.sourceDirectory}/main/endgen-templates - ${project.build.sourceDirectory}/generated-sources/endgen endpoints-output - ${project.basedir}/../test01.endpoints - - - - - - - \ No newline at end of file diff --git a/endgen-maven-plugin/src/main/java/nu/zoom/dsl/maven/EndgenMojo.java b/endgen-maven-plugin/src/main/java/nu/zoom/dsl/maven/EndgenMojo.java index 8d204aa..757eee5 100644 --- a/endgen-maven-plugin/src/main/java/nu/zoom/dsl/maven/EndgenMojo.java +++ b/endgen-maven-plugin/src/main/java/nu/zoom/dsl/maven/EndgenMojo.java @@ -17,23 +17,27 @@ import java.util.Optional; defaultPhase = LifecyclePhase.GENERATE_SOURCES ) public class EndgenMojo extends AbstractMojo { - @Parameter(defaultValue = "${project.build.sourceDirectory}/main/endgen-templates") + @Parameter( + name = "templates", + defaultValue = "${project.build.sourceDirectory}/main/endgen-templates" + ) File templates; - @Parameter(defaultValue = "${project.build.outputDirectory}/generated-sources/endgen") + @Parameter( + name = "output", + defaultValue = "${project.build.directory}/generated-sources/endgen" + ) File output; - @Parameter + @Parameter(name = "dsl", required = true) File dsl; - @Parameter + @Parameter(name = "parser") String parser; @Override public void execute() throws MojoExecutionException, MojoFailureException { - getLog().info("Running endgen"); - getLog().info("Using dsl: " + dsl); try { Runner.run( optional(dsl).map(File::toPath).orElseThrow(), diff --git a/sample-maven/pom.xml b/sample-maven/pom.xml new file mode 100644 index 0000000..99fdbab --- /dev/null +++ b/sample-maven/pom.xml @@ -0,0 +1,70 @@ + + + + 4.0.0 + + nu.zoom.dsl + sample-maven + 1.0-SNAPSHOT + jar + + + 21 + 21 + UTF-8 + + + + + ASF 2.0 + https://www.apache.org/licenses/LICENSE-2.0 + + + + + + Johan Maasing + johan@zoom.nu + + developer + + + + + + + + nu.zoom.dsl + endgen-maven-plugin + 1.2-SNAPSHOT + + + + endgen + + + ${project.basedir}/src/main/endpoint-templates + ${project.basedir}/src/main/endgen/test01.endpoints + + + + + + + + diff --git a/sample-maven/src/main/endgen/test01.endpoints b/sample-maven/src/main/endgen/test01.endpoints new file mode 100644 index 0000000..6f212ed --- /dev/null +++ b/sample-maven/src/main/endgen/test01.endpoints @@ -0,0 +1,28 @@ +/* + Copyright 2025 "Johan Maasing" + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +*/ + +{ + some: configvalue, + someother: value, + package: se.rutdev.senash +} + +/some/endpoint <- SomeType(foo:String) +Embedded(foo:Bar) +/some/other/endpoint <- (bar:Seq[Embedded]) +/yet/other/endpoint2 <- (bar2:Seq[AType]) -> NamedResponse(foo:Bar) +AType(data: java.util.List) +/yet/other/endpoint3 <- (bar2:Seq[AType]) -> (foo:Bar) diff --git a/sample-maven/src/main/endgen/test01.states b/sample-maven/src/main/endgen/test01.states new file mode 100644 index 0000000..3b9a67a --- /dev/null +++ b/sample-maven/src/main/endgen/test01.states @@ -0,0 +1,21 @@ +/* + Copyright 2025 "Johan Maasing" + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +*/ + +{ title: SomeNodes, package: nu.zoom.dsl.states } + +start(s:S) -> message(foo:foo) -> middle(foo:foo) , +middle -> selfmessage(bar:bar) -> middle(bar:bar), +middle -> message(bar:baz) -> end diff --git a/sample-maven/src/main/endpoint-templates/nu/zoom/dsl/sample/Endpoints.java.ftl b/sample-maven/src/main/endpoint-templates/nu/zoom/dsl/sample/Endpoints.java.ftl new file mode 100644 index 0000000..624c722 --- /dev/null +++ b/sample-maven/src/main/endpoint-templates/nu/zoom/dsl/sample/Endpoints.java.ftl @@ -0,0 +1,22 @@ +// Copyright 2025 "Johan Maasing" +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +<#list meta.templateDirectories>package<#items as dir>${dir}<#sep>.; + +class Endpoints() { +<#list endpoints as endpoint> + /* <#list endpoint.paths.paths><#items as segment>/${segment} */ + public void handle${endpoint.inputType?cap_first} + + +} \ No newline at end of file From 3e5befbb9a1a328917f90f87cc0153241882a9f0 Mon Sep 17 00:00:00 2001 From: Johan Maasing Date: Sun, 4 May 2025 15:05:56 +0200 Subject: [PATCH 19/20] [maven-release-plugin] prepare release v1.2 --- endgen-dist/pom.xml | 2 +- endgen-maven-plugin/pom.xml | 5 ++--- parser/pom.xml | 2 +- pom.xml | 4 ++-- 4 files changed, 6 insertions(+), 7 deletions(-) diff --git a/endgen-dist/pom.xml b/endgen-dist/pom.xml index dda6922..c337f36 100644 --- a/endgen-dist/pom.xml +++ b/endgen-dist/pom.xml @@ -5,7 +5,7 @@ nu.zoom.dsl endgen - 1.2-SNAPSHOT + 1.2 endgen-dist diff --git a/endgen-maven-plugin/pom.xml b/endgen-maven-plugin/pom.xml index 77bfe12..37d2a72 100644 --- a/endgen-maven-plugin/pom.xml +++ b/endgen-maven-plugin/pom.xml @@ -1,12 +1,11 @@ - + 4.0.0 nu.zoom.dsl endgen - 1.2-SNAPSHOT + 1.2 endgen-maven-plugin maven-plugin diff --git a/parser/pom.xml b/parser/pom.xml index ba9d182..2d54120 100644 --- a/parser/pom.xml +++ b/parser/pom.xml @@ -20,7 +20,7 @@ nu.zoom.dsl endgen - 1.2-SNAPSHOT + 1.2 parser diff --git a/pom.xml b/pom.xml index cb060c8..c320d3a 100644 --- a/pom.xml +++ b/pom.xml @@ -19,7 +19,7 @@ nu.zoom.dsl endgen - 1.2-SNAPSHOT + 1.2 pom @@ -59,7 +59,7 @@ scm:git:https://codeberg.org/darkstar/endgen.git scm:git:ssh://git@vcs.zoom.nu:1122/zoom/endgen.git - admin-parent-1.1 + v1.2 From f17ff0c152cad254d879a42670819ecbb817849a Mon Sep 17 00:00:00 2001 From: Johan Maasing Date: Sun, 4 May 2025 15:05:59 +0200 Subject: [PATCH 20/20] [maven-release-plugin] prepare for next development iteration --- endgen-dist/pom.xml | 2 +- endgen-maven-plugin/pom.xml | 2 +- parser/pom.xml | 2 +- pom.xml | 4 ++-- 4 files changed, 5 insertions(+), 5 deletions(-) diff --git a/endgen-dist/pom.xml b/endgen-dist/pom.xml index c337f36..8a2fedb 100644 --- a/endgen-dist/pom.xml +++ b/endgen-dist/pom.xml @@ -5,7 +5,7 @@ nu.zoom.dsl endgen - 1.2 + 1.3-SNAPSHOT endgen-dist diff --git a/endgen-maven-plugin/pom.xml b/endgen-maven-plugin/pom.xml index 37d2a72..9974c25 100644 --- a/endgen-maven-plugin/pom.xml +++ b/endgen-maven-plugin/pom.xml @@ -5,7 +5,7 @@ nu.zoom.dsl endgen - 1.2 + 1.3-SNAPSHOT endgen-maven-plugin maven-plugin diff --git a/parser/pom.xml b/parser/pom.xml index 2d54120..67c6925 100644 --- a/parser/pom.xml +++ b/parser/pom.xml @@ -20,7 +20,7 @@ nu.zoom.dsl endgen - 1.2 + 1.3-SNAPSHOT parser diff --git a/pom.xml b/pom.xml index c320d3a..82a7a48 100644 --- a/pom.xml +++ b/pom.xml @@ -19,7 +19,7 @@ nu.zoom.dsl endgen - 1.2 + 1.3-SNAPSHOT pom @@ -59,7 +59,7 @@ scm:git:https://codeberg.org/darkstar/endgen.git scm:git:ssh://git@vcs.zoom.nu:1122/zoom/endgen.git - v1.2 + admin-parent-1.1