Layer 3

A Layer3 structure separates responsibilities that are often combined in a simpler Schema/Model design:

The Domain API Developer defines an API layer of abstract consumer-facing model contracts. These usually expose the relevant values through Maps and collections and are technical rather than deployment-specific.
The Schema Developer provides the actual Schema classes, usually specific subclasses of API-layer types. They add named fields, lifecycle behavior, and validation so concrete modeling is expressive and domain-specific.
The Model Writer creates configured Model instances through scripts or structured inputs and connects them.
The Client Developer consumes the API layer without depending on the concrete Schema layer.

The role boundaries and variants of this approach need a dedicated design pass under issue #454. The established definition is the API–Schema–Model dependency pattern; it is not a package or Java-module boundary.

Example structure

Let's consider an infrastructure model. We have an application that consists of a database, and a number of microservices. This application will be deployed in a number of environments. An environment is thus a collection of related applications that are deployed together. Actual instances of an environment represent different stages for deployment, e.g. dev, test, prod.

In this structure, the API layer will provide the following classes:

Environment
Application
Database
Microservice

These are the classes that will be consumed by our Consumer application (for example, a deployment pipeline).

The schema layer contains classes modeling the actual applications, i.e., if we have two applications, each application will consist of a database class and several microservice classes.

@DSL class CustomerServiceEnvironment extends Environment {
    Shipping shipping
    Billing billing
}

// First Application: Shipping
@DSL class Shipping extends Application {
    ShippingDatabase database
    ShippingFrontend frontend
    ShippingBackend backend
    ShippingWorker worker 
}

@DSL class ShippingDatabase extends Database {
    @Required DbUser ddl
    @Required DbUser dml
    DbUser monitoring
}

// Second Application: Billing
@DSL class Billing extends Application {
    BillingDatabase database
    BillingService service
}

Now without going into much detail, a dsl-model (using KlumAST) could be something like this:

environment("dev") {
    shipping {
        database {
            ddl "admin"
            dml "shipping_user"
            monitoring "monitoring"
        }
        frontend {
            replicas 1
            ssl false
            //...
        }
        // ...
    }
    billing {
        database {
            ddl "admin"
            dml "billing_user"
        }
        service {
            //...
        }
    }
}
environment("prod") {
    shipping {
        database {
            ddl "xcvzh"
            dml "abcde"
            monitoring "mon_x"
        }
        frontend {
            replicas 3
            ssl true
            //...
        }
        // ...
    }
    //...
}

From the modeling perspective, this is a lot more expressive than using generic microservice or database classes. However, the API layer is still very simple and can be used by the consumer application without having to know about the actual structure of the application.

For each Cluster-Field of a class, a cluster factory named like the field is created, which only contains the matching fields of the cluster. This is especially useful if the name of the field lacks context:

environment("dev") {
  applications { // cluster factory for field "Environment.applications" 
      shipping {
          database {
              users { // cluster factory for field "Database.users"
                  ddl "admin"
                  dml "shipping_user"
                  monitoring "monitoring"
              }
          }
          frontend {
              replicas 1
              ssl false
              //...
          }
          // ...
      }
      billing {
          database {
            users { // cluster factory for field "Database.users"
                ddl "admin"
                dml "billing_user"
            }
          }
          service {
              //...
          }
      }
  }
}

By default, these factories are entirely optional (like collection factories). Using @Cluster.bounded, which can also be placed on a class, one of its superclasses, or a package, makes the cluster field methods on the generated Builder construction API protected. They are then reachable only inside the factory; for example, code completion presents users on a Database Builder rather than its ddl or dml members.

The Environment base class contains method to access the actual applications as a Map:

@DSL
abstract class Environment {
    @Key String name
    
    @Cluster Map<String, Application> applications 
}

That way a deployer service can simply iterate over the applications of our CustomerServiceEnvironment and deploy them.

def deploy(Environment env) {
    env.applications.each { name, app ->
        log.info "Deploying $name"
        deployApplication(app)
    }
}

@Cluster can also be placed on a getter method (for example, getApplications()), which can be abstract or have an empty/null body to satisfy an IDE. Prefer the field form for new Schemas. The roles, dependency direction, and variants of Layer 3 remain under the explicit terminology review in #454.

Validations in our ShippingApplication can also be done specifically for that application:

@Validate void SslNeedsValidationServer() {
    if (frontend.ssl && backend.validationServer == null)
        error "Backend must define validation server if SSL is enabled"
}

Implementation

Using the @Cluster annotation, this method will automatically be implemented using the respective methods of the ClusterModel helper class.

For example, the getApplications method is implemented like this:

Map<String, Application> getApplications() {
    ClusterModel.getPropertiesOfType(this, Application)
}

If the annotated method return Map<String, Collection<X>>, ClusterModel.getCollectionsOfType will be used instead.

Most ClusterModel methods have an additional parameter to filter the return values, which is usually one of the following:

A Predicate<AnnotatedElement>
A Closure<Boolean>, which accepts an AnnotatedElement as parameter
An Annotation class (which is a shortcut for it -> it.isAnnotationPresent(filter))

The most common usage is the last one, simply filtering on the presence of an annotation on the fields. This can also be implemented using the value field of the @Cluster annotation:

@Cluster(Important) Map<String, Application> applications

will be converted to

Map<String, Application> getApplications() {
    return ClusterModel.getPropertiesOfType(this, Application, Important)
}

AutoCreate

Any AutoCreate annotation placed on the cluster field will be used to automatically create all targeted field's objects during the auto-create phase, thus effectively working as if the annotation was placed on all fields of the cluster.

Benefits of a Layer3 model

There are various major benefits of using a Layer3 model vs. a generic schema/model approach:

Editing and code completion

With each application being a specific subclass of Application, the actual model gets more concise, and more domain specific. Consider the (partial) example above being modeled using a generic schema/model approach:

environment("dev") {
    application("shipping") {
        database {
            user("ddl") { "admin" }
            user("dml") { "shipping_user" }
            user("monitoring") { "monitoring" }
        }
        service("frontend") {
            replicas 1
            ssl false
            //...
        }
        // ...
    }
    application("billing") { 
    // ...

Besides being harder to read there is neither code completion help nor any protection against typos. The developer needs to know exactly which microservices the application consists of and which database users are needed.

In contrast, by using a specific ShippingApplication class, there is exactly one field for each microservice, and the developer can use code completion to see which fields are available. Also, typos like using the wrong user will be detected by the compiler and the IDE immediately.

Using a specific subclass also allows properly commenting the domain-specific fields (what is the use of the monitoring db user?), which is not possible with a generic schema/model approach.

Domain consumers

Since we are building an environment model in this example, there are two distinct types of consumers:

Generic consumers, like a deployment pipeline or a test framework, only use the generic methods of the API layer (behaving exactly like in the generic schema/model approach)
Specific consumers know the actual ShippingApplication and can access its various fields directly. Specific consumers can be, for example:
- The application itself, for example, in reading the jdbc url for the actual database directly from the model (instead of an application.yaml or such)
- A post-deployment test that runs against a specific environment and needs to know the actual database users and passwords can obtain them directly from the model. Combined with password retrieving techniques like an Hashicorp Vault accessor, this can be a very powerful approach. Since the tests are identical for every stage, they can be effectively reused. A developer can use the same tests against a local virtual machine as against the actual approval environment. The only difference being that the developer would not have the rights to access the actual passwords of the approval environment.

Validation

With ShipmentApplication being a class with domain knowledge, it can also contain domain-specific validations. For example:

if ssl is enabled in the frontend, the backend must have a configured validation server
if a monitoring service is defined, the monitoring database user must be defined

Making these validations with a domain schema is trivial.

Automatic creation and linking

Let's say that a monitoring microservice is used by multiple applications in the environment. In the generic schema/model approach, the monitoring service would be defined multiple times, once for each application. This is not only redundant but also error-prone, since the monitoring service might be configured differently for each application.

Using the schema layer with @AutoCreate, the monitoring service could automatically be created.

@DSL
abstract class MonitoredApplication extends Application {
  @AutoCreate
  MonitoringService monitoring
}

Now, our monitoring service needs access to a database, but we want to reuse the database for the application. So we link the database field of the monitoring service to the database of its owner:

class MonitoringService extends Microservice {
  @Owner MonitoredApplication application
  @LinkTo Database database
}

During the instantiation of the model, the database field will be automatically filled but can still be overwritten on instance level. @LinkTo now selects FieldType.OPTIONAL_LINK: a locally created same-session value is owned composition, while the Auto-Link fallback and any completed value are aggregation references. For an aggregation-only relationship, declare @Field(FieldType.LINK) @LinkTo; a normal unannotated relationship remains composition-only. See OptionalLinkRelationshipTest.optional relationships retain local composition and aggregation identity for single List and Map entries for the executable example.

Role fields

Fields can be annotated with @Role to indicate that they are used for a specific role as seen from their owner. Consider a Database class that has various users. Each user object has access to its owning database object, but it might be necessary for the User object to know how it is used in their database. Rather than forcing the modeler to set the role manually, it can simply be inferred from the field name of the Database that points to the user:

@DSL
class MyDatabase extends Database {
  String url

  DbUser ddl
  DbUser dml
  DbUser monitoring
}

@DSL
class DbUser {
  @Owner Database database
  @Role String role
  @Key String id
}

def db = MyDatabase.Create.With {
  url "jdbc:..."
  ddl("user1")
  dml("user2")
  monitoring("user3")
}

assert db.ddl.role == "ddl"
assert db.dml.role == "dml"
assert db.monitoring.role == "monitoring"

That way some kind of environment checker can, for example, use Completed Object Support to validate that all non ddl users have the correct privileges:

KlumObjectSupport.of(model).getStructure().findAll(DbUser).each { path, user ->
    if (user.role != "ddl")
      assertNoDdlPrivileges(user, path)

}

Note that this check should not be done during standard model validation because it requires access to the actual database.