distage-framework

Roles

Many software suites have more than one entrypoint. For example Scalac provides a REPL and a Compiler. A typical distributed application may provide both server and client code, as well as a host of utility applications for maintenance.

In distage such entrypoints are called “roles”. Role-based applications may contain roles of different types:

`RoleService[F]`, for persistent services and daemons
`RoleTask[F]`, for one-off tasks and applications

To use roles, add distage-framework library:

sbt

libraryDependencies += "io.7mind.izumi" %% "distage-framework" % "1.2.17-SNAPSHOT"

To declare roles use RoleModuleDef#makeRole:

import distage.plugins.PluginDef
import izumi.distage.roles.model.definition.RoleModuleDef
import izumi.distage.roles.model.RoleDescriptor
import izumi.distage.roles.model.RoleTask
import izumi.fundamentals.platform.cli.model.raw.RawEntrypointParams
import logstage.LogIO
import zio.UIO

object AppPlugin extends PluginDef {
  include(roleModule)

  def roleModule = new RoleModuleDef {
    makeRole[ExampleRoleTask]
  }
}

class ExampleRoleTask(log: LogIO[UIO]) extends RoleTask[UIO] {
  override def start(roleParameters: RawEntrypointParams, freeArgs: Vector[String]): UIO[Unit] = {
    log.info(s"Running ${ExampleRoleTask.id}!")
  }
}

object ExampleRoleTask extends RoleDescriptor {
  override def id = "exampletask"
}

Create a Role Launcher, RoleAppMain, with the example role:

import distage.plugins.PluginConfig
import izumi.distage.roles.RoleAppMain
import zio.IO

object ExampleLauncher extends RoleAppMain.LauncherBIO[IO] {
  override def pluginConfig = {
    PluginConfig.const(
      // add the plugin with ExampleRoleTask
      AppPlugin
    )
  }
}

You can now specify roles to launch on the command-line:

./launcher :exampletask

ExampleLauncher.main(Array(":exampletask"))

I 2020-12-14T10:00:16.172           (RoleAppPlanner.scala:70)  …oleAppPlanner.Impl.makePlan [ 944:run-main-1          ] phase=late Planning finished. main ops=7, integration ops=0, shared ops=0, runtime ops=17
I 2020-12-14T10:00:16.253        (RoleAppEntrypoint.scala:99)  …AppEntrypoint.Impl.runTasks [ 944:run-main-1          ] phase=late Going to run: tasks=1
I 2020-12-14T10:00:16.259       (RoleAppEntrypoint.scala:104)  …R.I.runTasks.101.loggedTask [ 944:run-main-1          ] phase=late Task is about to start: task=repl.MdocSession$App0$ExampleRoleTask@76b7592c
I 2020-12-14T10:00:16.263           (distage-framework.md:43)  ….App0.ExampleRoleTask.start [ 944:run-main-1          ] Running id=exampletask!
I 2020-12-14T10:00:16.267       (RoleAppEntrypoint.scala:106)  ….R.I.r.1.loggedTask.104.105 [ 944:run-main-1          ] phase=late Task finished: task=repl.MdocSession$App0$ExampleRoleTask@76b7592c

Only the components required by the specified roles will be created, everything else will be pruned. (see: Dependency Pruning)

Inspecting components in REPL

Use .replLocator method of a RoleAppMain to obtain an object graph of the application inspectable in the REPL:

import izumi.functional.bio.UnsafeRun2

val runner = UnsafeRun2.createZIO()
// runner: UnsafeRun2.ZIORunner[Any] = izumi.functional.bio.UnsafeRun2$ZIORunner@175bdc73

val objects = runner.unsafeRun(ExampleLauncher.replLocator(":exampletask"))
// objects: izumi.distage.model.Locator = Locator izumi.distage.LocatorDefaultImpl@5a475f30 with 3 parent(s)
//   with exposed keys:  
//      - {type.Async2[=λ %1:0,%1:1 → ZIO[-Any,+1:0,+1:1]]}
//      - {type.IO2[=λ %0,%1 → ZIO[-Any,+0,+1]]}
//      - {type.SyncSafe1[=λ %0 → ZIO[-Any,+Nothing,+0]]}
//      - {type.LogIO[=λ %1 → ZIO[-Any,+Nothing,+1]]}
//      - {type.MdocSession::MdocApp0::ExampleRoleTask}
//   with confined keys:  ø

// inspecting a component inside the application
val logger = objects.get[LogIO[UIO]]
// logger: LogIO[UIO] = logstage.LogIO$$anon$1@633e68a7

Bundled roles

BundledRolesModule contains two bundled roles:

Help - prints help message when launched ./launcher :help
ConfigWriter - writes reference config into files, split by roles (includes only parts of the config used by the application). Also generates a JSON Schema description for config.

Use include to add it to your application:

import distage.plugins.PluginDef
import izumi.distage.roles.bundled.BundledRolesModule
import zio.Task

object RolesPlugin extends PluginDef {
  include(BundledRolesModule[Task](version = "1.0"))
}

Compile-time checks

To use compile-time checks, add distage-framework library:

sbt

libraryDependencies += "io.7mind.izumi" %% "distage-framework" % "1.2.17-SNAPSHOT"

The easiest way to add full compile-time safety to your application is to add an object inheriting PlanCheck.Main in __ test scope__ of the same module where you define your Role Launcher

import izumi.distage.framework.PlanCheck
import com.example.myapp.MainLauncher

object WiringCheck extends PlanCheck.Main(MainLauncher)

This object will emit compile-time errors for any issues or omissions in your ModuleDefs. It will recompile itself as necessary to provide feedback during development.

By default, all possible roles and activations are checked efficiently.

You may exclude specific roles or activations from the check by passing a PlanCheckConfig case class to PlanCheck API. Most options in PlanCheckConfig can also be set using system properties, see DebugProperties.

Checking default config

By default PlanCheck will check parsing of config bindings (from distage-extension-config) using configs loaded with the same settings as the role launcher.

This allows you to ensure correctness of default configs during development without writing tests.

If you need to disable this check, set checkConfig option of PlanCheckConfig to false.

Using with `distage-teskit`

Use SpecWiring to spawn a test-suite that also triggers compile-time checks.

SpecWiring will check the passed application when compiled, then perform the check again at runtime when ran as a test.

import izumi.distage.framework.PlanCheckConfig
import izumi.distage.testkit.scalatest.SpecWiring
import com.example.myapp.MainLauncher

object WiringCheck extends SpecWiring(
  app = MainLauncher,
  cfg = PlanCheckConfig(checkConfig = false),
  checkAgainAtRuntime = false, // disable the runtime re-check
)

Checking `distage-core` apps

distage-framework’s Role-based applications are checkable out of the box, but applications assembled directly via distage-core’s distage.Injector APIs must implement the CheckableApp trait to provide all the data necessary for the checks. You may use CoreCheckableAppSimple implementation for applications definable by a single collection of modules.

Adding custom checks

You may add your own custom checks that will be executed at compile-time by overriding customCheck method of CheckableApp or its subtypes RoleAppMain or CoreCheckableAppSimple.

For example, you could check that all components that have types with names that contain “Required” – such as make[RequiredInt].from(RequiredInt(5)) – are used by other components and fail otherwise:

import distage.{DIKey, PlanVerifier}
import distage.PlanVerifier.PlanVerifierResult
import izumi.distage.framework.model.PlanCheckInput
import izumi.distage.model.planning.AxisPoint
import izumi.distage.plugins.PluginConfig
import izumi.distage.roles.RoleAppMain
import izumi.fundamentals.collections.nonempty.NESet

object CustomCheckLauncher extends RoleAppMain.LauncherCats[cats.effect.IO] {

  // test that all bindings with types that have 'Required'
  // in their name are used in the application
  override def customCheck(
    planVerifier: PlanVerifier,
    excludedActivations: Set[NESet[AxisPoint]],
    checkConfig: Boolean,
    planCheckInput: PlanCheckInput[AppEffectType],
  ): PlanVerifierResult = {
    val usedKeys = planVerifier.traceReachables(
      planCheckInput.module,
      planCheckInput.roots,
      planCheckInput.providedKeys,
      excludedActivations,
    )
    val requiredKeys = planCheckInput.module.keys.filter {
      // filter out any set elements (to remove weak set elements)
      case _: DIKey.SetElementKey => false
      case anyKey => anyKey.tpe.tag.shortName.contains("Required")
    }
    val unused = requiredKeys -- usedKeys
    if (unused.nonEmpty) {
      throw new RuntimeException(
        s"""Custom check failed, found unused Required bindings:
           |  ${unused.map(_.tpe.tag.repr).mkString(", ")}""".stripMargin
      )
    } else {
      PlanVerifierResult.empty
    }
  }

  override def pluginConfig = PluginConfig.cached(
    packagesEnabled = Seq("com.example.custom")
  )

}

Low-Level APIs

PlanCheckMaterializer, ForcedRecompilationToken and PlanCheck.runtime provide the low-level APIs behind distage compile-time checks, they can be used to wrap the capability in other APIs or implement similar functionality.

PlanVerifier hosts the multi-graph traversal doing the actual checking and can be invoked at runtime or in macro. It can also be invoked using Injector#assert and Injector#verify methods.

Scala 2.12

If you’re using Scala 2.12 and get compilation errors such as

[error]  type mismatch;
[error]  found   : String("mode:test")
[error]  required: izumi.fundamentals.platform.language.literals.LiteralString{type T = String("mode:test")} with izumi.fundamentals.platform.language.literals.LiteralString

Then you’ll have to refactor your instance of PlanCheck.Main (or similar) to make sure that PlanCheckConfig is defined in a separate val. You may do this by moving it from a constructor parameter to an early initializer.

Example:

object WiringTest extends PlanCheck.Main(
  MyApp,
  PlanCheckConfig(...)
)
// [error]

Fix:

object WiringTest extends {
  val config = PlanCheckConfig(... )
} with PlanCheck.Main(MyApp, config)

Note that this issue does not exist on Scala 2.13+, it is caused by a bug in Scala 2.12’s treatment of implicits in class parameter position.

Typesafe Config

distage-extension-config library allows parsing case classes and sealed traits from typesafe-config configuration files.

To use it, add the distage-extension-config library:

sbt

libraryDependencies += "io.7mind.izumi" %% "distage-extension-config" % "1.2.17-SNAPSHOT"

Use helper functions in ConfigModuleDef, makeConfig and make[_].fromConfig to parse the bound AppConfig instance into case classes, sealed traits or literals:

import distage.{Id, Injector}
import distage.config.{AppConfig, ConfigModuleDef}
import com.typesafe.config.ConfigFactory

final case class Conf(
  name: String,
  age: Int,
)

final case class OtherConf(
  other: Boolean
)

final class ConfigPrinter(
  conf: Conf,
  otherConf: OtherConf @Id("other"),
) {
  def print() = {
    println(s"name: ${conf.name}, age: ${conf.age}, other: ${otherConf.other}")
  }
}

val module = new ConfigModuleDef {
  make[ConfigPrinter]

  // declare paths to parse
  makeConfig[Conf]("conf")
  makeConfig[OtherConf]("conf").named("other")

  // add config instance
  make[AppConfig].from(AppConfig.provided(
    ConfigFactory.parseString(
      """conf {
        |  name = "John"
        |  age = 33
        |  other = true
        |}""".stripMargin
    )
  ))
}
// module: AnyRef with ConfigModuleDef = 
// make[{type.repl.MdocSession.MdocApp5.ConfigPrinter}].from(call(π:Constructor(repl.MdocSession::MdocApp5::Conf, repl.MdocSession::MdocApp5::OtherConf): repl.MdocSession::MdocApp5::ConfigPrinter)) (BindingOrigin((distage-framework.md:253)))
// make[{type.repl.MdocSession.MdocApp5.Conf}].from(call(ƒ:izumi.distage.config.codec.AbstractDIConfigReader$$Lambda/0x00007fff83671ae8@4f182f18(izumi.distage.config.model.AppConfig): repl.MdocSession::MdocApp5::Conf)).tagged(Set(ConfTag(conf,izumi.distage.config.codec.AbstractDIConfigReader$$Lambda/0x00007fff83671ae8@4f182f18,Class repl.MdocSession.MdocApp5.Conf with fields   
//    - name: TBasic(TString)
//    - age: TBasic(TInt)))) (BindingOrigin((distage-framework.md:256)))
// make[{type.repl.MdocSession.MdocApp5.OtherConf@other}].from(call(ƒ:izumi.distage.config.codec.AbstractDIConfigReader$$Lambda/0x00007fff83671ae8@51a9054f(izumi.distage.config.model.AppConfig): repl.MdocSession::MdocApp5::OtherConf)).tagged(Set(ConfTag(conf,izumi.distage.config.codec.AbstractDIConfigReader$$Lambda/0x00007fff83671ae8@51a9054f,Class repl.MdocSession.MdocApp5.OtherConf with fields   
//    - other: TBasic(TBoolean)))) (BindingOrigin((distage-framework.md:257)))
// make[{type.izumi.distage.config.model.AppConfig}].from(call(ƒ:<function0>(): izumi.distage.config.model.AppConfig)) (BindingOrigin((distage-framework.md:260)))

Injector().produceRun(module) {
  (configPrinter: ConfigPrinter) =>
    configPrinter.print()
}
// name: John, age: 33, other: true
// res6: distage.package.Identity[Unit] = ()

Automatic derivation of config codecs is based on pureconfig-magnolia.

When a given type already has custom pureconfig.ConfigReader instances in implicit scope, they will be used, otherwise they will be derived automatically.

You don’t have to explicitly make[AppConfig] in distage-testkit’s tests and in distage-framework’s Roles, unless you want to override default behavior.

By default, tests and roles will try to read the configurations from resources with the following names, in order:

${roleName}.conf
${roleName}-reference.conf
${roleName}-reference-dev.conf
application.conf
application-reference.conf
application-reference-dev.conf
common.conf
common-reference.conf
common-reference-dev.conf

Where distage-testkit uses `TestConfig#configBaseName` instead of roleName.

Explicit config files passed to the role launcher -c file.conf the command-line flag have a higher priority than resource configs.

Role-specific configs on the command-line, passed via -c file.conf option after a :roleName argument, in turn have a higher priority than the explicit config files passed before the first :role argument.

Example:

  ./launcher -c global.conf :role1 -c role1.conf :role2 -c role2.conf

Here configs will be loaded in the following order, with higher priority earlier and earlier configs overriding the values in later configs:

explicits: role1.conf, role2.conf, global.conf,
resources: role1[-reference,-dev].conf, role2[-reference,-dev].conf, ,application[-reference,-dev].conf, common[-reference,-dev].conf

Plugins

distage-extension-plugins module adds classpath discovery for modules that inherit a marker trait PluginBase.

Plugins reduce friction in adding new components, a programmer needs only to define new plugins and does not also have to stop to add a new plugin to a central wiring point. Plugins also enable extreme late-binding: they allow a program to extend itself at launch time with new Plugin classes on the classpath.

Plugins are compatible with compile-time checks as long as they’re defined in a separate module.

To use plugins, add the distage-extension-plugins library:

sbt

libraryDependencies += "io.7mind.izumi" %% "distage-extension-plugins" % "1.2.17-SNAPSHOT"

To declare a plugin, create a top-level object extending the PluginDef trait instead of ModuleDef:

package com.example.petstore

import distage.Injector
import distage.plugins.{PluginConfig, PluginDef, PluginLoader}

object PetStorePlugin extends PluginDef {
  make[PetRepository]
  make[PetStoreService]
  make[PetStoreController]
}

Use PluginLoader to find all the PluginDef objects in a specific package or its subpackages:

val pluginConfig = PluginConfig.cached(
  // packages to scan
  packagesEnabled = Seq("com.example.petstore")
)
// pluginConfig: PluginConfig = PluginConfig(List(com.example.petstore),List(),true,false,List(),List())

val appModules = PluginLoader().load(pluginConfig)
// appModules: izumi.distage.plugins.load.LoadedPlugins = LoadedPlugins(List(
// make[{type.com.example.petstore.PetRepository}].from(call(π:Constructor(): com.example.petstore.PetRepository)) (BindingOrigin((PetStorePlugin.scala:6)))
// make[{type.com.example.petstore.PetStoreController}].from(call(π:Constructor(): com.example.petstore.PetStoreController)) (BindingOrigin((PetStorePlugin.scala:8)))
// make[{type.com.example.petstore.PetStoreService}].from(call(π:Constructor(): com.example.petstore.PetStoreService)) (BindingOrigin((PetStorePlugin.scala:7)))),List(),List())

You may then pass plugins to Injector as ordinary modules:

// combine all plugins into one

val appModule = appModules.result.merge
// appModule: izumi.distage.model.definition.ModuleBase = 
// make[{type.com.example.petstore.PetRepository}].from(call(π:Constructor(): com.example.petstore.PetRepository)) (BindingOrigin((PetStorePlugin.scala:6)))
// make[{type.com.example.petstore.PetStoreService}].from(call(π:Constructor(): com.example.petstore.PetStoreService)) (BindingOrigin((PetStorePlugin.scala:7)))
// make[{type.com.example.petstore.PetStoreController}].from(call(π:Constructor(): com.example.petstore.PetStoreController)) (BindingOrigin((PetStorePlugin.scala:8)))

// launch

Injector()
  .produceGet[PetStoreController](appModule)
  .use(_.run())
// PetStoreController: running!
// res10: distage.package.Identity[Unit] = ()

Compile-time scanning

Plugin scan can be performed at compile-time, this is mainly useful for deployment on platforms with reduced runtime reflection capabilities compared to the JVM, such as Graal Native Image, Scala.js and Scala Native. Use PluginConfig.compileTime to perform a compile-time scan.

Be warned though, for compile-time scanning to find plugins, they must be placed in a separate module from the one in which scan is performed. When placed in the same module, scanning will fail.

Example:

package com.example.petstore.another.module

val pluginConfig = PluginConfig.compileTime("com.example.petstore")
// pluginConfig: PluginConfig = PluginConfig(
//   packagesEnabled = List(),
//   packagesDisabled = List(),
//   cachePackages = false,
//   debug = false,
//   merges = List(
//     
// make[{type.com.example.petstore.PetRepository}].from(call(π:Constructor(): com.example.petstore.PetRepository)) (BindingOrigin((PetStorePlugin.scala:6)))
// make[{type.com.example.petstore.PetStoreController}].from(call(π:Constructor(): com.example.petstore.PetStoreController)) (BindingOrigin((PetStorePlugin.scala:8)))
// make[{type.com.example.petstore.PetStoreService}].from(call(π:Constructor(): com.example.petstore.PetStoreService)) (BindingOrigin((PetStorePlugin.scala:7)))
//   ),
//   overrides = List()
// )

val loadedPlugins = PluginLoader().load(pluginConfig)
// loadedPlugins: izumi.distage.plugins.load.LoadedPlugins = LoadedPlugins(
//   loaded = List(),
//   merges = List(
//     
// make[{type.com.example.petstore.PetRepository}].from(call(π:Constructor(): com.example.petstore.PetRepository)) (BindingOrigin((PetStorePlugin.scala:6)))
// make[{type.com.example.petstore.PetStoreController}].from(call(π:Constructor(): com.example.petstore.PetStoreController)) (BindingOrigin((PetStorePlugin.scala:8)))
// make[{type.com.example.petstore.PetStoreService}].from(call(π:Constructor(): com.example.petstore.PetStoreService)) (BindingOrigin((PetStorePlugin.scala:7)))
//   ),
//   overrides = List()
// )

Edit this page

1.2.17*

distage-framework

Roles

Inspecting components in REPL

Bundled roles

Compile-time checks

Checking default config

Using with distage-teskit

Checking distage-core apps

Adding custom checks

Low-Level APIs

Scala 2.12

Typesafe Config

Plugins

Compile-time scanning

Using with `distage-teskit`

Checking `distage-core` apps