Zombienet¶
Introduction¶
Zombienet is a testing framework designed for Polkadot SDK-based blockchains. It provides a simple CLI tool for creating and testing blockchain environments locally or across networks. This allows developers to easily run and interact with blockchain nodes in a controlled environment. Zombienet is a JavaScript library designed to run on Node.js and supports various backend providers, including Kubernetes, Podman, and local setups for running blockchain nodes.
The framework enables developers to create tests using natural language tools to verify on-chain storage, metrics, logs, and custom interactions with the blockchain. It is particularly effective for setting up local relaychains with validators and parachains with collators.
Parity Technologies has designed and developed this framework, now maintained by the Zombienet team. For further support and information, refer to the following contact points:
Install Zombienet¶
Zombienet releases are available on the Zombienet repository.
In order to install Zombienet, there are multiple options available, depending on the user's preferences and the environment where it will be used. The following section will guide you through the installation process for each of the available options.
Zombienet executables can be downloaded using the latest release uploaded on the Zombienet repository. You can download the executable for your operating system and architecture and then move it to a directory in your PATH. Each release includes executables for Linux and macOS, which are generated using pkg. This allows the Zombienet CLI to operate without requiring Node.js to be installed.
Alternatively, you can also download the executable using either curl or wget:
Note
Ensure to replace the URL with the <INSERT_ZOMBIENET_VERSION> that you want to download, as well as the <INSERT_ZOMBIENET_EXECUTABLE> with the name of the executable file that matches your operating system and architecture. This guide uses v1.3.106 and zombienet-macos-arm64.
Note
This documentation explains the functionality of Chopsticks version 0.13.1. Make sure you're using the correct version to match these instructions.
Then, ensure the downloaded file is executable:
Finally, you can run the following command to check if the installation was successful. If so, it will display the version of the installed Zombienet:
If you want to add the zombienet executable to your PATH, you can move it to a directory in your PATH, such as /usr/local/bin:
So then, you can refer to the zombienet executable directly:
So then, you can refer to the zombienet executable directly:
For Nix users, the Zombienet repository provides a flake.nix file that can be used to install Zombienet. This means that users can easily incorporate Zombienet into their Nix-based projects.
To install Zombienet utilizing Nix, users can run the following command, triggering the fetching of the flake and subsequently installing the Zombienet package:
nix run github:paritytech/zombienet/INSERT_ZOMBIENET_VERSION -- \
spawn INSERT_ZOMBIENET_CONFIG_FILE_NAME.toml
Note
Ensure to replace the INSERT_ZOMBIENET_VERSION with the desired version of Zombienet. Also, replace the INSERT_ZOMBIENET_CONFIG_FILE_NAME with the name of the configuration file you want to use.
To run the command above, you need to have Flakes enabled.
Alternatively, you can also include the Zombienet binary in the PATH for the current shell. This can be achieved by:
Zombienet can also be run using Docker. The Zombienet repository provides a Docker image that can be used to run the Zombienet CLI. To run Zombienet using Docker, you can use the following command:
The command above will run the Zombienet CLI inside a Docker container and mount the current directory to the/home/nonroot/zombie-net/host-current-files directory inside the container. This allows Zombienet to access the configuration file and other files in the current directory. If you want to mount a different directory, replace $(pwd) with the desired directory path.
Inside the Docker container, you can run the Zombienet CLI commands. First, you need to set up Zombienet downloading the necessary binaries:
After that, you need to add those binaries to the PATH:
Finally, you can run the Zombienet CLI commands. For example, to spawn a network using a specific configuration file, you can run the following command:
The command above mounts the current directory to the /workspace directory inside the Docker container. This allows Zombienet to access the configuration file and other files in the current directory. If you want to mount a different directory, replace $(pwd) with the desired directory path.
Providers¶
Zombienet supports different backend providers for running the nodes. At this moment, Kubernetes, Podman, and local are supported, which can be declared as kubernetes, podman, or native, respectively.
To use a particular provider, you can specify it in the network file or use the --provider flag in the CLI:
Alternatively, you can set the provider in the network file:
At the moment, Zombienet supports the following providers: kubernetes, podman, and native.
It's important to note that each provider has specific requirements and associated features. The subsequent sections will guide you through the installation process for each provider and the requirements and features each provider offers.
Kubernetes¶
Requirements¶
Zombienet is designed to be compatible with a variety of Kubernetes clusters, including Google Kubernetes Engine (GKE), Docker Desktop, and kind. To effectively interact with your cluster, you'll need to ensure that kubectl is installed on your system, which is the Kubernetes command-line tool that allows you to run commands against Kubernetes clusters. If you don't have kubectl installed, you can follow the instructions provided on the Kubernetes website.
Moreover, in order to create resources such as namespaces, pods, and CronJobs within the target cluster, you must have the appropriate permissions granted to your user or service account. These permissions are essential for managing and deploying applications effectively within Kubernetes.
Features¶
In Kubernetes, Zombienet uses the Prometheus operator (if available) to oversee monitoring and visibility. This configuration ensures that only essential networking-related pods are deployed. Using the Prometheus operator, Zombienet improves its ability to efficiently monitor and manage network activities within the Kubernetes cluster.
Podman¶
Requirements¶
Zombienet supports Podman rootless as a provider. To use Podman as a provider, you need to have Podman installed on your system. Podman is a daemonless container engine for developing, managing, and running Open Container Initiative (OCI) containers and container images on Linux-based systems. You can install Podman by following the instructions provided on the Podman website.
Warning
Currently, Podman can only be used with Zombienet on Linux machines. Although Podman has support for macOS through an internal VM, the Zombienet provider code requires Podman to run natively on Linux.
Features¶
Using Podman, Zombienet deploys additional pods to enhance the monitoring and visibility of the active network. Specifically, pods for Prometheus, Tempo, and Grafana are included in the deployment. Grafana is configured with Prometheus and Tempo as data sources.
Upon launching Zombienet, access to these monitoring services is facilitated through specific URLs provided in the output:
- Prometheus - http://127.0.0.1:34123
- Tempo - http://127.0.0.1:34125
- Grafana - http://127.0.0.1:41461
It's important to note that Grafana is deployed with default administrator access.
Note
When network operations cease —either by halting a running spawn with Ctrl+C or upon completion of the test— Zombienet automatically removes all associated pods.
Local¶
Requirements¶
The Zombienet local provider, also referred to as native, enables you to run nodes as local processes in your environment. You must have the necessary binaries for your network (such as polkadot and polkadot-parachain). These binaries should be available in your PATH, allowing Zombienet to spawn the nodes as local processes.
To install the necessary binaries, you can use the Zombienet CLI command:
This command will download and prepare the necessary binaries for Zombienet’s use.
Warning
The polkadot and polkadot-parachain binaries releases are not compatible with macOS. As a result, macOS users will need to clone the Polkadot repository, build the Polkadot binary, and manually add it to their PATH for polkadot and polkadot-parachain to work.
If you need to use a custom binary, ensure the binary is available in your PATH. You can also specify the binary path in the network configuration file. To showcase this, this guide will use the custom OpenZeppelin template as an example.
First, clone the OpenZeppelin template repository:
git clone https://github.com/OpenZeppelin/polkadot-runtime-templates \
&& cd polkadot-runtime-templates/generic-template
Then, build the custom binary:
```bash
cargo build --release
After that, add the custom binary to your PATH:
Alternatively, you can specify the binary path in the network configuration file:
[relaychain]
chain = "rococo-local"
default_command = "./bin-v1.6.0/polkadot"
[parachain]
id = 1000
[parachain.collators]
name = "collator01"
command = "./target/release/parachain-template-node"
Note
The local provider exclusively utilizes the command configuration for nodes/collators, which supports both relative and absolute paths. You can employ the default_command configuration to specify the binary for spawning all nodes in the relay chain.
Features¶
Currently, the local provider does not execute any additional layers or processes.
CLI Usage¶
Zombienet provides a CLI that allows interaction with the tool. The CLI can receive commands and flags to perform different kinds of operations. These operations can be initiated using the following syntax:
The following sections will guide you through the primary usage of the Zombienet CLI and the available commands and flags.
CLI Commands¶
spawn - spawn the network defined in the configuration file
<networkConfig>- a file that declares the desired network to be spawned in.tomlor.jsonformat. For further information, check out the Configuration Files section
Warning
For the spawn command to work on macOS, users need to be aware that the Polkadot binary is currently not compatible with macOS. As a result, macOS users will need to clone the Polkadot repository, build Polkadot binary, and manually add it to their PATH.
test - run test on the network spawned
<testFile>- a file that defines assertions and tests against the spawned network, using natural language expressions to evaluate metrics, logs, and built-in functions
setup - set up the Zombienet development environment
<binaries>- executables that will be downloaded and prepared to be used by Zombienet. Options:polkadot,polkadot-parachain
convert - transforms a (now deprecated) polkadot-launch configuration file to a Zombienet configuration file
<filePath>- path to a polkadot-launch configuration file with a.jsor.jsonextension defined by theLaunchConfiginterface
version - prints Zombienet version
None
help - prints help information
None
CLI Flags¶
You can use the following flags to customize the behavior of the CLI:
-p, --provider - override provider to use. Defaults to kubernetes
<provider>- the provider to use. Options:podman,kubernetes,native
-d, --dir - directory path for placing the network files instead of random temp one
<path>- desired path for network files
zombienet -d /home/user/my-zombienet
-f, --force - force override all prompt commands
None
-l, --logType - type of logging on the console. Defaults to table
<logType>desired type of logging. Options:table,text,silent
-m, --monitor - start as monitor, do not auto clean up network
None
-c, --spawn-concurrency - number of concurrent spawning processes to launch. Defaults to 1
<concurrency>- desired quantity of processes
-h, --help - display help for command
None
Configuration Files¶
The network configuration can be given in either JSON or TOML format. The Zombienet repository also provides a folder with some examples of configuration files that can be used as a reference.
Note
Each section may include provider-specific keys that are not recognized by other providers. For example, if you use the local provider, any references to images for nodes will be disregarded.
Settings¶
Through the keyword settings, it's possible to define the general settings for the network. The available keys are:
global_volumes?GlobalVolume[] - a list of global volumes to use. TheGlobalVolumeinterface is defined as follows:bootnodeboolean - add bootnode to network. Default istruebootnode_domain?string - domain to use for bootnodetimeoutnumber - global timeout to use for spawning the whole network"node_spawn_timeout?number - timeout to spawn pod/processgrafana?boolean - deploy an instance of Grafanaprometheus?boolean - deploy an instance of Prometheustelemetry?boolean - enable telemetry for the networkjaeger_agent?string - the Jaeger agent endpoint passed to the nodes. Only available on Kubernetestracing_collator_url?string - the URL of the tracing collator used to query by the tracing assertion. Should be tempo query compatibletracing_collator_service_name?string - service name for tempo query frontend. Only available on Kubernetes. Defaults totempo-tempo-distributed-query-frontendtracing_collator_service_namespace?string - namespace where tempo is running. Only available on Kubernetes. Defaults totempotracing_collator_service_port?number - port of the query instance of tempo. Only available on Kubernetes. Defaults to3100enable_tracing?boolean - enable the tracing system. Only available on Kubernetes. Defaults totrueproviderstring - provider to use. Default iskubernetes"polkadot_introspector?boolean - deploy an instance of polkadot-introspector. Only available on Podman and Kubernetes. Defaults tofalsebackchannel?boolean - deploy an instance of backchannel server. Only available on Kubernetes. Defaults tofalseimage_pull_policy?string - image pull policy to use in the network. Possible values areAlways,IfNotPresent, andNeverlocal_ip?string - IP used for exposing local services (rpc/metrics/monitors). Defaults to"127.0.0.1"global_delay_network_global_settings?number - delay in seconds to apply to the networknode_verifier?string - specify how to verify node readiness or deactivate by usingNone. Possible values areNoneandMetric. Defaults toMetric
For example, the following configuration file defines a minimal example for the settings:
Relay Chain Configuration¶
You can use the relaychain keyword to define further parameters for the relay chain at start-up. The available keys are:
default_command?string - the default command to run. Defaults topolkadotdefault_image?string - the default Docker image to usedefault_resources?Resources - represents the resources limits/reservations needed by the nodes by default. Only available on Kubernetes. TheResourcesinterface is defined as follows:default_db_snapshot?string - the default database snapshot to usedefault_prometheus_prefixstring - a parameter for customizing the metric's prefix. Defaults tosubstratedefault_substrate_cli_args_version?SubstrateCliArgsVersion - set the Substrate CLI arguments version. TheSubstrateCliArgsVersionenum is defined as follows:default_keystore_key_types?string[] - defines which keystore keys should be createdchainstring - the chain namechain_spec_path?string - path to the chain spec file. Should be the plain version to allow customizationschain_spec_command?string - command to generate the chain spec. It can't be used in combination withchain_spec_pathdefault_args?string[] - an array of arguments to use as default to pass to the commanddefault_overrides?Override[] - an array of overrides to upload to the node. TheOverrideinterface is defined as follows:random_nominators_count?number - if set and the stacking pallet is enabled, Zombienet will generate the input quantity of nominators and inject them into the genesismax_nominationsnumber - the max number of nominations allowed by a nominator. Should match the value set in the runtime. Defaults to24nodes?Node[] - an array of nodes to spawn. It is further defined on the Node Configuration sectionnode_groups?NodeGroup[] - an array of node groups to spawn. It is further defined on the Node Group Configuration sectiontotal_node_in_group?number - the total number of nodes in the group. Defaults to1genesisJSON - the genesis configurationdefault_delay_network_settings?DelayNetworkSettings - sets the expected configuration to delay the network. TheDelayNetworkSettingsinterface is defined as follows:
Node Configuration¶
There is one specific key capable of receiving more subkeys: the nodes key. This key is used to define further parameters for the nodes. The available keys:
namestring - name of the node. Any whitespace will be replaced with a dash (e.g.,new alicewill be converted tonew-alice)image?string - override default Docker image to use for this nodecommand?string - override default command to runcommand_with_args?string - override default command and argumentsargs?string[] - arguments to be passed to the commandenv?envVars[] - environment variables to set in the container. TheenvVarsinterface is defined as follows:overrides?Override[] - array of overrides definitions. TheOverrideinterface is defined as follows:prometheus_prefix?string - customizes the metric's prefix for the specific node. Defaults tosubstratedb_snapshot?string - database snapshot to usesubstrate_cli_args_version?SubstrateCliArgsVersion - set the Substrate CLI arguments version directly to skip binary evaluation overhead. TheSubstrateCliArgsVersionenum is defined as follows:resources?Resources - represent the resources limits/reservations needed by the node. TheResourcesinterface is defined as follows:keystore_key_types?string[] - defines which keystore keys should be createdvalidatorboolean - pass the--validatorflag to the command. Defaults totrueinvulnerableboolean - if true, add the node to invulnerables in the chain spec. Defaults tofalsebalancenumber - balance to set in balances for node's account. Defaults to2000000000000bootnodes?string[] - array of bootnodes to useadd_to_bootnodes?boolean - add this node to the bootnode list. Defaults tofalsews_port?number - WS port to userpc_port?number - RPC port to useprometheus_port?number - Prometheus port to usep2p_cert_hash?string - libp2p certhash to use with webRTC transportdelay_network_settings?DelayNetworkSettings - sets the expected configuration to delay the network. TheDelayNetworkSettingsinterface is defined as follows:
The following configuration file defines a minimal example for the relay chain, including the nodes key:
[relaychain]
default_command = "polkadot"
default_image = "polkadot-debug:master"
chain = "rococo-local"
chain_spec_path = "/path/to/chain-spec.json"
default_args = ["--chain", "rococo-local"]
[[relaychain.nodes]]
name = "alice"
validator = true
balance = 1000000000000
[[relaychain.nodes]]
name = "bob"
validator = true
balance = 1000000000000
...
{
...,
"relaychain": {
"default_command": "polkadot",
"default_image": "polkadot-debug:master",
"chain": "rococo-local",
"chain_spec_path": "/path/to/chain-spec.json",
"default_args": ["--chain", "rococo-local"],
"nodes": [
{
"name": "alice",
"validator": true,
"balance": 1000000000000
},
{
"name": "bob",
"validator": true,
"balance": 1000000000000
}
],
...
},
...
}
Node Group Configuration¶
The node_groups key is used to define further parameters for the node groups. The available keys are:
namestring - name of the node. Any whitespace will be replaced with a dash (e.g.,new alicewill be converted tonew-alice)image?string - override default Docker image to use for this nodecommand?string - override default command to runargs?string[] - arguments to be passed to the commandenv?envVars[] - environment variables to set in the container. TheenvVarsinterface is defined as follows:overrides?Override[] - array of overrides definitions. TheOverrideinterface is defined as follows:prometheus_prefix?string - customizes the metric's prefix for the specific node. Defaults tosubstratedb_snapshot?string - database snapshot to usesubstrate_cli_args_version?SubstrateCliArgsVersion - set the Substrate CLI arguments version directly to skip binary evaluation overhead. TheSubstrateCliArgsVersionenum is defined as follows:resources?Resources - represent the resources limits/reservations needed by the node. TheResourcesinterface is defined as follows:keystore_key_types?string[] - defines which keystore keys should be createdcountnumber | string - number of nodes to launch for this groupdelay_network_settings?DelayNetworkSettings - sets the expected configuration to delay the network. TheDelayNetworkSettingsinterface is defined as follows:
The following configuration file defines a minimal example for the relay chain, including the node_groups key:
[relaychain]
default_command = "polkadot"
default_image = "polkadot-debug:master"
chain = "rococo-local"
chain_spec_path = "/path/to/chain-spec.json"
default_args = ["--chain", "rococo-local"]
[[relaychain.node_groups]]
name = "group-1"
count = 2
image = "polkadot-debug:master"
command = "polkadot"
args = ["--chain", "rococo-local"]
...
{
...,
"relaychain": {
"default_command": "polkadot",
"default_image": "polkadot-debug:master",
"chain": "rococo-local",
"chain_spec_path": "/path/to/chain-spec.json",
"default_args": ["--chain", "rococo-local"],
"node_groups": [
{
"name": "group-1",
"count": 2,
"image": "polkadot-debug:master",
"command": "polkadot",
"args": ["--chain", "rococo-local"]
}
],
...
},
...
}
Parachain Configuration¶
The parachain keyword is used to define further parameters for the parachain. The available keys are:
idnumber - the id to assign to this parachain. Must be uniquechain?string - the chain nameforce_decorator?string - force the use of a specific decoratorgenesis?JSON - the genesis configurationbalance?number - balance to set in balances for parachain's accountdelay_network_settings?DelayNetworkSettings - sets the expected configuration to delay the network. TheDelayNetworkSettingsinterface is defined as follows:add_to_genesis?boolean - flag to add parachain to genesis or register in runtime. Defaults totrueregister_para?boolean - flag to specify whether the para should be registered. Theadd_to_genesisflag must be set to false for this flag to have any effect. Defaults totrueonboard_as_parachain?boolean - flag to specify whether the para should be onboarded as a parachain, rather than remaining a parathread. Defaults totruegenesis_wasm_path?string - path to the Wasm file to usegenesis_wasm_generator?string - command to generate the Wasm filegenesis_state_path?string - path to the state file to usegenesis_state_generator?string - command to generate the state filechain_spec_path?string - path to the chain spec filechain_spec_command?string - command to generate the chain speccumulus_based?boolean - flag to use cumulus command generation. Defaults totruebootnodes?string[] - array of bootnodes to useprometheus_prefix?string - parameter for customizing the metric's prefix for all parachain nodes/collators. Defaults tosubstratecollator?Collator - further defined on the Collator Configuration sectioncollators?Collator[] - an array of collators to spawn. It is further defined on the Collator Configuration sectioncollator_groups?CollatorGroup[] - an array of collator groups to spawn. It is further defined on the Collator Groups section
For example, the following configuration file defines a minimal example for the parachain:
Collator Configuration¶
One specific key capable of receiving more subkeys is the collator key. This key is used to define further parameters for the nodes. The available keys are:
namestring - name of the collator. Any whitespace will be replaced with a dash (e.g.,new alicewill be converted tonew-alice)image?string - image to use for the collatorcommand_with_args?string - overrides both command and arguments for the collatorvalidatorboolean - pass the--validatorflag to the command. Defaults totrueinvulnerableboolean - if true, add the collator to invulnerables in the chain spec. Defaults tofalsebalancenumber - balance to set in balances for collator's account. Defaults to2000000000000bootnodes?string[] - array of bootnodes to useadd_to_bootnodes?boolean - add this collator to the bootnode list. Defaults tofalsews_port?number - WS port to userpc_port?number - RPC port to useprometheus_port?number - Prometheus port to usep2p_port?number - P2P port to usep2p_cert_hash?string - libp2p certhash to use with webRTC transportdelay_network_settings?DelayNetworkSettings - sets the expected configuration to delay the network. TheDelayNetworkSettingsinterface is defined as follows:command?string - override default command to runargs?string[] - arguments to be passed to the commandenv?envVars[] - environment variables to set in the container. TheenvVarsinterface is defined as follows:overrides?Override[] - array of overrides definitions. TheOverrideinterface is defined as follows:prometheus_prefix?string - customizes the metric's prefix for the specific node. Defaults tosubstratedb_snapshot?string - database snapshot to usesubstrate_cli_args_version?SubstrateCliArgsVersion - set the Substrate CLI arguments version directly to skip binary evaluation overhead. TheSubstrateCliArgsVersionenum is defined as follows:resources?Resources - represent the resources limits/reservations needed by the node. TheResourcesinterface is defined as follows:keystore_key_types?string[] - defines which keystore keys should be created
The configuration file below defines a minimal example for the collator:
Collator Groups¶
The collator_groups key is used to define further parameters for the collator groups. The available keys are:
namestring - name of the node. Any whitespace will be replaced with a dash (e.g.,new alicewill be converted tonew-alice)image?string - override default Docker image to use for this nodecommand?string - override default command to runargs?string[] - arguments to be passed to the commandenv?envVars[] - environment variables to set in the container. TheenvVarsinterface is defined as follows:overrides?Override[] - array of overrides definitions. TheOverrideinterface is defined as follows:prometheus_prefix?string - customizes the metric's prefix for the specific node. Defaults tosubstratedb_snapshot?string - database snapshot to usesubstrate_cli_args_version?SubstrateCliArgsVersion - set the Substrate CLI arguments version directly to skip binary evaluation overhead. TheSubstrateCliArgsVersionenum is defined as follows:resources?Resources - represent the resources limits/reservations needed by the node. TheResourcesinterface is defined as follows:keystore_key_types?string[] - defines which keystore keys should be createdcountnumber | string - number of nodes to launch for this groupdelay_network_settings?DelayNetworkSettings - sets the expected configuration to delay the network. TheDelayNetworkSettingsinterface is defined as follows:
For instance, the configuration file below defines a minimal example for the collator groups:
{
"parachain": {
"id": 100,
"add_to_genesis": true,
"cumulus_based": true,
"genesis_wasm_path": "/path/to/wasm",
"genesis_state_path": "/path/to/state",
"collator_groups": [
{
"name": "group-1",
"count": 2,
"image": "polkadot-parachain",
"command": "polkadot-parachain",
...
},
],
},
...
}
XCM Configuration¶
You can use the hrmp_channels keyword to define further parameters for the XCM channels at start-up. The available keys are:
-
hrmp_channelsHrmpChannelsConfig[] - array of Horizontal Relay-routed Message Passing (HRMP) channel configurations. TheHrmpChannelsConfiginterface is defined as follows:Each of theexport interface HrmpChannelsConfig { sender: number; recipient: number; max_capacity: number; max_message_size: number; }HrmpChannelsConfigkeys are defined as follows: -
sendernumber - parachain ID of the sender recipientnumber - parachain ID of the recipientmax_capacitynumber - maximum capacity of the HRMP channelmax_message_sizenumber - maximum message size allowed in the HRMP channel