Zombienet¶
Introduction¶
Zombienet is a testing framework designed for Polkadot SDK-based blockchains. It provides a simple CLI tool for creating and testing blockchain environments locally or across networks. This allows developers to easily run and interact with blockchain nodes in a controlled environment. Zombienet is a JavaScript library designed to run on Node.js and supports various backend providers, including Kubernetes, Podman, and local setups for running blockchain nodes.
The framework enables developers to create tests using natural language tools to verify on-chain storage, metrics, logs, and custom interactions with the blockchain. It is particularly effective for setting up local relaychains with validators and parachains with collators.
Parity Technologies has designed and developed this framework, now maintained by the Zombienet team. For further support and information, refer to the following contact points:
Install Zombienet¶
Zombienet releases are available on the Zombienet repository.
In order to install Zombienet, there are multiple options available, depending on the user's preferences and the environment where it will be used. The following section will guide you through the installation process for each of the available options.
Zombienet executables can be downloaded using the latest release uploaded on the Zombienet repository. You can download the executable for your operating system and architecture and then move it to a directory in your PATH. Each release includes executables for Linux and macOS, which are generated using pkg. This allows the Zombienet CLI to operate without requiring Node.js to be installed.
Alternatively, you can also download the executable using either curl
or wget
:
Note
Ensure to replace the URL with the <INSERT_ZOMBIENET_VERSION>
that you want to download, as well as the <INSERT_ZOMBIENET_EXECUTABLE>
with the name of the executable file that matches your operating system and architecture. This guide uses v1.3.106
and zombienet-macos-arm64
.
Note
This documentation explains the functionality of Chopsticks version 0.13.1
. Make sure you're using the correct version to match these instructions.
Then, ensure the downloaded file is executable:
Finally, you can run the following command to check if the installation was successful. If so, it will display the version of the installed Zombienet:
If you want to add the zombienet
executable to your PATH, you can move it to a directory in your PATH, such as /usr/local/bin
:
So then, you can refer to the zombienet
executable directly:
So then, you can refer to the zombienet
executable directly:
For Nix users, the Zombienet repository provides a flake.nix
file that can be used to install Zombienet. This means that users can easily incorporate Zombienet into their Nix-based projects.
To install Zombienet utilizing Nix, users can run the following command, triggering the fetching of the flake and subsequently installing the Zombienet package:
nix run github:paritytech/zombienet/INSERT_ZOMBIENET_VERSION -- \
spawn INSERT_ZOMBIENET_CONFIG_FILE_NAME.toml
Note
Ensure to replace the INSERT_ZOMBIENET_VERSION
with the desired version of Zombienet. Also, replace the INSERT_ZOMBIENET_CONFIG_FILE_NAME
with the name of the configuration file you want to use.
To run the command above, you need to have Flakes enabled.
Alternatively, you can also include the Zombienet binary in the PATH for the current shell. This can be achieved by:
Zombienet can also be run using Docker. The Zombienet repository provides a Docker image that can be used to run the Zombienet CLI. To run Zombienet using Docker, you can use the following command:
The command above will run the Zombienet CLI inside a Docker container and mount the current directory to the/home/nonroot/zombie-net/host-current-files
directory inside the container. This allows Zombienet to access the configuration file and other files in the current directory. If you want to mount a different directory, replace $(pwd)
with the desired directory path.
Inside the Docker container, you can run the Zombienet CLI commands. First, you need to set up Zombienet downloading the necessary binaries:
After that, you need to add those binaries to the PATH:
Finally, you can run the Zombienet CLI commands. For example, to spawn a network using a specific configuration file, you can run the following command:
The command above mounts the current directory to the /workspace
directory inside the Docker container. This allows Zombienet to access the configuration file and other files in the current directory. If you want to mount a different directory, replace $(pwd)
with the desired directory path.
Providers¶
Zombienet supports different backend providers for running the nodes. At this moment, Kubernetes, Podman, and local are supported, which can be declared as kubernetes
, podman
, or native
, respectively.
To use a particular provider, you can specify it in the network file or use the --provider
flag in the CLI:
Alternatively, you can set the provider in the network file:
At the moment, Zombienet supports the following providers: kubernetes
, podman
, and native
.
It's important to note that each provider has specific requirements and associated features. The subsequent sections will guide you through the installation process for each provider and the requirements and features each provider offers.
Kubernetes¶
Requirements¶
Zombienet is designed to be compatible with a variety of Kubernetes clusters, including Google Kubernetes Engine (GKE), Docker Desktop, and kind. To effectively interact with your cluster, you'll need to ensure that kubectl
is installed on your system, which is the Kubernetes command-line tool that allows you to run commands against Kubernetes clusters. If you don't have kubectl
installed, you can follow the instructions provided on the Kubernetes website.
Moreover, in order to create resources such as namespaces, pods, and CronJobs within the target cluster, you must have the appropriate permissions granted to your user or service account. These permissions are essential for managing and deploying applications effectively within Kubernetes.
Features¶
In Kubernetes, Zombienet uses the Prometheus operator (if available) to oversee monitoring and visibility. This configuration ensures that only essential networking-related pods are deployed. Using the Prometheus operator, Zombienet improves its ability to efficiently monitor and manage network activities within the Kubernetes cluster.
Podman¶
Requirements¶
Zombienet supports Podman rootless as a provider. To use Podman as a provider, you need to have Podman installed on your system. Podman is a daemonless container engine for developing, managing, and running Open Container Initiative (OCI) containers and container images on Linux-based systems. You can install Podman by following the instructions provided on the Podman website.
Warning
Currently, Podman can only be used with Zombienet on Linux machines. Although Podman has support for macOS through an internal VM, the Zombienet provider code requires Podman to run natively on Linux.
Features¶
Using Podman, Zombienet deploys additional pods to enhance the monitoring and visibility of the active network. Specifically, pods for Prometheus, Tempo, and Grafana are included in the deployment. Grafana is configured with Prometheus and Tempo as data sources.
Upon launching Zombienet, access to these monitoring services is facilitated through specific URLs provided in the output:
- Prometheus - http://127.0.0.1:34123
- Tempo - http://127.0.0.1:34125
- Grafana - http://127.0.0.1:41461
It's important to note that Grafana is deployed with default administrator access.
Note
When network operations cease —either by halting a running spawn with Ctrl+C or upon completion of the test— Zombienet automatically removes all associated pods.
Local¶
Requirements¶
The Zombienet local provider, also referred to as native, enables you to run nodes as local processes in your environment. You must have the necessary binaries for your network (such as polkadot
and polkadot-parachain
). These binaries should be available in your PATH, allowing Zombienet to spawn the nodes as local processes.
To install the necessary binaries, you can use the Zombienet CLI command:
This command will download and prepare the necessary binaries for Zombienet’s use.
Warning
The polkadot
and polkadot-parachain
binaries releases are not compatible with macOS. As a result, macOS users will need to clone the Polkadot repository, build the Polkadot binary, and manually add it to their PATH for polkadot
and polkadot-parachain
to work.
If you need to use a custom binary, ensure the binary is available in your PATH. You can also specify the binary path in the network configuration file. To showcase this, this guide will use the custom OpenZeppelin template as an example.
First, clone the OpenZeppelin template repository:
git clone https://github.com/OpenZeppelin/polkadot-runtime-templates \
&& cd polkadot-runtime-templates/generic-template
Then, build the custom binary:
```bash
cargo build --release
After that, add the custom binary to your PATH:
Alternatively, you can specify the binary path in the network configuration file:
[relaychain]
chain = "rococo-local"
default_command = "./bin-v1.6.0/polkadot"
[parachain]
id = 1000
[parachain.collators]
name = "collator01"
command = "./target/release/parachain-template-node"
Note
The local provider exclusively utilizes the command configuration for nodes/collators, which supports both relative and absolute paths. You can employ the default_command
configuration to specify the binary for spawning all nodes in the relay chain.
Features¶
Currently, the local provider does not execute any additional layers or processes.
CLI Usage¶
Zombienet provides a CLI that allows interaction with the tool. The CLI can receive commands and flags to perform different kinds of operations. These operations can be initiated using the following syntax:
The following sections will guide you through the primary usage of the Zombienet CLI and the available commands and flags.
CLI Commands¶
spawn
- spawn the network defined in the configuration file
<networkConfig>
- a file that declares the desired network to be spawned in.toml
or.json
format. For further information, check out the Configuration Files section
Warning
For the spawn
command to work on macOS, users need to be aware that the Polkadot binary is currently not compatible with macOS. As a result, macOS users will need to clone the Polkadot repository, build Polkadot binary, and manually add it to their PATH.
test
- run test on the network spawned
<testFile>
- a file that defines assertions and tests against the spawned network, using natural language expressions to evaluate metrics, logs, and built-in functions
setup
- set up the Zombienet development environment
<binaries>
- executables that will be downloaded and prepared to be used by Zombienet. Options:polkadot
,polkadot-parachain
convert
- transforms a (now deprecated) polkadot-launch configuration file to a Zombienet configuration file
<filePath>
- path to a polkadot-launch configuration file with a.js
or.json
extension defined by theLaunchConfig
interface
version
- prints Zombienet version
None
help
- prints help information
None
CLI Flags¶
You can use the following flags to customize the behavior of the CLI:
-p
, --provider
- override provider to use. Defaults to kubernetes
<provider>
- the provider to use. Options:podman
,kubernetes
,native
-d
, --dir
- directory path for placing the network files instead of random temp one
<path>
- desired path for network files
zombienet -d /home/user/my-zombienet
-f
, --force
- force override all prompt commands
None
-l
, --logType
- type of logging on the console. Defaults to table
<logType>
desired type of logging. Options:table
,text
,silent
-m
, --monitor
- start as monitor, do not auto clean up network
None
-c
, --spawn-concurrency
- number of concurrent spawning processes to launch. Defaults to 1
<concurrency>
- desired quantity of processes
-h
, --help
- display help for command
None
Configuration Files¶
The network configuration can be given in either JSON or TOML format. The Zombienet repository also provides a folder with some examples of configuration files that can be used as a reference.
Note
Each section may include provider-specific keys that are not recognized by other providers. For example, if you use the local provider, any references to images for nodes will be disregarded.
Settings¶
Through the keyword settings
, it's possible to define the general settings for the network. The available keys are:
global_volumes?
GlobalVolume[] - a list of global volumes to use. TheGlobalVolume
interface is defined as follows:bootnode
boolean - add bootnode to network. Default istrue
bootnode_domain?
string - domain to use for bootnodetimeout
number - global timeout to use for spawning the whole network"node_spawn_timeout?
number - timeout to spawn pod/processgrafana?
boolean - deploy an instance of Grafanaprometheus?
boolean - deploy an instance of Prometheustelemetry?
boolean - enable telemetry for the networkjaeger_agent?
string - the Jaeger agent endpoint passed to the nodes. Only available on Kubernetestracing_collator_url?
string - the URL of the tracing collator used to query by the tracing assertion. Should be tempo query compatibletracing_collator_service_name?
string - service name for tempo query frontend. Only available on Kubernetes. Defaults totempo-tempo-distributed-query-frontend
tracing_collator_service_namespace?
string - namespace where tempo is running. Only available on Kubernetes. Defaults totempo
tracing_collator_service_port?
number - port of the query instance of tempo. Only available on Kubernetes. Defaults to3100
enable_tracing?
boolean - enable the tracing system. Only available on Kubernetes. Defaults totrue
provider
string - provider to use. Default iskubernetes
"polkadot_introspector?
boolean - deploy an instance of polkadot-introspector. Only available on Podman and Kubernetes. Defaults tofalse
backchannel?
boolean - deploy an instance of backchannel server. Only available on Kubernetes. Defaults tofalse
image_pull_policy?
string - image pull policy to use in the network. Possible values areAlways
,IfNotPresent
, andNever
local_ip?
string - IP used for exposing local services (rpc/metrics/monitors). Defaults to"127.0.0.1"
global_delay_network_global_settings?
number - delay in seconds to apply to the networknode_verifier?
string - specify how to verify node readiness or deactivate by usingNone
. Possible values areNone
andMetric
. Defaults toMetric
For example, the following configuration file defines a minimal example for the settings:
Relay Chain Configuration¶
You can use the relaychain
keyword to define further parameters for the relay chain at start-up. The available keys are:
default_command?
string - the default command to run. Defaults topolkadot
default_image?
string - the default Docker image to usedefault_resources?
Resources - represents the resources limits/reservations needed by the nodes by default. Only available on Kubernetes. TheResources
interface is defined as follows:default_db_snapshot?
string - the default database snapshot to usedefault_prometheus_prefix
string - a parameter for customizing the metric's prefix. Defaults tosubstrate
default_substrate_cli_args_version?
SubstrateCliArgsVersion - set the Substrate CLI arguments version. TheSubstrateCliArgsVersion
enum is defined as follows:default_keystore_key_types?
string[] - defines which keystore keys should be createdchain
string - the chain namechain_spec_path?
string - path to the chain spec file. Should be the plain version to allow customizationschain_spec_command?
string - command to generate the chain spec. It can't be used in combination withchain_spec_path
default_args?
string[] - an array of arguments to use as default to pass to the commanddefault_overrides?
Override[] - an array of overrides to upload to the node. TheOverride
interface is defined as follows:random_nominators_count?
number - if set and the stacking pallet is enabled, Zombienet will generate the input quantity of nominators and inject them into the genesismax_nominations
number - the max number of nominations allowed by a nominator. Should match the value set in the runtime. Defaults to24
nodes?
Node[] - an array of nodes to spawn. It is further defined on the Node Configuration sectionnode_groups?
NodeGroup[] - an array of node groups to spawn. It is further defined on the Node Group Configuration sectiontotal_node_in_group?
number - the total number of nodes in the group. Defaults to1
genesis
JSON - the genesis configurationdefault_delay_network_settings?
DelayNetworkSettings - sets the expected configuration to delay the network. TheDelayNetworkSettings
interface is defined as follows:
Node Configuration¶
There is one specific key capable of receiving more subkeys: the nodes
key. This key is used to define further parameters for the nodes. The available keys:
name
string - name of the node. Any whitespace will be replaced with a dash (e.g.,new alice
will be converted tonew-alice
)image?
string - override default Docker image to use for this nodecommand?
string - override default command to runcommand_with_args?
string - override default command and argumentsargs?
string[] - arguments to be passed to the commandenv?
envVars[] - environment variables to set in the container. TheenvVars
interface is defined as follows:overrides?
Override[] - array of overrides definitions. TheOverride
interface is defined as follows:prometheus_prefix?
string - customizes the metric's prefix for the specific node. Defaults tosubstrate
db_snapshot?
string - database snapshot to usesubstrate_cli_args_version?
SubstrateCliArgsVersion - set the Substrate CLI arguments version directly to skip binary evaluation overhead. TheSubstrateCliArgsVersion
enum is defined as follows:resources?
Resources - represent the resources limits/reservations needed by the node. TheResources
interface is defined as follows:keystore_key_types?
string[] - defines which keystore keys should be createdvalidator
boolean - pass the--validator
flag to the command. Defaults totrue
invulnerable
boolean - if true, add the node to invulnerables in the chain spec. Defaults tofalse
balance
number - balance to set in balances for node's account. Defaults to2000000000000
bootnodes?
string[] - array of bootnodes to useadd_to_bootnodes?
boolean - add this node to the bootnode list. Defaults tofalse
ws_port?
number - WS port to userpc_port?
number - RPC port to useprometheus_port?
number - Prometheus port to usep2p_cert_hash?
string - libp2p certhash to use with webRTC transportdelay_network_settings?
DelayNetworkSettings - sets the expected configuration to delay the network. TheDelayNetworkSettings
interface is defined as follows:
The following configuration file defines a minimal example for the relay chain, including the nodes
key:
[relaychain]
default_command = "polkadot"
default_image = "polkadot-debug:master"
chain = "rococo-local"
chain_spec_path = "/path/to/chain-spec.json"
default_args = ["--chain", "rococo-local"]
[[relaychain.nodes]]
name = "alice"
validator = true
balance = 1000000000000
[[relaychain.nodes]]
name = "bob"
validator = true
balance = 1000000000000
...
{
...,
"relaychain": {
"default_command": "polkadot",
"default_image": "polkadot-debug:master",
"chain": "rococo-local",
"chain_spec_path": "/path/to/chain-spec.json",
"default_args": ["--chain", "rococo-local"],
"nodes": [
{
"name": "alice",
"validator": true,
"balance": 1000000000000
},
{
"name": "bob",
"validator": true,
"balance": 1000000000000
}
],
...
},
...
}
Node Group Configuration¶
The node_groups
key is used to define further parameters for the node groups. The available keys are:
name
string - name of the node. Any whitespace will be replaced with a dash (e.g.,new alice
will be converted tonew-alice
)image?
string - override default Docker image to use for this nodecommand?
string - override default command to runargs?
string[] - arguments to be passed to the commandenv?
envVars[] - environment variables to set in the container. TheenvVars
interface is defined as follows:overrides?
Override[] - array of overrides definitions. TheOverride
interface is defined as follows:prometheus_prefix?
string - customizes the metric's prefix for the specific node. Defaults tosubstrate
db_snapshot?
string - database snapshot to usesubstrate_cli_args_version?
SubstrateCliArgsVersion - set the Substrate CLI arguments version directly to skip binary evaluation overhead. TheSubstrateCliArgsVersion
enum is defined as follows:resources?
Resources - represent the resources limits/reservations needed by the node. TheResources
interface is defined as follows:keystore_key_types?
string[] - defines which keystore keys should be createdcount
number | string - number of nodes to launch for this groupdelay_network_settings?
DelayNetworkSettings - sets the expected configuration to delay the network. TheDelayNetworkSettings
interface is defined as follows:
The following configuration file defines a minimal example for the relay chain, including the node_groups
key:
[relaychain]
default_command = "polkadot"
default_image = "polkadot-debug:master"
chain = "rococo-local"
chain_spec_path = "/path/to/chain-spec.json"
default_args = ["--chain", "rococo-local"]
[[relaychain.node_groups]]
name = "group-1"
count = 2
image = "polkadot-debug:master"
command = "polkadot"
args = ["--chain", "rococo-local"]
...
{
...,
"relaychain": {
"default_command": "polkadot",
"default_image": "polkadot-debug:master",
"chain": "rococo-local",
"chain_spec_path": "/path/to/chain-spec.json",
"default_args": ["--chain", "rococo-local"],
"node_groups": [
{
"name": "group-1",
"count": 2,
"image": "polkadot-debug:master",
"command": "polkadot",
"args": ["--chain", "rococo-local"]
}
],
...
},
...
}
Parachain Configuration¶
The parachain
keyword is used to define further parameters for the parachain. The available keys are:
id
number - the id to assign to this parachain. Must be uniquechain?
string - the chain nameforce_decorator?
string - force the use of a specific decoratorgenesis?
JSON - the genesis configurationbalance?
number - balance to set in balances for parachain's accountdelay_network_settings?
DelayNetworkSettings - sets the expected configuration to delay the network. TheDelayNetworkSettings
interface is defined as follows:add_to_genesis?
boolean - flag to add parachain to genesis or register in runtime. Defaults totrue
register_para?
boolean - flag to specify whether the para should be registered. Theadd_to_genesis
flag must be set to false for this flag to have any effect. Defaults totrue
onboard_as_parachain?
boolean - flag to specify whether the para should be onboarded as a parachain, rather than remaining a parathread. Defaults totrue
genesis_wasm_path?
string - path to the Wasm file to usegenesis_wasm_generator?
string - command to generate the Wasm filegenesis_state_path?
string - path to the state file to usegenesis_state_generator?
string - command to generate the state filechain_spec_path?
string - path to the chain spec filechain_spec_command?
string - command to generate the chain speccumulus_based?
boolean - flag to use cumulus command generation. Defaults totrue
bootnodes?
string[] - array of bootnodes to useprometheus_prefix?
string - parameter for customizing the metric's prefix for all parachain nodes/collators. Defaults tosubstrate
collator?
Collator - further defined on the Collator Configuration sectioncollators?
Collator[] - an array of collators to spawn. It is further defined on the Collator Configuration sectioncollator_groups?
CollatorGroup[] - an array of collator groups to spawn. It is further defined on the Collator Groups section
For example, the following configuration file defines a minimal example for the parachain:
Collator Configuration¶
One specific key capable of receiving more subkeys is the collator
key. This key is used to define further parameters for the nodes. The available keys are:
name
string - name of the collator. Any whitespace will be replaced with a dash (e.g.,new alice
will be converted tonew-alice
)image?
string - image to use for the collatorcommand_with_args?
string - overrides both command and arguments for the collatorvalidator
boolean - pass the--validator
flag to the command. Defaults totrue
invulnerable
boolean - if true, add the collator to invulnerables in the chain spec. Defaults tofalse
balance
number - balance to set in balances for collator's account. Defaults to2000000000000
bootnodes?
string[] - array of bootnodes to useadd_to_bootnodes?
boolean - add this collator to the bootnode list. Defaults tofalse
ws_port?
number - WS port to userpc_port?
number - RPC port to useprometheus_port?
number - Prometheus port to usep2p_port?
number - P2P port to usep2p_cert_hash?
string - libp2p certhash to use with webRTC transportdelay_network_settings?
DelayNetworkSettings - sets the expected configuration to delay the network. TheDelayNetworkSettings
interface is defined as follows:command?
string - override default command to runargs?
string[] - arguments to be passed to the commandenv?
envVars[] - environment variables to set in the container. TheenvVars
interface is defined as follows:overrides?
Override[] - array of overrides definitions. TheOverride
interface is defined as follows:prometheus_prefix?
string - customizes the metric's prefix for the specific node. Defaults tosubstrate
db_snapshot?
string - database snapshot to usesubstrate_cli_args_version?
SubstrateCliArgsVersion - set the Substrate CLI arguments version directly to skip binary evaluation overhead. TheSubstrateCliArgsVersion
enum is defined as follows:resources?
Resources - represent the resources limits/reservations needed by the node. TheResources
interface is defined as follows:keystore_key_types?
string[] - defines which keystore keys should be created
The configuration file below defines a minimal example for the collator:
Collator Groups¶
The collator_groups
key is used to define further parameters for the collator groups. The available keys are:
name
string - name of the node. Any whitespace will be replaced with a dash (e.g.,new alice
will be converted tonew-alice
)image?
string - override default Docker image to use for this nodecommand?
string - override default command to runargs?
string[] - arguments to be passed to the commandenv?
envVars[] - environment variables to set in the container. TheenvVars
interface is defined as follows:overrides?
Override[] - array of overrides definitions. TheOverride
interface is defined as follows:prometheus_prefix?
string - customizes the metric's prefix for the specific node. Defaults tosubstrate
db_snapshot?
string - database snapshot to usesubstrate_cli_args_version?
SubstrateCliArgsVersion - set the Substrate CLI arguments version directly to skip binary evaluation overhead. TheSubstrateCliArgsVersion
enum is defined as follows:resources?
Resources - represent the resources limits/reservations needed by the node. TheResources
interface is defined as follows:keystore_key_types?
string[] - defines which keystore keys should be createdcount
number | string - number of nodes to launch for this groupdelay_network_settings?
DelayNetworkSettings - sets the expected configuration to delay the network. TheDelayNetworkSettings
interface is defined as follows:
For instance, the configuration file below defines a minimal example for the collator groups:
{
"parachain": {
"id": 100,
"add_to_genesis": true,
"cumulus_based": true,
"genesis_wasm_path": "/path/to/wasm",
"genesis_state_path": "/path/to/state",
"collator_groups": [
{
"name": "group-1",
"count": 2,
"image": "polkadot-parachain",
"command": "polkadot-parachain",
...
},
],
},
...
}
XCM Configuration¶
You can use the hrmp_channels
keyword to define further parameters for the XCM channels at start-up. The available keys are:
-
hrmp_channels
HrmpChannelsConfig[] - array of Horizontal Relay-routed Message Passing (HRMP) channel configurations. TheHrmpChannelsConfig
interface is defined as follows:Each of theexport interface HrmpChannelsConfig { sender: number; recipient: number; max_capacity: number; max_message_size: number; }
HrmpChannelsConfig
keys are defined as follows: -
sender
number - parachain ID of the sender recipient
number - parachain ID of the recipientmax_capacity
number - maximum capacity of the HRMP channelmax_message_size
number - maximum message size allowed in the HRMP channel