180 lines
8.6 KiB
Markdown
180 lines
8.6 KiB
Markdown
# How to Use Antithesis
|
|
|
|
## Context
|
|
Antithesis is a third party vendor with an environment that can perform network fuzzing. We can
|
|
upload images containing `docker-compose.yml` files, which represent various MongoDB topologies, to
|
|
the Antithesis Docker registry. Antithesis runs `docker-compose up` from these images to spin up
|
|
the corresponding multi-container application in their environment and run a test suite. Network
|
|
fuzzing is performed on the topology while the test suite runs & a report is generated by
|
|
Antithesis identifying bugs. Check out
|
|
https://github.com/mongodb/mongo/wiki/Testing-MongoDB-with-Antithesis to see an example of how we
|
|
use Antithesis today.
|
|
|
|
## Base Images
|
|
The `base_images` directory consists of the building blocks for creating a MongoDB test topology.
|
|
These images are uploaded to the Antithesis Docker registry weekly during the
|
|
`antithesis_image_build` task. For more visibility into how these images are built and uploaded to
|
|
the Antithesis Docker registry, please see `evergreen/antithesis_image_build.sh`.
|
|
|
|
### mongo_binaries
|
|
This image contains the latest `mongo`, `mongos` and `mongod` binaries. It can be used to
|
|
start a `mongod` instance, `mongos` instance or execute `mongo` commands. This is the main building
|
|
block for creating the System Under Test topology.
|
|
|
|
### workload
|
|
This image contains the latest `mongo` binary as well as the `resmoke` test runner. The `workload`
|
|
container is not part of the actual toplogy. The purpose of a `workload` container is to execute
|
|
`mongo` commands to complete the topology setup, and to run a test suite on an existing topology
|
|
like so:
|
|
```shell
|
|
buildscript/resmoke.py run --suite antithesis_concurrency_sharded_with_stepdowns_and_balancer --shellConnString "mongodb://mongos:27017"
|
|
```
|
|
|
|
**Every topology must have 1 workload container.**
|
|
|
|
Note: During `workload` image build, `buildscripts/antithesis_suite.py` runs, which generates
|
|
"antithesis compatible" test suites and prepends them with `antithesis_`. These are the test suites
|
|
that can run in antithesis and are available from witihin the `workload` container.
|
|
|
|
## Topologies
|
|
The `topologies` directory consists of subdirectories representing various mongo test topologies.
|
|
Each topology has a `Dockerfile`, a `docker-compose.yml` file and a `scripts` directory.
|
|
|
|
### Dockerfile
|
|
This assembles an image with the necessary files for spinning up the corresponding topology. It
|
|
consists of a `docker-compose.yml`, a `logs` directory, a `scripts` directory and a `data`
|
|
directory. If this is structured properly, you should be able to copy the files & directories
|
|
from this image and run `docker-compose up` to set up the desired topology.
|
|
|
|
Example from `buildscripts/antithesis/topologies/replica_set/Dockerfile`:
|
|
```Dockerfile
|
|
FROM scratch
|
|
COPY docker-compose.yml /
|
|
ADD scripts /scripts
|
|
ADD logs /logs
|
|
ADD data /data
|
|
```
|
|
|
|
All topology images are built and uploaded to the Antithesis Docker registry during the
|
|
`antithesis_image_build` task in the `evergreen/antithesis_image_build.sh` script. Some of these
|
|
directories are created in `evergreen/antithesis_image_build.sh` such as `/data` and `/logs`.
|
|
|
|
Note: These images serve solely as a filesystem containing all necessary files for a topology,
|
|
therefore use `FROM scratch`.
|
|
|
|
### docker-compose.yml
|
|
This describes how to construct the corresponding topology using the
|
|
`mongo-binaries` and `workload` images.
|
|
|
|
Example from `buildscripts/antithesis/topologies/replica_set/docker-compose.yml`:
|
|
```yml
|
|
version: '3.0'
|
|
|
|
services:
|
|
database1:
|
|
container_name: database1
|
|
hostname: database1
|
|
image: mongo-binaries:evergreen-latest-master
|
|
command: /bin/bash /scripts/database_init.sh
|
|
volumes:
|
|
- ./logs/database1:/var/log/mongodb/
|
|
- ./scripts:/scripts/
|
|
- ./data/database1:/data/db/
|
|
networks:
|
|
antithesis-net:
|
|
ipv4_address: 10.20.20.3
|
|
# Set the an IPv4 with an address of 10.20.20.130 or higher
|
|
# to be ignored by the fault injector
|
|
#
|
|
database2: ...
|
|
database3: ...
|
|
workload:
|
|
container_name: workload
|
|
hostname: workload
|
|
image: workload:evergreen-latest-master
|
|
command: /bin/bash /scripts/workload_init.sh
|
|
volumes:
|
|
- ./logs/workload:/var/log/resmoke/
|
|
- ./scripts:/scripts/
|
|
depends_on:
|
|
- "database1"
|
|
- "database2"
|
|
- "database3"
|
|
networks:
|
|
antithesis-net:
|
|
ipv4_address: 10.20.20.130
|
|
# The subnet provided here is an example
|
|
# An alternative subnet can be used
|
|
|
|
networks:
|
|
antithesis-net:
|
|
driver: bridge
|
|
ipam:
|
|
config:
|
|
- subnet: 10.20.20.0/24
|
|
|
|
```
|
|
|
|
Each container must have a `command`in `docker-compose.yml` that runs an init script. The init
|
|
script belongs in the `scripts` directory, which is included as a volume. The `command` should be
|
|
set like so: `/bin/bash /scripts/[script_name].sh`. This is a requirement for the topology to start
|
|
up properly in Antithesis.
|
|
|
|
When creating `mongod` or `mongos` instances, route the logs like so:
|
|
`--logpath /var/log/mongodb/mongodb.log` and utilize `volumes` -- as in `database1`.
|
|
This enables us to easily retrieve logs if a bug is detected by Antithesis.
|
|
|
|
The `ipv4_address` should be set to `10.20.20.130` or higher if you do not want that container to
|
|
be affected by network fuzzing. For instance, you would likely not want the `workload` container
|
|
to be affected by network fuzzing -- as shown in the example above.
|
|
|
|
Use the `evergreen-latest-master` tag for all images. This is updated automatically in
|
|
`evergreen/antithesis_image_build.sh` during the `antithesis_image_build` task -- if needed.
|
|
|
|
### scripts
|
|
|
|
Example from `buildscripts/antithesis/topologies/replica_set/scripts/workload_init.sh`:
|
|
```shell
|
|
sleep 5s
|
|
mongo --host database1 --port 27017 --eval "config={\"_id\" : \"RollbackFuzzerTest\",\"protocolVersion\" : 1,\"members\" : [{\"_id\" : 0,\"host\" : \"database1:27017\"}, {\"_id\" : 1,\"host\" : \"database2:27017\"}, {\"_id\" : 2,\"host\" : \"database3:27017\"} ],\"settings\" : {\"chainingAllowed\" : false,\"electionTimeoutMillis\" : 500, \"heartbeatTimeoutSecs\" : 1, \"catchUpTimeoutMillis\": 700}}; rs.initiate(config)"
|
|
|
|
# this cryptic statement keeps the workload container running.
|
|
tail -f /dev/null
|
|
```
|
|
The `sleep` command can be useful to ensure that other containers have had a chance to start. In
|
|
this example, the `workload` container waits 5 seconds while the database containers start.
|
|
After that, it initiates the replica set. The `tail -f /dev/null` is required for `workload`
|
|
containers otherwise the container shuts down.
|
|
|
|
## How do I create a new topology for Antithesis testing?
|
|
To create a new topology for Antithesis testing is easy & requires a few simple steps.
|
|
1. Add a new directory in `buildscripts/antithesis/topologies` to represent your desired topology.
|
|
You can use existing topologies as an example.
|
|
2. Update the `evergreen/antithesis_image_build.sh` file so that your new topology image is
|
|
uploaded to the Antithesis Docker registry.
|
|
3. Reach out to #server-testing on Slack & provide the new topology image name as well as the
|
|
desired test suite to run.
|
|
4. Include a member of the STM team on the code review.
|
|
|
|
These are the required updates to `evergreen/antithesis_image_build.sh`:
|
|
- Add the following command for each of your `mongos` and `mongod` containers in your topology to
|
|
create your log directories.
|
|
```shell
|
|
mkdir -p antithesis/topologies/[topology_name]/{logs,data}/[container_name]
|
|
```
|
|
|
|
- Build an image for your new topology ending in `-config`
|
|
```shell
|
|
cd [your_topology_dir]
|
|
sed -i s/evergreen-latest-master/$tag/ docker-compose.yml
|
|
sudo docker build . -t [your-topology-name]-config:$tag
|
|
```
|
|
- Push your new image to the Antithesis Docker registry
|
|
```shell
|
|
sudo docker tag "[your-topology-name]-config:$tag" "us-central1-docker.pkg.dev/molten-verve-216720/mongodb-repository/[your-topology-name]-config:$tag"
|
|
sudo docker push "us-central1-docker.pkg.dev/molten-verve-216720/mongodb-repository/[your-topology-name]-config:$tag"
|
|
```
|
|
|
|
## Additional Resources
|
|
If you are interested in leveraging Antithesis feel free to reach out to #server-testing on Slack.
|