# How to Use Antithesis ## Context Antithesis is a third party vendor with an environment that can perform network fuzzing. We can upload images containing `docker-compose.yml` files, which represent various MongoDB topologies, to the Antithesis Docker registry. Antithesis runs `docker-compose up` from these images to spin up the corresponding multi-container application in their environment and run a test suite. Network fuzzing is performed on the topology while the test suite runs & a report is generated by Antithesis identifying bugs. Check out https://github.com/mongodb/mongo/wiki/Testing-MongoDB-with-Antithesis to see an example of how we use Antithesis today. ## Base Images The `base_images` directory consists of the building blocks for creating a MongoDB test topology. These images are uploaded to the Antithesis Docker registry [nightly](https://github.com/mongodb/mongo/blob/6cf8b162a61173eb372b54213def6dd61e1fd684/etc/evergreen_yml_components/variants/ubuntu/test_dev_master_and_lts_branches_only.yml#L28) during the [`antithesis image build and push`](https://github.com/mongodb/mongo/blob/020632e3ae328f276b2c251417b5a39389af6141/etc/evergreen_yml_components/definitions.yml#L2823) function. ### mongo_binaries This image contains the latest `mongo`, `mongos` and `mongod` binaries. It can be used to start a `mongod` instance, `mongos` instance or execute `mongo` commands. This is the main building block for creating the System Under Test topology. ### workload This image contains the latest `mongo` binary as well as the `resmoke` test runner. The `workload` container is not part of the actual toplogy. The purpose of a `workload` container is to execute `mongo` commands to complete the topology setup, and to run a test suite on an existing topology like so: ```shell buildscript/resmoke.py run --suite antithesis_concurrency_sharded_with_stepdowns_and_balancer ``` **Every topology must have 1 workload container.** Note: During `workload` image build, `evergreen/antithesis_image_build_and_push.sh` runs, which generates "antithesis compatible" test suites and prepends them with `antithesis_`. These are the test suites that can run in antithesis and are available from within the `workload` container. ### Dockerfile This assembles an image with the necessary files for spinning up the corresponding topology. It consists of a `docker-compose.yml`, a `logs` directory, a `scripts` directory and a `data` directory. If this is structured properly, you should be able to copy the files & directories from this image and run `docker-compose up` to set up the desired topology. Example from what `buildscripts/resmokelib/testing/docker_cluster_image_builder.py` generates: ```Dockerfile FROM scratch COPY docker-compose.yml / ADD scripts /scripts ADD logs /logs ADD data /data ADD debug /debug ``` All topology images are built and uploaded to the Antithesis Docker registry during the `antithesis image build and push` task. Some of these directories are created during the `evergreen/antithesis_image_build_and_push.sh` script such as `/data` and `/logs`. Note: These images serve solely as a filesystem containing all necessary files for a topology, therefore use `FROM scratch`. ### docker-compose.yml This describes how to construct the corresponding topology using the `mongo-binaries` and `workload` images. Example from `buildscripts/antithesis/topologies/sharded_cluster/docker-compose.yml`: ```yml version: '3.0' services: configsvr1: container_name: configsvr1 hostname: configsvr1 image: mongo-binaries:evergreen-latest-master volumes: - ./logs/configsvr1:/var/log/mongodb/ - ./scripts:/scripts/ - ./data/configsvr1:/data/configdb/ command: /bin/bash /scripts/configsvr_init.sh networks: antithesis-net: ipv4_address: 10.20.20.6 # Set the an IPv4 with an address of 10.20.20.130 or higher # to be ignored by the fault injector # configsvr2: ... configsvr3: ... database1: ... container_name: database1 hostname: database1 image: mongo-binaries:evergreen-latest-master volumes: - ./logs/database1:/var/log/mongodb/ - ./scripts:/scripts/ - ./data/database1:/data/db/ command: /bin/bash /scripts/database_init.sh Shard1 networks: antithesis-net: ipv4_address: 10.20.20.3 # Set the an IPv4 with an address of 10.20.20.130 or higher # to be ignored by the fault injector # database2: ... database3: ... database4: ... database5: ... database6: ... mongos: container_name: mongos hostname: mongos image: mongo-binaries:evergreen-latest-master volumes: - ./logs/mongos:/var/log/mongodb/ - ./scripts:/scripts/ command: python3 /scripts/mongos_init.py depends_on: - "database1" - "database2" - "database3" - "database4" - "database5" - "database6" - "configsvr1" - "configsvr2" - "configsvr3" networks: antithesis-net: ipv4_address: 10.20.20.9 # The subnet provided here is an example # An alternative subnet can be used workload: container_name: workload hostname: workload image: workload:evergreen-latest-master volumes: - ./logs/workload:/var/log/resmoke/ - ./scripts:/scripts/ command: python3 /scripts/workload_init.py depends_on: - "mongos" networks: antithesis-net: ipv4_address: 10.20.20.130 # The subnet provided here is an example # An alternative subnet can be used networks: antithesis-net: driver: bridge ipam: config: - subnet: 10.20.20.0/24 ``` Each container must have a `command` in `docker-compose.yml` that runs an init script. The init script belongs in the `scripts` directory, which is included as a volume. The `command` should be set like so: `/bin/bash /scripts/[script_name].sh` or `python3 /scripts/[script_name].py`. This is a requirement for the topology to start up properly in Antithesis. When creating `mongod` or `mongos` instances, route the logs like so: `--logpath /var/log/mongodb/mongodb.log` and utilize `volumes` -- as in `database1`. This enables us to easily retrieve logs if a bug is detected by Antithesis. The `ipv4_address` should be set to `10.20.20.130` or higher if you do not want that container to be affected by network fuzzing. For instance, you would likely not want the `workload` container to be affected by network fuzzing -- as shown in the example above. Use the `evergreen-latest-master` tag for all images. This is updated automatically in `evergreen/antithesis_image_build_and_push.sh` -- if needed. ### scripts Take a look at `buildscripts/antithesis/topologies/sharded_cluster/scripts/mongos_init.py` to see how to use util methods from `buildscripts/antithesis/topologies/sharded_cluster/scripts/utils.py` to set up the desired topology. You can also use simple shell scripts as in the case of `buildscripts/antithesis/topologies/sharded_cluster/scripts/database_init.py`. These init scripts must not end in order to keep the underlying container alive. You can use an infinite while loop for `python` scripts or you can use `tail -f /dev/null` for shell scripts. ## How do I create a new topology for Antithesis testing? This should be done with care to ensure we are using our limited resources efficiently. Create a new task extending the `antithesis_task_template`, tagged with `antithesis`, passing the specified `suite` to the `antithesis image build and push` task. See other examples to get started. ## Additional Resources If you are interested in leveraging Antithesis feel free to reach out to #server-testing on Slack.