mongo/buildscripts/antithesis/README.md

# How to Use Antithesis

## Context

Antithesis is a third party vendor with an environment that can perform network fuzzing. We can
upload images containing `docker-compose.yml` files, which represent various MongoDB topologies, to
the Antithesis Docker registry. Antithesis runs `docker-compose up` from these images to spin up
the corresponding multi-container application in their environment and run a test suite. Network
fuzzing is performed on the topology while the test suite runs & a report is generated by
Antithesis identifying bugs. Check out
https://github.com/mongodb/mongo/wiki/Testing-MongoDB-with-Antithesis to see an example of how we
use Antithesis today.

## Base Images

The `base_images` directory consists of the building blocks for creating a MongoDB test topology.
These images are uploaded to the Antithesis Docker registry [nightly](https://github.com/mongodb/mongo/blob/6cf8b162a61173eb372b54213def6dd61e1fd684/etc/evergreen_yml_components/variants/ubuntu/test_dev_master_and_lts_branches_only.yml#L28) during the
[`antithesis image build and push`](https://github.com/mongodb/mongo/blob/020632e3ae328f276b2c251417b5a39389af6141/etc/evergreen_yml_components/definitions.yml#L2823) function.

### mongo_binaries

This image contains the latest `mongo`, `mongos` and `mongod` binaries. It can be used to
start a `mongod` instance, `mongos` instance or execute `mongo` commands. This is the main building
block for creating the System Under Test topology.

### workload

This image contains the latest `mongo` binary as well as the `resmoke` test runner. The `workload`
container is not part of the actual toplogy. The purpose of a `workload` container is to execute
`mongo` commands to complete the topology setup, and to run a test suite on an existing topology
like so:

```shell
buildscript/resmoke.py run --suite antithesis_concurrency_sharded_with_stepdowns_and_balancer
```

**Every topology must have 1 workload container.**

Note: During `workload` image build, `evergreen/antithesis_image_build_and_push.sh` runs, which generates
"antithesis compatible" test suites and prepends them with `antithesis_`. These are the test suites
that can run in antithesis and are available from within the `workload` container.

### Dockerfile

This assembles an image with the necessary files for spinning up the corresponding topology. It
consists of a `docker-compose.yml`, a `logs` directory, a `scripts` directory and a `data`
directory. If this is structured properly, you should be able to copy the files & directories
from this image and run `docker-compose up` to set up the desired topology.

Example from what `buildscripts/resmokelib/testing/docker_cluster_image_builder.py` generates:

```Dockerfile
FROM scratch
COPY docker-compose.yml /
ADD scripts /scripts
ADD logs /logs
ADD data /data
ADD debug /debug
```

All topology images are built and uploaded to the Antithesis Docker registry during the
`antithesis image build and push` task. Some of these directories are created during the
`evergreen/antithesis_image_build_and_push.sh` script such as `/data` and `/logs`.

Note: These images serve solely as a filesystem containing all necessary files for a topology,
therefore use `FROM scratch`.

### docker-compose.yml

This describes how to construct the corresponding topology using the
`mongo-binaries` and `workload` images.

Example from `buildscripts/antithesis/topologies/sharded_cluster/docker-compose.yml`:

```yml
version: '3.0'

services:
        configsvr1:
                container_name: configsvr1
                hostname: configsvr1
                image: mongo-binaries:evergreen-latest-master
                volumes:
                  - ./logs/configsvr1:/var/log/mongodb/
                  - ./scripts:/scripts/
                  - ./data/configsvr1:/data/configdb/
                command: /bin/bash /scripts/configsvr_init.sh
                networks:
                        antithesis-net:
                                ipv4_address: 10.20.20.6
                                # Set the an IPv4 with an address of 10.20.20.130 or higher
                                # to be ignored by the fault injector
                                #

        configsvr2: ...
        configsvr3: ...
        database1: ...
                container_name: database1
                hostname: database1
                image: mongo-binaries:evergreen-latest-master
                volumes:
                  - ./logs/database1:/var/log/mongodb/
                  - ./scripts:/scripts/
                  - ./data/database1:/data/db/
                command: /bin/bash /scripts/database_init.sh Shard1
                networks:
                        antithesis-net:
                                ipv4_address: 10.20.20.3
                                # Set the an IPv4 with an address of 10.20.20.130 or higher
                                # to be ignored by the fault injector
                                #
        database2: ...
        database3: ...
        database4: ...
        database5: ...
        database6: ...
        mongos:
                container_name: mongos
                hostname: mongos
                image: mongo-binaries:evergreen-latest-master
                volumes:
                  - ./logs/mongos:/var/log/mongodb/
                  - ./scripts:/scripts/
                command: python3 /scripts/mongos_init.py
                depends_on:
                        - "database1"
                        - "database2"
                        - "database3"
                        - "database4"
                        - "database5"
                        - "database6"
                        - "configsvr1"
                        - "configsvr2"
                        - "configsvr3"
                networks:
                        antithesis-net:
                                ipv4_address: 10.20.20.9
                                # The subnet provided here is an example
                                # An alternative subnet can be used
        workload:
                container_name: workload
                hostname: workload
                image: workload:evergreen-latest-master
                volumes:
                  - ./logs/workload:/var/log/resmoke/
                  - ./scripts:/scripts/
                command: python3 /scripts/workload_init.py
                depends_on:
                        - "mongos"
                networks:
                        antithesis-net:
                                ipv4_address: 10.20.20.130
                                # The subnet provided here is an example
                                # An alternative subnet can be used
networks:
        antithesis-net:
                driver: bridge
                ipam:
                        config:
                        - subnet: 10.20.20.0/24
```

Each container must have a `command` in `docker-compose.yml` that runs an init script. The init
script belongs in the `scripts` directory, which is included as a volume. The `command` should be
set like so: `/bin/bash /scripts/[script_name].sh` or `python3 /scripts/[script_name].py`. This is
a requirement for the topology to start up properly in Antithesis.

When creating `mongod` or `mongos` instances, route the logs like so:
`--logpath /var/log/mongodb/mongodb.log` and utilize `volumes` -- as in `database1`.
This enables us to easily retrieve logs if a bug is detected by Antithesis.

The `ipv4_address` should be set to `10.20.20.130` or higher if you do not want that container to
be affected by network fuzzing. For instance, you would likely not want the `workload` container  
to be affected by network fuzzing -- as shown in the example above.

Use the `evergreen-latest-master` tag for all images. This is updated automatically in
`evergreen/antithesis_image_build_and_push.sh` -- if needed.

### scripts

Take a look at `buildscripts/antithesis/topologies/sharded_cluster/scripts/mongos_init.py` to see
how to use util methods from `buildscripts/antithesis/topologies/sharded_cluster/scripts/utils.py`
to set up the desired topology. You can also use simple shell scripts as in the case of
`buildscripts/antithesis/topologies/sharded_cluster/scripts/database_init.py`. These init scripts
must not end in order to keep the underlying container alive. You can use an infinite while
loop for `python` scripts or you can use `tail -f /dev/null` for shell scripts.

## How do I create a new topology for Antithesis testing?

This should be done with care to ensure we are using our limited resources efficiently.

Create a new task extending the `antithesis_task_template`, tagged with `antithesis`, passing the specified `suite` to the `antithesis image build and push` task. See other examples to get started.

## Additional Resources

If you are interested in leveraging Antithesis feel free to reach out to #server-testing on Slack.