Merge pull request #76 from flashbots/readme-kube-4

Update README to use Kubernetes setup - Remove docker compose

.dockerignore

@@ -1,2 +1 @@
-tests
 cache

.env

@@ -1,8 +0,0 @@
-# Postgres
-POSTGRES_SERVER=db
-POSTGRES_USER=postgres
-POSTGRES_PASSWORD=password
-POSTGRES_DB=mev_inspect
-
-# SQLAlchemy
-SQLALCHEMY_DATABASE_URI=postgresql://$POSTGRES_USER:$POSTGRES_PASSWORD@$POSTGRES_SERVER/$POSTGRES_DB

.github/workflows/github-actions.yml

@@ -55,4 +55,4 @@ jobs:
 
       - name: Test with pytest
         shell: bash
-        run: poetry run test
+        run: poetry run pytest --cov=mev_inspect tests

README.md

@@ -1,104 +1,143 @@
-# mev-inspect
+# mev-inspect-py
-A [WIP] Ethereum MEV Inspector in Python managed by Poetry
+> illuminating the dark forest 🌲🔦
 
-## Containers
+**mev-inspect-py** is an MEV inspector for Ethereum
-mev-inspect's local setup is built on [Docker Compose](https://docs.docker.com/compose/)
 
-By default it starts up:
+Given a block, mev-inspect finds:
-- `mev-inspect` - a container with the code in this repo used for running scripts
+- miner payments (gas + coinbase)
-- `db` - a postgres database instance
+- tokens transfers and profit
-- `pgadmin` - a postgres DB UI for querying and more (avaiable at localhost:5050)
+- swaps and [arbitrages](https://twitter.com/bertcmiller/status/142763202826305946://twitter.com/bertcmiller/status/1427632028263059462)
+- ...and more
+
+Data is stored in Postgres for analysis
 
 ## Running locally
-Setup [Docker](https://www.docker.com/products/docker-desktop)
+mev-inspect-py is built to run on kubernetes locally and in production
-Setup [Poetry](https://python-poetry.org/docs/#osx--linux--bashonwindows-install-instructions)
 
-Install dependencies through poetry
+### Install dependencies
+
+1. Setup a local kubernetes deployment (we use [kind](https://kind.sigs.k8s.io/docs/user/quick-start))
+
+2. Setup [Tilt](https://docs.tilt.dev/install.html) which manages the local deployment
+
+### Start up
+
+Set an environment variable `RPC_URL` to an RPC for fetching blocks
+Example:
+```
+export RPC_URL="http://111.111.111.111:8546"
+```
+
+**Note: mev-inspect-py currently requires and RPC with support for parity traces**
+
+Next, start all servcies with:
+```
+tilt up
+```
+
+Press "space" to see a browser of the services starting up
+
+On first startup, you'll need to apply database migrations. Apply with:
+```
+kubectl exec deploy/mev-inspect-deployment -- alembic upgrade head
+```
+
+## Inspecting
+
+### Inspect a single block
+
+Inspecting block [12914944](https://twitter.com/mevalphaleak/status/1420416437575901185)
+```
+kubectl exec deploy/mev-inspect-deployment -- poetry run inspect-block 12914944
+```
+
+### Inspect many blocks
+
+Inspecting blocks 12914944 to 12914954
+```
+kubectl exec deploy/mev-inspect-deployment -- poetry run inspect-many-blocks 12914944 12914954
+```
+
+### Inspect all incoming blocks
+
+Start a block listener with
+```
+kubectl exec deploy/mev-inspect-deployment -- /app/listener start
+```
+
+By default, it will pick up wherever you left off.
+If running for the first time, listener starts at the latest block
+
+See logs for the listener with
+```
+kubectl exec deploy/mev-inspect-deployment -- tail -f listener.log
+```
+
+And stop the listener with
+```
+kubectl exec deploy/mev-inspect-deployment -- /app/listener stop
+```
+
+## Contributing
+
+### Guide
+
+✨ Coming soon
+
+### Pre-commit
+
+We use pre-commit to maintain a consistent style, prevent errors, and ensure test coverage. 
+
+To set up, install dependencies through poetry
 ```
 poetry install
 ```
 
-Start the services (optionally as daemon)
+Then install pre-commit hooks with
-```
-poetry run start [-d]
-```
-
-Apply the latest migrations against the local DB:
-```
-poetry run exec alembic upgrade head
-``` 
-
-Run inspect on a block
-```
-poetry run inspect -b/--block-number 11931270 -r/--rpc 'http://111.11.11.111:8545/'
-``` 
-
-To stop the services (if running in the background, otherwise just ctrl+c)
-```
-poetry run stop
-```
-
-MEV container can be attached via
-```
-poetry run attach
-```
-
-Running additional compose commands are possible through standard `docker
-compose ...` calls.  Check `docker compose help` for more tools available
-
-## Executing scripts
-Any script can be run from the mev-inspect container like
-```
-poetry run exec <your command here>
-```
-
-For example
-```
-poetry run exec python examples/uniswap_inspect.py -block_number=123 -rpc='111.111.111'
-```
-
-### Poetry Scripts
-```bash
-# code check
-poetry run lint # linting via Pylint
-poetry run test # testing and code coverage with Pytest
-poetry run isort # fixing imports 
-poetry run mypy # type checking 
-poetry run black # style guide 
-poetry run pre-commit run --all-files # runs Black, PyLint and MyPy
-# docker management
-poetry run start [-d] # starts all services, optionally as a daemon
-poetry run stop # shutsdown all services or just ctrl + c if foreground
-poetry run build # rebuilds containers
-poetry run attach # enters the mev-inspect container in interactive mode
-# launches inspection script
-poetry run inspect -b/--block-number 11931270 -r/--rpc 'http://111.11.11.111:8545/'
-```
-
-
-## Rebuilding containers
-After changes to the app's Dockerfile, rebuild with
-```
-poetry run build
-```
-
-## Using PGAdmin
-
-1. Go to [localhost:5050](localhost:5050)
-
-2. Login with the PGAdmin username and password in `.env`
-
-3. Add a new engine for mev_inspect with
-    - host: db
-    - user / password: see `.env`
-
-## Contributing
-
-Pre-commit is used to maintain a consistent style, prevent errors and ensure test coverage. 
-
-Install pre-commit with:
 ```
 poetry run pre-commit install
 ```
 
-Update README if needed
+### Tests
+
+Run tests with
+```
+kubectl exec deploy/mev-inspect-deployment -- poetry run pytest --cov=mev_inspect tests
+```
+
+## FAQ
+
+### How do I delete / reset my local postgres data?
+
+Stop the system if running
+```
+tilt down
+```
+
+Delete it with
+```
+kubectl delete pvc data-postgresql-postgresql-0
+```
+
+Start back up again
+```
+tilt up
+```
+
+And rerun migrations to create the tables again
+```
+kubectl exec deploy/mev-inspect-deployment -- alembic upgrade head
+```
+
+### I was using the docker-compose setup and want to switch to kube, now what?
+
+Re-add the old `docker-compose.yml` file to your mev-inspect-py directory
+
+A copy can be found [here](https://github.com/flashbots/mev-inspect-py/blob/ef60c097719629a7d2dc56c6e6c9a100fb706f76/docker-compose.yml)
+
+Tear down docker-compose resources
+```
+docker compose down
+```
+
+Then go through the steps in the current README for kube setup

cli.py

@@ -1,3 +1,5 @@
+import os
+
 import click
 from web3 import Web3
 
@@ -6,6 +8,9 @@ from mev_inspect.inspect_block import inspect_block
 from mev_inspect.provider import get_base_provider
 
 
+RPC_URL_ENV = "RPC_URL"
+
+
 @click.group()
 def cli():
     pass
@@ -13,7 +18,7 @@ def cli():
 
 @cli.command()
 @click.argument("block_number", type=int)
-@click.argument("rpc")
+@click.option("--rpc", default=lambda: os.environ.get(RPC_URL_ENV, ""))
 @click.option("--cache/--no-cache", default=True)
 def inspect_block_command(block_number: int, rpc: str, cache: bool):
     db_session = get_session()
@@ -29,7 +34,7 @@ def inspect_block_command(block_number: int, rpc: str, cache: bool):
 @cli.command()
 @click.argument("after_block", type=int)
 @click.argument("before_block", type=int)
-@click.argument("rpc")
+@click.option("--rpc", default=lambda: os.environ.get(RPC_URL_ENV, ""))
 @click.option("--cache/--no-cache", default=True)
 def inspect_many_blocks_command(
     after_block: int, before_block: int, rpc: str, cache: bool
@@ -61,5 +66,9 @@ def inspect_many_blocks_command(
         )
 
 
+def get_rpc_url() -> str:
+    return os.environ["RPC_URL"]
+
+
 if __name__ == "__main__":
     cli()

docker-compose.yml

@@ -1,24 +0,0 @@
-services:
-  mev-inspect:
-    build: .
-    depends_on:
-      - db
-    env_file:
-      - .env
-    volumes:
-      - .:/app
-    tty: true
-
-  db:
-    image: postgres:12
-    volumes:
-      - mev-inspect-db-data:/var/lib/postgresql/data/pgdata
-    env_file:
-      - .env
-    environment:
-      - PGDATA=/var/lib/postgresql/data/pgdata
-    ports:
-      - 5432:5432
-
-volumes:
-  mev-inspect-db-data:

mev_inspect/config.ini

@@ -1,7 +0,0 @@
-[RPC]
-Endpoint = http://localhost:8545/
-
-[ADDRESSES]
-UniswapV2Router = 0x7a250d5630B4cF539739dF2C5dAcb4c659F2488D
-SushiswapV2Router = 0xd9e1cE17f2641f24aE83637ab66a2cca9C378B9F
-WETH = 0xc02aaa39b223fe8d0a0e5c4f27ead9083c756cc2

mev_inspect/config.py

@@ -1,13 +0,0 @@
-import os
-import configparser
-
-
-THIS_FILE_DIRECTORY = os.path.dirname(__file__)
-CONFIG_PATH = os.path.join(THIS_FILE_DIRECTORY, "config.ini")
-
-
-def load_config():
-    config = configparser.ConfigParser()
-    config.read(CONFIG_PATH)
-
-    return config

mev_inspect/tokenflow.py

@@ -1,13 +1,8 @@
 from typing import List, Optional
 
-from mev_inspect.config import load_config
 from mev_inspect.schemas import Block, Trace, TraceType
 
-config = load_config()
+weth_address = "0xc02aaa39b223fe8d0a0e5c4f27ead9083c756cc2"
-
-rpc_url = config["RPC"]["Endpoint"]
-weth_address = config["ADDRESSES"]["WETH"]
-# w3 = Web3(HTTPProvider(rpc_url))
 
 cache_directory = "./cache"
 

pyproject.toml

@@ -29,20 +29,8 @@ requires = ["poetry-core>=1.0.0"]
 build-backend = "poetry.core.masonry.api"
 
 [tool.poetry.scripts]
-lint = 'scripts.poetry.dev_tools:lint'
+inspect-block = 'cli:inspect_block_command'
-test = 'scripts.poetry.dev_tools:test'
+inspect-many-blocks = 'cli:inspect_many_blocks_command'
-isort = 'scripts.poetry.dev_tools:isort'
-black = 'scripts.poetry.dev_tools:black'
-pre_commit = 'scripts.poetry.dev_tools:pre_commit'
-start = 'scripts.poetry.docker:start'
-stop = 'scripts.poetry.docker:stop'
-build = 'scripts.poetry.docker:build'
-attach = 'scripts.poetry.docker:attach'
-exec = 'scripts.poetry.docker:exec'
-inspect = 'scripts.poetry.inspect:inspect'
-inspect-many = 'scripts.poetry.inspect:inspect_many'
-inspect-block = 'scripts.inspect_commands:inspect_block_command'
-inspect-many-blocks = 'scripts.inspect_commands:inspect_many_blocks_command'
 
 [tool.black]
 exclude = '''

scripts/poetry/dev_tools.py

@@ -1,34 +0,0 @@
-from subprocess import check_call
-import click
-
-
-def lint():
-    check_call(["pylint", "."])
-
-
-def test():
-    check_call(["pytest", "--cov=mev_inspect", "tests"])
-
-
-@click.command()
-@click.option("-c", required=False, is_flag=True)
-def isort(c: str):
-    """if c is present run isort in diff mode"""
-    if c:
-        check_call(["isort", "."])
-    else:
-        check_call(["isort", "--diff", "."])
-
-
-def mypy():
-    check_call(["mypy", "."])
-
-
-@click.command()
-@click.option("-c", required=False, is_flag=True)
-def black(c: str = None):
-    """if c is present run black in diff mode"""
-    if c:
-        check_call(["black", "--diff", "--color", "."])
-    else:
-        check_call(["black", "."])

scripts/poetry/docker.py

@@ -1,33 +0,0 @@
-from subprocess import check_call
-from typing import List
-
-import click
-
-
-@click.command()
-@click.option("-d", required=False, is_flag=True)
-def start(d: str):
-    """if d is present, run docker compose as daemon"""
-    if d:
-        check_call(["docker-compose", "up", "-d"])
-        click.echo("docker running in the background...")
-    else:
-        check_call(["docker-compose", "up"])
-
-
-def stop():
-    check_call(["docker-compose", "down"])
-
-
-def build():
-    check_call(["docker-compose", "build"])
-
-
-def attach():
-    check_call(["docker", "exec", "-it", "mev-inspect-py_mev-inspect_1", "bash"])
-
-
-@click.command()
-@click.argument("args", nargs=-1)
-def exec(args: List[str]):
-    check_call(["docker-compose", "exec", "mev-inspect", *args])

scripts/poetry/inspect.py

@@ -1,59 +0,0 @@
-from subprocess import check_call
-
-import click
-
-
-@click.command()
-@click.option(
-    "-b", "--block-number", type=str, help="the block number you are targetting"
-)
-@click.option(
-    "-r", "--rpc", help="rpc endpoint, this needs to have parity style traces"
-)
-@click.option(
-    "--cache/--no-cache",
-    help="whether to read / write to the cache",
-    default=True,
-)
-def inspect(block_number: str, rpc: str, cache: bool):
-    check_call(
-        [
-            "docker",
-            "compose",
-            "exec",
-            "mev-inspect",
-            "python",
-            "./scripts/inspect_block.py",
-            "inspect-block",
-            block_number,
-            rpc,
-            "--cache" if cache else "--no-cache",
-        ]
-    )
-
-
-@click.command()
-@click.argument("after_block", type=str)
-@click.argument("before_block", type=str)
-@click.argument("rpc")
-@click.option(
-    "--cache/--no-cache",
-    help="whether to read / write to the cache",
-    default=True,
-)
-def inspect_many(after_block: str, before_block: str, rpc: str, cache: bool):
-    check_call(
-        [
-            "docker",
-            "compose",
-            "exec",
-            "mev-inspect",
-            "python",
-            "./scripts/inspect_block.py",
-            "inspect-many-blocks",
-            after_block,
-            before_block,
-            rpc,
-            "--cache" if cache else "--no-cache",
-        ]
-    )