From 8e42bede105fa979cbc35b377cc50064c9328911 Mon Sep 17 00:00:00 2001 From: Leo Arias Date: Tue, 12 Oct 2021 17:22:01 +0000 Subject: [PATCH 1/7] Prettify the README. --- CONTRIBUTING.md | 36 ++++++++++++ README.md | 153 +++++++++++++++++++++++++++--------------------- 2 files changed, 121 insertions(+), 68 deletions(-) create mode 100644 CONTRIBUTING.md diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000..e7a3828 --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,36 @@ +# Contributing guide + +Welcome to the Flashbots collective! We just ask you to be nice when you play with us. + +## Pre-commit + +We use pre-commit to maintain a consistent style, prevent errors, and ensure test coverage. + +To set up, install dependencies through `poetry`: + +``` +poetry install +``` + +Then install pre-commit hooks with: + +``` +poetry run pre-commit install +``` + +## Tests + +Run tests with: + +``` +kubectl exec deploy/mev-inspect-deployment -- poetry run pytest --cov=mev_inspect tests +``` + +## Send a pull request + +- Your proposed changes should be first described and discussed in an issue. +- Open the branch in a personal fork, not in the team repository. +- Every pull request should be small and represent a single change. If the problem is complicated, split it in multiple issues and pull requests. +- Every pull request should be covered by unit tests. + +We appreciate you, friend <3. diff --git a/README.md b/README.md index 221a5d5..318893b 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,9 @@ # mev-inspect-py -> illuminating the dark forest 🌲💡 -**mev-inspect-py** is an MEV inspector for Ethereum +[![standard-readme compliant](https://img.shields.io/badge/readme%20style-standard-brightgreen.svg?style=flat-square)](https://github.com/RichardLitt/standard-readme) +![Discord](https://img.shields.io/discord/755466764501909692) + +[Maximmal extractable value](https://ethereum.org/en/developers/docs/mev/) inspector for Ethereum, to illuminate the [dark forest](https://www.paradigm.xyz/2020/08/ethereum-is-a-dark-forest/) 🌲💡 Given a block, mev-inspect finds: - miner payments (gas + coinbase) @@ -9,106 +11,118 @@ Given a block, mev-inspect finds: - swaps and [arbitrages](https://twitter.com/bertcmiller/status/1427632028263059462) - ...and more -Data is stored in Postgres for analysis +Data is stored in Postgres for analysis. -## Running locally -mev-inspect-py is built to run on kubernetes locally and in production +## Install -### Install dependencies +mev-inspect-py is built to run on kubernetes locally and in production. -First, setup a local kubernetes deployment - we use [Docker](https://www.docker.com/products/docker-desktop) and [kind](https://kind.sigs.k8s.io/docs/user/quick-start) +### Dependencies + +- [docker](https://www.docker.com/products/docker-desktop) +- [kind](https://kind.sigs.k8s.io/docs/user/quick-start) +- [kubectl](https://kubernetes.io/docs/tasks/tools/) +- [helm](https://helm.sh/docs/intro/install/) + +### Set up + +Ceate a new cluster with: -If using kind, create a new cluster with: ``` kind create cluster ``` -Next, install the kubernetes CLI [`kubectl`](https://kubernetes.io/docs/tasks/tools/) +Set an environment variable `RPC_URL` to an RPC for fetching blocks. -Then, install [helm](https://helm.sh/docs/intro/install/) - helm is a package manager for kubernetes - -Lastly, setup [Tilt](https://docs.tilt.dev/install.html) which manages running and updating kubernetes resources locally - -### Start up - -Set an environment variable `RPC_URL` to an RPC for fetching blocks Example: + ``` export RPC_URL="http://111.111.111.111:8546" ``` -**Note: mev-inspect-py currently requires an RPC with support for Erigon traces and receipts (not geth 😔)** +**Note: mev-inspect-py currently requires an RPC of a full archive node with support for Erigon traces and receipts (not geth 😔)** Next, start all services with: + ``` tilt up ``` -Press "space" to see a browser of the services starting up +Press "space" to see a browser of the services starting up. + +On first startup, you'll need to apply database migrations with: -On first startup, you'll need to apply database migrations. Apply with: ``` kubectl exec deploy/mev-inspect -- alembic upgrade head ``` -## Inspecting +## Usage ### Inspect a single block -Inspecting block [12914944](https://twitter.com/mevalphaleak/status/1420416437575901185) +Inspecting block [12914944](https://twitter.com/mevalphaleak/status/1420416437575901185): + ``` kubectl exec deploy/mev-inspect -- poetry run inspect-block 12914944 ``` ### Inspect many blocks -Inspecting blocks 12914944 to 12914954 +Inspecting blocks 12914944 to 12914954: + ``` kubectl exec deploy/mev-inspect -- poetry run inspect-many-blocks 12914944 12914954 ``` ### Inspect all incoming blocks -Start a block listener with +Start a block listener with: + ``` kubectl exec deploy/mev-inspect -- /app/listener start ``` By default, it will pick up wherever you left off. -If running for the first time, listener starts at the latest block +If running for the first time, listener starts at the latest block. + +See logs for the listener with: -See logs for the listener with ``` kubectl exec deploy/mev-inspect -- tail -f listener.log ``` -And stop the listener with +And stop the listener with: + ``` kubectl exec deploy/mev-inspect -- /app/listener stop ``` -## Exploring +### Exploring All inspect output data is stored in Postgres. To connect to the local Postgres database for querying, launch a client container with: + ``` kubectl run -i --rm --tty postgres-client --env="PGPASSWORD=password" --image=jbergknoff/postgresql-client -- mev_inspect --host=postgresql --user=postgres ``` -When you see the prompt +When you see the prompt: + ``` mev_inspect=# ``` You're ready to query! -Try finding the total number of swaps decoded with UniswapV3Pool +Try finding the total number of swaps decoded with UniswapV3Pool: + ``` SELECT COUNT(*) FROM swaps WHERE abi_name='UniswapV3Pool'; ``` -or top 10 arbs by gross profit that took profit in WETH +or top 10 arbs by gross profit that took profit in WETH: + ``` SELECT * FROM arbitrages @@ -117,78 +131,81 @@ ORDER BY profit_amount DESC LIMIT 10; ``` -Postgres tip: Enter `\x` to enter "Explanded display" mode which looks nicer for results with many columns - -## Contributing - -### Guide - -✨ Coming soon - -### Pre-commit - -We use pre-commit to maintain a consistent style, prevent errors, and ensure test coverage. - -To set up, install dependencies through poetry -``` -poetry install -``` - -Then install pre-commit hooks with -``` -poetry run pre-commit install -``` - -### Tests - -Run tests with -``` -kubectl exec deploy/mev-inspect -- poetry run pytest --cov=mev_inspect tests -``` +Postgres tip: Enter `\x` to enter "Explanded display" mode which looks nicer for results with many columns. ## FAQ ### How do I delete / reset my local postgres data? -Stop the system if running +Stop the system if running: + ``` tilt down ``` -Delete it with +Delete it with: + ``` kubectl delete pvc data-postgresql-postgresql-0 ``` -Start back up again +Start back up again: + ``` tilt up ``` -And rerun migrations to create the tables again +And rerun migrations to create the tables again: + ``` kubectl exec deploy/mev-inspect -- alembic upgrade head ``` ### I was using the docker-compose setup and want to switch to kube, now what? -Re-add the old `docker-compose.yml` file to your mev-inspect-py directory +Re-add the old `docker-compose.yml` file to your mev-inspect-py directory. A copy can be found [here](https://github.com/flashbots/mev-inspect-py/blob/ef60c097719629a7d2dc56c6e6c9a100fb706f76/docker-compose.yml) -Tear down docker-compose resources +Tear down docker-compose resources: + ``` docker compose down ``` -Then go through the steps in the current README for kube setup +Then go through the steps in the current README for kube setup. ### Error from server (AlreadyExists): pods "postgres-client" already exists -This means the postgres client container didn't shut down correctly -Delete this one with +This means the postgres client container didn't shut down correctly. + +Delete this one with: + ``` kubectl delete pod/postgres-client ``` -Then start it back up again +Then start it back up again. + +## Maintainers + +- [@lukevs](https://github.com/lukevs) + +## Contributing + +[Flashbots](https://flashbots.net) is a research and development collective working on mitigating the negative externalities of decentralized economies. We contribute with the larger free software community to illuminate the dark forest. + +You are welcome here <3. + +- If you want to join us, come and say hi in our [Discord chat](https://discord.gg/7hvTycdNcK). +- If you have a question, feedback or a bug report for this project, please [open a new Issue](https://github.com/flashbots/flashbots-repository-template/issues). +- If you would like to contribute with code, check the [CONTRIBUTING file](CONTRIBUTING.md). +- We just ask you to be nice. + +## Security + +If you find a security vulnerability on this project or any other initiative related to Flashbots, please let us know sending an email to security@flashbots.net. + +--- + +Made with ☀️ by the ⚡🤖 collective. From f3687c9102e278fc1df6e6d1bdb8959a755506f5 Mon Sep 17 00:00:00 2001 From: Leo Arias Date: Tue, 12 Oct 2021 17:23:50 +0000 Subject: [PATCH 2/7] Fix typo --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 318893b..fe17de3 100644 --- a/README.md +++ b/README.md @@ -3,7 +3,7 @@ [![standard-readme compliant](https://img.shields.io/badge/readme%20style-standard-brightgreen.svg?style=flat-square)](https://github.com/RichardLitt/standard-readme) ![Discord](https://img.shields.io/discord/755466764501909692) -[Maximmal extractable value](https://ethereum.org/en/developers/docs/mev/) inspector for Ethereum, to illuminate the [dark forest](https://www.paradigm.xyz/2020/08/ethereum-is-a-dark-forest/) 🌲💡 +[Maximal extractable value](https://ethereum.org/en/developers/docs/mev/) inspector for Ethereum, to illuminate the [dark forest](https://www.paradigm.xyz/2020/08/ethereum-is-a-dark-forest/) 🌲💡 Given a block, mev-inspect finds: - miner payments (gas + coinbase) From 53a1afd5f779110c8d5e8b17679b987869aad0de Mon Sep 17 00:00:00 2001 From: Leo Arias Date: Tue, 12 Oct 2021 17:24:57 +0000 Subject: [PATCH 3/7] Improve format --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index fe17de3..480f9fb 100644 --- a/README.md +++ b/README.md @@ -40,7 +40,7 @@ Example: export RPC_URL="http://111.111.111.111:8546" ``` -**Note: mev-inspect-py currently requires an RPC of a full archive node with support for Erigon traces and receipts (not geth 😔)** +**Note**: mev-inspect-py currently requires an RPC of a full archive node with support for Erigon traces and receipts (not geth 😔). Next, start all services with: From c36e2445af949df6de3733937ab2db2b598ce3c3 Mon Sep 17 00:00:00 2001 From: Leo Arias Date: Tue, 12 Oct 2021 18:47:16 +0000 Subject: [PATCH 4/7] Link the badge to discord --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 480f9fb..a9b0a1e 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,7 @@ # mev-inspect-py [![standard-readme compliant](https://img.shields.io/badge/readme%20style-standard-brightgreen.svg?style=flat-square)](https://github.com/RichardLitt/standard-readme) -![Discord](https://img.shields.io/discord/755466764501909692) +[![Discord](https://img.shields.io/discord/755466764501909692)](https://discord.gg/7hvTycdNcK) [Maximal extractable value](https://ethereum.org/en/developers/docs/mev/) inspector for Ethereum, to illuminate the [dark forest](https://www.paradigm.xyz/2020/08/ethereum-is-a-dark-forest/) 🌲💡 From a4c21b765db0749fdb27c3704ed9093e76b44191 Mon Sep 17 00:00:00 2001 From: Leo Arias Date: Tue, 12 Oct 2021 18:48:13 +0000 Subject: [PATCH 5/7] Fix the issues link --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index a9b0a1e..882992e 100644 --- a/README.md +++ b/README.md @@ -198,7 +198,7 @@ Then start it back up again. You are welcome here <3. - If you want to join us, come and say hi in our [Discord chat](https://discord.gg/7hvTycdNcK). -- If you have a question, feedback or a bug report for this project, please [open a new Issue](https://github.com/flashbots/flashbots-repository-template/issues). +- If you have a question, feedback or a bug report for this project, please [open a new Issue](https://github.com/flashbots/mev-inspect-py/issues). - If you would like to contribute with code, check the [CONTRIBUTING file](CONTRIBUTING.md). - We just ask you to be nice. From ad4acfa0438ca4bde4b4d3ec45ea71f357010109 Mon Sep 17 00:00:00 2001 From: Leo Arias Date: Tue, 19 Oct 2021 16:37:42 +0000 Subject: [PATCH 6/7] Add maintainers --- README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/README.md b/README.md index 882992e..af83a5c 100644 --- a/README.md +++ b/README.md @@ -190,6 +190,8 @@ Then start it back up again. ## Maintainers - [@lukevs](https://github.com/lukevs) +- [@gheise](https://github.com/gheise) +- [@bertmiller](https://github.com/bertmiller) ## Contributing From 60f1a651bb5c2e9c791210dc01ecc3ee669cfa1b Mon Sep 17 00:00:00 2001 From: Leo Arias Date: Tue, 19 Oct 2021 16:47:35 +0000 Subject: [PATCH 7/7] Apply review feedback --- README.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index af83a5c..bef5711 100644 --- a/README.md +++ b/README.md @@ -20,13 +20,14 @@ mev-inspect-py is built to run on kubernetes locally and in production. ### Dependencies - [docker](https://www.docker.com/products/docker-desktop) -- [kind](https://kind.sigs.k8s.io/docs/user/quick-start) +- [kind](https://kind.sigs.k8s.io/docs/user/quick-start), or a similar tool for running local Kubernetes clusters - [kubectl](https://kubernetes.io/docs/tasks/tools/) - [helm](https://helm.sh/docs/intro/install/) +- [tilt](https://docs.tilt.dev/install.html) ### Set up -Ceate a new cluster with: +Create a new cluster with: ``` kind create cluster