Use inspector class -- remove global Semaphore and improve error handling

CONTRIBUTING.md

@@ -0,0 +1,36 @@
+# Contributing guide
+
+Welcome to the Flashbots collective! We just ask you to be nice when you play with us.
+
+## Pre-commit
+
+We use pre-commit to maintain a consistent style, prevent errors, and ensure test coverage.
+
+To set up, install dependencies through `poetry`:
+
+```
+poetry install
+```
+
+Then install pre-commit hooks with:
+
+```
+poetry run pre-commit install
+```
+
+## Tests
+
+Run tests with:
+
+```
+kubectl exec deploy/mev-inspect-deployment -- poetry run pytest --cov=mev_inspect tests
+```
+
+## Send a pull request
+
+- Your proposed changes should be first described and discussed in an issue.
+- Open the branch in a personal fork, not in the team repository.
+- Every pull request should be small and represent a single change. If the problem is complicated, split it in multiple issues and pull requests.
+- Every pull request should be covered by unit tests.
+
+We appreciate you, friend <3.

README.md

@@ -1,7 +1,9 @@
 # mev-inspect-py
-> illuminating the dark forest 🌲💡
 
-**mev-inspect-py** is an MEV inspector for Ethereum
+[![standard-readme compliant](https://img.shields.io/badge/readme%20style-standard-brightgreen.svg?style=flat-square)](https://github.com/RichardLitt/standard-readme)
+[![Discord](https://img.shields.io/discord/755466764501909692)](https://discord.gg/7hvTycdNcK)
+
+[Maximal extractable value](https://ethereum.org/en/developers/docs/mev/) inspector for Ethereum, to illuminate the [dark forest](https://www.paradigm.xyz/2020/08/ethereum-is-a-dark-forest/) 🌲💡
 
 Given a block, mev-inspect finds:
 - miner payments (gas + coinbase)
@@ -9,106 +11,141 @@ Given a block, mev-inspect finds:
 - swaps and [arbitrages](https://twitter.com/bertcmiller/status/1427632028263059462)
 - ...and more
 
-Data is stored in Postgres for analysis
+Data is stored in Postgres for analysis.
 
-## Running locally
-mev-inspect-py is built to run on kubernetes locally and in production
+## Install
 
-### Install dependencies
+mev-inspect-py is built to run on kubernetes locally and in production.
 
-First, setup a local kubernetes deployment - we use [Docker](https://www.docker.com/products/docker-desktop) and [kind](https://kind.sigs.k8s.io/docs/user/quick-start)
+### Dependencies
+
+- [docker](https://www.docker.com/products/docker-desktop)
+- [kind](https://kind.sigs.k8s.io/docs/user/quick-start), or a similar tool for running local Kubernetes clusters
+- [kubectl](https://kubernetes.io/docs/tasks/tools/)
+- [helm](https://helm.sh/docs/intro/install/)
+- [tilt](https://docs.tilt.dev/install.html)
+
+### Set up
+
+Create a new cluster with:
 
-If using kind, create a new cluster with:
 ```
 kind create cluster
 ```
 
-Next, install the kubernetes CLI [`kubectl`](https://kubernetes.io/docs/tasks/tools/)
+Set an environment variable `RPC_URL` to an RPC for fetching blocks.
 
-Then, install [helm](https://helm.sh/docs/intro/install/) - helm is a package manager for kubernetes
-
-Lastly, setup [Tilt](https://docs.tilt.dev/install.html) which manages running and updating kubernetes resources locally
-
-### Start up
-
-Set an environment variable `RPC_URL` to an RPC for fetching blocks
 Example:
+
 ```
 export RPC_URL="http://111.111.111.111:8546"
 ```
 
-**Note: mev-inspect-py currently requires an RPC with support for Erigon traces and receipts (not geth 😔)**
+**Note**: mev-inspect-py currently requires an RPC of a full archive node with support for Erigon traces and receipts (not geth 😔).
 
 Next, start all services with:
+
 ```
 tilt up
 ```
 
-Press "space" to see a browser of the services starting up
+Press "space" to see a browser of the services starting up.
+
+On first startup, you'll need to apply database migrations with:
 
-On first startup, you'll need to apply database migrations. Apply with:
 ```
 kubectl exec deploy/mev-inspect -- alembic upgrade head
 ```
 
-## Inspecting
+## Usage
 
 ### Inspect a single block
 
-Inspecting block [12914944](https://twitter.com/mevalphaleak/status/1420416437575901185)
+Inspecting block [12914944](https://twitter.com/mevalphaleak/status/1420416437575901185):
+
 ```
-kubectl exec deploy/mev-inspect -- poetry run inspect-block 12914944
+./mev inspect 12914944
 ```
 
 ### Inspect many blocks
 
-Inspecting blocks 12914944 to 12914954
+Inspecting blocks 12914944 to 12914954:
+
 ```
-kubectl exec deploy/mev-inspect -- poetry run inspect-many-blocks 12914944 12914954
+./mev inspect-many 12914944 12914954
 ```
 
 ### Inspect all incoming blocks
 
-Start a block listener with
+Start a block listener with:
+
 ```
-kubectl exec deploy/mev-inspect -- /app/listener start
+./mev listener start
 ```
 
 By default, it will pick up wherever you left off.
-If running for the first time, listener starts at the latest block
+If running for the first time, listener starts at the latest block.
+
+Tail logs for the listener with:
 
-See logs for the listener with
 ```
-kubectl exec deploy/mev-inspect -- tail -f listener.log
+./mev listener tail
 ```
 
-And stop the listener with
+And stop the listener with:
+
 ```
-kubectl exec deploy/mev-inspect -- /app/listener stop
+./mev listener stop
 ```
 
-## Exploring
+### Backfilling
+
+For larger backfills, you can inspect many blocks in parallel using kubernetes
+
+To inspect blocks 12914944 to 12915044 divided across 10 worker pods:
+```
+./mev backfill 12914944 12915044 10
+```
+
+You can see worker pods spin up then complete by watching the status of all pods
+```
+watch kubectl get pods
+```
+
+To watch the logs for a given pod, take its pod name using the above, then run:
+```
+kubectl logs -f pod/mev-inspect-backfill-abcdefg
+```
+
+(where `mev-inspect-backfill-abcdefg` is your actual pod name)
+
+
+### Exploring
 
 All inspect output data is stored in Postgres.
 
 To connect to the local Postgres database for querying, launch a client container with:
+
 ```
-kubectl run -i --rm --tty postgres-client --env="PGPASSWORD=password" --image=jbergknoff/postgresql-client -- mev_inspect --host=postgresql --user=postgres
+./mev db
 ```
 
-When you see the prompt
+When you see the prompt:
+
 ```
 mev_inspect=#
 ```
 
 You're ready to query!
 
-Try finding the total number of swaps decoded with UniswapV3Pool
+Try finding the total number of swaps decoded with UniswapV3Pool:
+
 ```
 SELECT COUNT(*) FROM swaps WHERE abi_name='UniswapV3Pool';
 ```
 
-or top 10 arbs by gross profit that took profit in WETH
+or top 10 arbs by gross profit that took profit in WETH:
+
 ```
 SELECT *
 FROM arbitrages
@@ -117,78 +154,83 @@ ORDER BY profit_amount DESC
 LIMIT 10;
 ```
 
-Postgres tip: Enter `\x` to enter "Explanded display" mode which looks nicer for results with many columns
-
-## Contributing
-
-### Guide
-
-✨ Coming soon
-
-### Pre-commit
-
-We use pre-commit to maintain a consistent style, prevent errors, and ensure test coverage.
-
-To set up, install dependencies through poetry
-```
-poetry install
-```
-
-Then install pre-commit hooks with
-```
-poetry run pre-commit install
-```
-
-### Tests
-
-Run tests with
-```
-kubectl exec deploy/mev-inspect -- poetry run pytest --cov=mev_inspect tests
-```
+Postgres tip: Enter `\x` to enter "Explanded display" mode which looks nicer for results with many columns.
 
 ## FAQ
 
 ### How do I delete / reset my local postgres data?
 
-Stop the system if running
+Stop the system if running:
+
 ```
 tilt down
 ```
 
-Delete it with
+Delete it with:
+
 ```
 kubectl delete pvc data-postgresql-postgresql-0
 ```
 
-Start back up again
+Start back up again:
+
 ```
 tilt up
 ```
 
-And rerun migrations to create the tables again
+And rerun migrations to create the tables again:
+
 ```
 kubectl exec deploy/mev-inspect -- alembic upgrade head
 ```
 
 ### I was using the docker-compose setup and want to switch to kube, now what?
 
-Re-add the old `docker-compose.yml` file to your mev-inspect-py directory
+Re-add the old `docker-compose.yml` file to your mev-inspect-py directory.
 
 A copy can be found [here](https://github.com/flashbots/mev-inspect-py/blob/ef60c097719629a7d2dc56c6e6c9a100fb706f76/docker-compose.yml)
 
-Tear down docker-compose resources
+Tear down docker-compose resources:
+
 ```
 docker compose down
 ```
 
-Then go through the steps in the current README for kube setup
+Then go through the steps in the current README for kube setup.
 
 ### Error from server (AlreadyExists): pods "postgres-client" already exists
-This means the postgres client container didn't shut down correctly
 
-Delete this one with
+This means the postgres client container didn't shut down correctly.
+
+Delete this one with:
+
 ```
 kubectl delete pod/postgres-client
 ```
 
-Then start it back up again
+Then start it back up again.
+
+## Maintainers
+
+- [@lukevs](https://github.com/lukevs)
+- [@gheise](https://github.com/gheise)
+- [@bertmiller](https://github.com/bertmiller)
+
+## Contributing
+
+[Flashbots](https://flashbots.net) is a research and development collective working on mitigating the negative externalities of decentralized economies. We contribute with the larger free software community to illuminate the dark forest.
+
+You are welcome here <3.
+
+- If you want to join us, come and say hi in our [Discord chat](https://discord.gg/7hvTycdNcK).
+- If you have a question, feedback or a bug report for this project, please [open a new Issue](https://github.com/flashbots/mev-inspect-py/issues).
+- If you would like to contribute with code, check the [CONTRIBUTING file](CONTRIBUTING.md).
+- We just ask you to be nice.
+
+## Security
+
+If you find a security vulnerability on this project or any other initiative related to Flashbots, please let us know sending an email to security@flashbots.net.
+
+---
+
+Made with ☀️  by the ⚡🤖 collective.

cli.py

@@ -1,8 +1,6 @@
 import asyncio
-import logging
 import os
 import signal
-import sys
 from functools import wraps
 
 import click
@@ -11,9 +9,6 @@ from mev_inspect.inspector import MEVInspector
 
 RPC_URL_ENV = "RPC_URL"
 
-logging.basicConfig(stream=sys.stdout, level=logging.INFO)
-logger = logging.getLogger(__name__)
-
 
 @click.group()
 def cli():
@@ -49,6 +44,16 @@ async def inspect_block_command(block_number: int, rpc: str, cache: bool):
     await inspector.inspect_single_block(block=block_number)
 
 
+@cli.command()
+@click.argument("block_number", type=int)
+@click.option("--rpc", default=lambda: os.environ.get(RPC_URL_ENV, ""))
+@coro
+async def fetch_block_command(block_number: int, rpc: str):
+    inspector = MEVInspector(rpc=rpc)
+    block = await inspector.create_from_block(block_number=block_number)
+    print(block.json())
+
+
 @cli.command()
 @click.argument("after_block", type=int)
 @click.argument("before_block", type=int)

k8s/mev-inspect-backfill/templates/job.yaml

@@ -43,6 +43,24 @@ spec:
               secretKeyRef:
                 name: mev-inspect-db-credentials
                 key: password
+          - name: TRACE_DB_HOST
+            valueFrom:
+              secretKeyRef:
+                name: trace-db-credentials
+                key: host
+                optional: true
+          - name: TRACE_DB_USER
+            valueFrom:
+              secretKeyRef:
+                name: trace-db-credentials
+                key: username
+                optional: true
+          - name: TRACE_DB_PASSWORD
+            valueFrom:
+              secretKeyRef:
+                name: trace-db-credentials
+                key: password
+                optional: true
           - name: RPC_URL
             valueFrom:
               configMapKeyRef:

listener

@@ -25,6 +25,9 @@ case "$1" in
 	start-stop-daemon --stop --quiet --oknodo --pidfile $PIDFILE
         echo "."
 	;;
+  tail)
+        tail -f listener.log
+	;;
   restart)
         echo -n "Restarting daemon: "$NAME
 	start-stop-daemon --stop --quiet --oknodo --retry 30 --pidfile $PIDFILE
@@ -40,7 +43,7 @@ case "$1" in
     ;;
 
   *)
-	echo "Usage: "$1" {start|stop|restart}"
+	echo "Usage: "$1" {start|stop|restart|tail}"
 	exit 1
 esac
 

mev

@@ -24,6 +24,9 @@ case "$1" in
         echo "Connecting to $DB_NAME"
         db
 	;;
+  listener)
+        ./listener $2
+	;;
   backfill)
         start_block_number=$2
         end_block_number=$3
@@ -37,12 +40,24 @@ case "$1" in
         echo "Inspecting block $block_number"
         kubectl exec -ti deploy/mev-inspect -- poetry run inspect-block $block_number
 	;;
+  inspect-many)
+        start_block_number=$2
+        end_block_number=$3
+        echo "Inspecting from block $start_block_number to $end_block_number"
+        kubectl exec -ti deploy/mev-inspect -- \
+            poetry run inspect-many-blocks $start_block_number $end_block_number
+	;;
   test)
         echo "Running tests"
         kubectl exec -ti deploy/mev-inspect -- poetry run pytest tests
 	;;
+  fetch)
+      block_number=$2
+      echo "Fetching block $block_number"
+      kubectl exec -ti deploy/mev-inspect -- poetry run fetch-block $block_number
+  ;;
   *)
-	echo "Usage: "$1" {inspect|test}"
+	echo "Usage: "$1" {db|backfill|inspect|test}"
 	exit 1
 esac
 

mev_inspect/aave_liquidations.py

@@ -4,7 +4,7 @@ from mev_inspect.traces import (
     get_child_traces,
     is_child_of_any_address,
 )
-from mev_inspect.schemas.classified_traces import (
+from mev_inspect.schemas.traces import (
     ClassifiedTrace,
     DecodedCallTrace,
     Classification,

mev_inspect/abi.py

@@ -4,8 +4,8 @@ from typing import Optional
 
 from pydantic import parse_obj_as
 
-from mev_inspect.schemas import ABI
-from mev_inspect.schemas.classified_traces import Protocol
+from mev_inspect.schemas.abi import ABI
+from mev_inspect.schemas.traces import Protocol
 
 
 THIS_FILE_DIRECTORY = Path(__file__).parents[0]

mev_inspect/arbitrages.py

@@ -1,5 +1,5 @@
 from itertools import groupby
-from typing import List, Optional
+from typing import List, Tuple
 
 from mev_inspect.schemas.arbitrages import Arbitrage
 from mev_inspect.schemas.swaps import Swap
@@ -23,70 +23,111 @@ def get_arbitrages(swaps: List[Swap]) -> List[Arbitrage]:
 
 
 def _get_arbitrages_from_swaps(swaps: List[Swap]) -> List[Arbitrage]:
-    pool_addresses = {swap.pool_address for swap in swaps}
+    """
+    An arbitrage is defined as multiple swaps in a series that result in the initial token being returned
+    to the initial sender address.
+
+    There are 2 types of swaps that are most common (99%+).
+    Case I (fully routed):
+    BOT -> A/B -> B/C -> C/A -> BOT
+
+    Case II (always return to bot):
+    BOT -> A/B -> BOT -> B/C -> BOT -> A/C -> BOT
+
+    There is only 1 correct way to route Case I, but for Case II the following valid routes could be found:
+    A->B->C->A / B->C->A->B / C->A->B->C. Thus when multiple valid routes are found we filter to the set that
+    happen in valid order.
+    """
 
     all_arbitrages = []
 
-    for index, first_swap in enumerate(swaps):
-        other_swaps = swaps[:index] + swaps[index + 1 :]
+    start_ends = _get_all_start_end_swaps(swaps)
+    if len(start_ends) == 0:
+        return []
 
-        if first_swap.from_address not in pool_addresses:
-            arbitrage = _get_arbitrage_starting_with_swap(first_swap, other_swaps)
+    # for (start, end) in filtered_start_ends:
+    for (start, end) in start_ends:
+        potential_intermediate_swaps = [
+            swap for swap in swaps if swap is not start and swap is not end
+        ]
+        routes = _get_all_routes(start, end, potential_intermediate_swaps)
 
-            if arbitrage is not None:
-                all_arbitrages.append(arbitrage)
-
-    return all_arbitrages
-
-
-def _get_arbitrage_starting_with_swap(
-    start_swap: Swap,
-    other_swaps: List[Swap],
-) -> Optional[Arbitrage]:
-    swap_path = [start_swap]
-    current_swap: Swap = start_swap
-
-    while True:
-        next_swap = _get_swap_from_address(
-            current_swap.to_address,
-            current_swap.token_out_address,
-            other_swaps,
-        )
-
-        if next_swap is None:
-            return None
-
-        swap_path.append(next_swap)
-        current_swap = next_swap
-
-        if (
-            current_swap.to_address == start_swap.from_address
-            and current_swap.token_out_address == start_swap.token_in_address
-        ):
-
-            start_amount = start_swap.token_in_amount
-            end_amount = current_swap.token_out_amount
+        for route in routes:
+            start_amount = route[0].token_in_amount
+            end_amount = route[-1].token_out_amount
             profit_amount = end_amount - start_amount
 
-            return Arbitrage(
-                swaps=swap_path,
-                block_number=start_swap.block_number,
-                transaction_hash=start_swap.transaction_hash,
-                account_address=start_swap.from_address,
-                profit_token_address=start_swap.token_in_address,
+            arb = Arbitrage(
+                swaps=route,
+                block_number=route[0].block_number,
+                transaction_hash=route[0].transaction_hash,
+                account_address=route[0].from_address,
+                profit_token_address=route[0].token_in_address,
                 start_amount=start_amount,
                 end_amount=end_amount,
                 profit_amount=profit_amount,
             )
-
-    return None
+            all_arbitrages.append(arb)
+    if len(all_arbitrages) == 1:
+        return all_arbitrages
+    else:
+        return [
+            arb
+            for arb in all_arbitrages
+            if (arb.swaps[0].trace_address < arb.swaps[-1].trace_address)
+        ]
 
 
-def _get_swap_from_address(
-    address: str, token_address: str, swaps: List[Swap]
-) -> Optional[Swap]:
-    for swap in swaps:
-        if swap.pool_address == address and swap.token_in_address == token_address:
-            return swap
+def _get_all_start_end_swaps(swaps: List[Swap]) -> List[Tuple[Swap, Swap]]:
+    """
+    Gets the set of all possible opening and closing swap pairs in an arbitrage via
+    - swap[start].token_in == swap[end].token_out
+    - swap[start].from_address == swap[end].to_address
+    - not swap[start].from_address in all_pool_addresses
+    - not swap[end].to_address in all_pool_addresses
+    """
+    pool_addrs = [swap.pool_address for swap in swaps]
+    valid_start_ends: List[Tuple[Swap, Swap]] = []
+    for potential_start_swap in swaps:
+        for potential_end_swap in swaps:
+            if (
+                potential_start_swap.token_in_address
+                == potential_end_swap.token_out_address
+                and potential_start_swap.from_address == potential_end_swap.to_address
+                and not potential_start_swap.from_address in pool_addrs
+            ):
+                valid_start_ends.append((potential_start_swap, potential_end_swap))
+    return valid_start_ends
 
-    return None
+
+def _get_all_routes(
+    start_swap: Swap, end_swap: Swap, other_swaps: List[Swap]
+) -> List[List[Swap]]:
+    """
+    Returns all routes (List[Swap]) from start to finish between a start_swap and an end_swap only accounting for token_address_in and token_address_out.
+    """
+    # If the path is complete, return
+    if start_swap.token_out_address == end_swap.token_in_address:
+        return [[start_swap, end_swap]]
+    elif len(other_swaps) == 0:
+        return []
+
+    # Collect all potential next steps, check if valid, recursively find routes from next_step to end_swap
+    routes: List[List[Swap]] = []
+    for potential_next_swap in other_swaps:
+        if start_swap.token_out_address == potential_next_swap.token_in_address and (
+            start_swap.pool_address == potential_next_swap.from_address
+            or start_swap.to_address == potential_next_swap.pool_address
+            or start_swap.to_address == potential_next_swap.from_address
+        ):
+            remaining_swaps = [
+                swap for swap in other_swaps if swap != potential_next_swap
+            ]
+            next_swap_routes = _get_all_routes(
+                potential_next_swap, end_swap, remaining_swaps
+            )
+            if len(next_swap_routes) > 0:
+                for next_swap_route in next_swap_routes:
+                    next_swap_route.insert(0, start_swap)
+                    routes.append(next_swap_route)
+    return routes

mev_inspect/block.py

@@ -8,8 +8,9 @@ from sqlalchemy import orm
 from web3 import Web3
 
 from mev_inspect.fees import fetch_base_fee_per_gas
-from mev_inspect.schemas import Block, Trace, TraceType
+from mev_inspect.schemas.blocks import Block
 from mev_inspect.schemas.receipts import Receipt
+from mev_inspect.schemas.traces import Trace, TraceType
 
 
 cache_directory = "./cache"

mev_inspect/classifiers/specs/__init__.py

@@ -1,6 +1,6 @@
 from typing import Dict, Optional, Tuple, Type
 
-from mev_inspect.schemas.classified_traces import DecodedCallTrace, Protocol
+from mev_inspect.schemas.traces import DecodedCallTrace, Protocol
 from mev_inspect.schemas.classifiers import ClassifierSpec, Classifier
 
 from .aave import AAVE_CLASSIFIER_SPECS

mev_inspect/classifiers/specs/aave.py

@@ -1,13 +1,10 @@
-from mev_inspect.schemas.classified_traces import (
-    Protocol,
-)
-
 from mev_inspect.schemas.classifiers import (
     ClassifierSpec,
     DecodedCallTrace,
     TransferClassifier,
     LiquidationClassifier,
 )
+from mev_inspect.schemas.traces import Protocol
 from mev_inspect.schemas.transfers import Transfer
 
 

mev_inspect/classifiers/specs/balancer.py

@@ -1,4 +1,4 @@
-from mev_inspect.schemas.classified_traces import (
+from mev_inspect.schemas.traces import (
     DecodedCallTrace,
     Protocol,
 )

mev_inspect/classifiers/specs/compound.py

@@ -1,4 +1,4 @@
-from mev_inspect.schemas.classified_traces import (
+from mev_inspect.schemas.traces import (
     Protocol,
 )
 from mev_inspect.schemas.classifiers import (

mev_inspect/classifiers/specs/curve.py

@@ -1,4 +1,4 @@
-from mev_inspect.schemas.classified_traces import (
+from mev_inspect.schemas.traces import (
     Protocol,
 )
 

mev_inspect/classifiers/specs/erc20.py

@@ -1,4 +1,4 @@
-from mev_inspect.schemas.classified_traces import DecodedCallTrace
+from mev_inspect.schemas.traces import DecodedCallTrace
 from mev_inspect.schemas.classifiers import (
     ClassifierSpec,
     TransferClassifier,

mev_inspect/classifiers/specs/uniswap.py

@@ -1,4 +1,4 @@
-from mev_inspect.schemas.classified_traces import (
+from mev_inspect.schemas.traces import (
     DecodedCallTrace,
     Protocol,
 )

mev_inspect/classifiers/specs/weth.py

@@ -1,4 +1,4 @@
-from mev_inspect.schemas.classified_traces import (
+from mev_inspect.schemas.traces import (
     Protocol,
 )
 from mev_inspect.schemas.classifiers import (

mev_inspect/classifiers/specs/zero_ex.py

@@ -1,4 +1,4 @@
-from mev_inspect.schemas.classified_traces import (
+from mev_inspect.schemas.traces import (
     Protocol,
 )
 from mev_inspect.schemas.classifiers import (

mev_inspect/classifiers/trace.py

@@ -2,13 +2,14 @@ from typing import Dict, List, Optional
 
 from mev_inspect.abi import get_abi
 from mev_inspect.decode import ABIDecoder
-from mev_inspect.schemas.blocks import CallAction, CallResult, Trace, TraceType
-from mev_inspect.schemas.classified_traces import (
+from mev_inspect.schemas.blocks import CallAction, CallResult
+from mev_inspect.schemas.traces import (
     Classification,
     ClassifiedTrace,
     CallTrace,
     DecodedCallTrace,
 )
+from mev_inspect.schemas.traces import Trace, TraceType
 
 from .specs import ALL_CLASSIFIER_SPECS
 

mev_inspect/compound_liquidations.py

@@ -2,7 +2,7 @@ from typing import Dict, List, Optional
 from web3 import Web3
 
 from mev_inspect.traces import get_child_traces
-from mev_inspect.schemas.classified_traces import (
+from mev_inspect.schemas.traces import (
     ClassifiedTrace,
     Classification,
     Protocol,

mev_inspect/crud/traces.py

@@ -1,8 +1,8 @@
 import json
 from typing import List
 
-from mev_inspect.models.classified_traces import ClassifiedTraceModel
-from mev_inspect.schemas.classified_traces import ClassifiedTrace
+from mev_inspect.models.traces import ClassifiedTraceModel
+from mev_inspect.schemas.traces import ClassifiedTrace
 
 
 def delete_classified_traces_for_block(

mev_inspect/inspect_block.py

@@ -11,7 +11,7 @@ from mev_inspect.crud.arbitrages import (
     delete_arbitrages_for_block,
     write_arbitrages,
 )
-from mev_inspect.crud.classified_traces import (
+from mev_inspect.crud.traces import (
     delete_classified_traces_for_block,
     write_classified_traces,
 )

mev_inspect/inspector.py

@@ -7,6 +7,7 @@ from asyncio import CancelledError
 from web3 import Web3
 from web3.eth import AsyncEth
 
+from mev_inspect.block import create_from_block_number
 from mev_inspect.classifiers.trace import TraceClassifier
 from mev_inspect.db import get_inspect_session, get_trace_session
 from mev_inspect.inspect_block import inspect_block
@@ -20,7 +21,7 @@ class MEVInspector:
     def __init__(
         self,
         rpc: str,
-        cache: bool,
+        cache: bool = False,
         max_concurrency: int = 1,
         request_timeout: int = 300,
     ):
@@ -34,6 +35,14 @@ class MEVInspector:
         self.trace_classifier = TraceClassifier()
         self.max_concurrency = asyncio.Semaphore(max_concurrency)
 
+    async def create_from_block(self, block_number: int):
+        return await create_from_block_number(
+            base_provider=self.base_provider,
+            w3=self.w3,
+            block_number=block_number,
+            trace_db_session=self.trace_db_session,
+        )
+
     async def inspect_single_block(self, block: int):
         return await inspect_block(
             self.inspect_db_session,

mev_inspect/liquidations.py

@@ -1,7 +1,7 @@
 from typing import List
 
 from mev_inspect.aave_liquidations import get_aave_liquidations
-from mev_inspect.schemas.classified_traces import (
+from mev_inspect.schemas.traces import (
     ClassifiedTrace,
     Classification,
 )

mev_inspect/miner_payments.py

@@ -1,6 +1,6 @@
 from typing import List
 
-from mev_inspect.schemas.classified_traces import ClassifiedTrace
+from mev_inspect.schemas.traces import ClassifiedTrace
 from mev_inspect.schemas.miner_payments import MinerPayment
 from mev_inspect.schemas.receipts import Receipt
 from mev_inspect.traces import get_traces_by_transaction_hash

mev_inspect/schemas/__init__.py

@@ -1,2 +0,0 @@
-from .abi import ABI
-from .blocks import Block, Trace, TraceType

mev_inspect/schemas/blocks.py

@@ -1,11 +1,11 @@
-from enum import Enum
-from typing import List, Optional
+from typing import List
 
 from pydantic import validator
 
 from mev_inspect.utils import hex_to_int
 
 from .receipts import Receipt
+from .traces import Trace
 from .utils import CamelModel, Web3Model
 
 
@@ -36,27 +36,6 @@ class CallAction(Web3Model):
         fields = {"from_": "from"}
 
 
-class TraceType(Enum):
-    call = "call"
-    create = "create"
-    delegate_call = "delegateCall"
-    reward = "reward"
-    suicide = "suicide"
-
-
-class Trace(CamelModel):
-    action: dict
-    block_hash: str
-    block_number: int
-    result: Optional[dict]
-    subtraces: int
-    trace_address: List[int]
-    transaction_hash: Optional[str]
-    transaction_position: Optional[int]
-    type: TraceType
-    error: Optional[str]
-
-
 class Block(Web3Model):
     block_number: int
     miner: str

mev_inspect/schemas/classifiers.py

@@ -3,7 +3,7 @@ from typing import Dict, List, Optional, Type
 
 from pydantic import BaseModel
 
-from .classified_traces import Classification, DecodedCallTrace, Protocol
+from .traces import Classification, DecodedCallTrace, Protocol
 from .transfers import Transfer
 
 

mev_inspect/schemas/liquidations.py

@@ -1,6 +1,6 @@
 from typing import List, Optional
 from pydantic import BaseModel
-from mev_inspect.schemas.classified_traces import Protocol
+from mev_inspect.schemas.traces import Protocol
 
 
 class Liquidation(BaseModel):

mev_inspect/schemas/swaps.py

@@ -2,7 +2,7 @@ from typing import List, Optional
 
 from pydantic import BaseModel
 
-from mev_inspect.schemas.classified_traces import Protocol
+from mev_inspect.schemas.traces import Protocol
 
 
 class Swap(BaseModel):

mev_inspect/schemas/traces.py

@@ -1,7 +1,28 @@
 from enum import Enum
 from typing import Any, Dict, List, Optional
 
-from .blocks import Trace
+from .utils import CamelModel
+
+
+class TraceType(Enum):
+    call = "call"
+    create = "create"
+    delegate_call = "delegateCall"
+    reward = "reward"
+    suicide = "suicide"
+
+
+class Trace(CamelModel):
+    action: dict
+    block_hash: str
+    block_number: int
+    result: Optional[dict]
+    subtraces: int
+    trace_address: List[int]
+    transaction_hash: Optional[str]
+    transaction_position: Optional[int]
+    type: TraceType
+    error: Optional[str]
 
 
 class Classification(Enum):
@@ -26,16 +47,13 @@ class Protocol(Enum):
 
 
 class ClassifiedTrace(Trace):
-    transaction_hash: str
-    block_number: int
-    trace_address: List[int]
     classification: Classification
-    error: Optional[str]
     to_address: Optional[str]
     from_address: Optional[str]
     gas: Optional[int]
     value: Optional[int]
     gas_used: Optional[int]
+    transaction_hash: str
     protocol: Optional[Protocol]
     function_name: Optional[str]
     function_signature: Optional[str]

mev_inspect/swaps.py

@@ -1,7 +1,7 @@
 from typing import List, Optional
 
 from mev_inspect.classifiers.specs import get_classifier
-from mev_inspect.schemas.classified_traces import (
+from mev_inspect.schemas.traces import (
     ClassifiedTrace,
     Classification,
     DecodedCallTrace,

mev_inspect/tokenflow.py

@@ -1,6 +1,7 @@
 from typing import List, Optional
 
-from mev_inspect.schemas import Block, Trace, TraceType
+from mev_inspect.schemas.blocks import Block
+from mev_inspect.schemas.traces import Trace, TraceType
 
 weth_address = "0xc02aaa39b223fe8d0a0e5c4f27ead9083c756cc2"
 

mev_inspect/traces.py

@@ -1,7 +1,7 @@
 from itertools import groupby
 from typing import Dict, List
 
-from mev_inspect.schemas.classified_traces import ClassifiedTrace
+from mev_inspect.schemas.traces import ClassifiedTrace
 
 
 def is_child_trace_address(

mev_inspect/transfers.py

@@ -2,7 +2,7 @@ from typing import Dict, List, Optional, Sequence
 
 from mev_inspect.classifiers.specs import get_classifier
 from mev_inspect.schemas.classifiers import TransferClassifier
-from mev_inspect.schemas.classified_traces import (
+from mev_inspect.schemas.traces import (
     ClassifiedTrace,
     DecodedCallTrace,
 )

pyproject.toml

@@ -32,6 +32,7 @@ build-backend = "poetry.core.masonry.api"
 [tool.poetry.scripts]
 inspect-block = 'cli:inspect_block_command'
 inspect-many-blocks = 'cli:inspect_many_blocks_command'
+fetch-block = 'cli:fetch_block_command'
 
 [tool.black]
 exclude = '''

tests/helpers.py

@@ -1,11 +1,11 @@
 from typing import List, Optional
 
-from mev_inspect.schemas.blocks import TraceType
-from mev_inspect.schemas.classified_traces import (
+from mev_inspect.schemas.traces import (
     Classification,
     ClassifiedTrace,
     DecodedCallTrace,
     Protocol,
+    TraceType,
 )
 
 

tests/liquidation_test.py

@@ -2,7 +2,7 @@ from typing import List
 
 from mev_inspect.aave_liquidations import get_aave_liquidations
 from mev_inspect.schemas.liquidations import Liquidation
-from mev_inspect.schemas.classified_traces import Protocol
+from mev_inspect.schemas.traces import Protocol
 from mev_inspect.classifiers.trace import TraceClassifier
 from tests.utils import load_test_block
 

tests/test_arbitrage_integration.py

@@ -15,12 +15,47 @@ def test_arbitrage_real_block():
     assert len(swaps) == 51
 
     arbitrages = get_arbitrages(list(swaps))
-    assert len(arbitrages) == 1
+    assert len(arbitrages) == 2
 
-    arbitrage = arbitrages[0]
+    arbitrage_1 = [
+        arb
+        for arb in arbitrages
+        if arb.transaction_hash
+        == "0x448245bf1a507b73516c4eeee01611927dada6610bf26d403012f2e66800d8f0"
+    ][0]
+    arbitrage_2 = [
+        arb
+        for arb in arbitrages
+        if arb.transaction_hash
+        == "0xfcf4558f6432689ea57737fe63124a5ec39fd6ba6aaf198df13a825dd599bffc"
+    ][0]
 
-    assert len(arbitrage.swaps) == 3
+    assert len(arbitrage_1.swaps) == 3
     assert (
-        arbitrage.profit_token_address == "0xc02aaa39b223fe8d0a0e5c4f27ead9083c756cc2"
+        arbitrage_1.profit_token_address == "0xc02aaa39b223fe8d0a0e5c4f27ead9083c756cc2"
     )
-    assert arbitrage.profit_amount == 53560707941943273628
+    assert len(arbitrage_1.swaps) == 3
+    assert (
+        arbitrage_1.swaps[1].token_in_address
+        == "0x25f8087ead173b73d6e8b84329989a8eea16cf73"
+    )
+    assert (
+        arbitrage_1.swaps[1].token_out_address
+        == "0xa0b86991c6218b36c1d19d4a2e9eb0ce3606eb48"
+    )
+    assert arbitrage_1.profit_amount == 750005273675102326
+
+    assert len(arbitrage_2.swaps) == 3
+    assert (
+        arbitrage_2.profit_token_address == "0xc02aaa39b223fe8d0a0e5c4f27ead9083c756cc2"
+    )
+    assert len(arbitrage_2.swaps) == 3
+    assert (
+        arbitrage_2.swaps[1].token_in_address
+        == "0x25f8087ead173b73d6e8b84329989a8eea16cf73"
+    )
+    assert (
+        arbitrage_2.swaps[1].token_out_address
+        == "0xa0b86991c6218b36c1d19d4a2e9eb0ce3606eb48"
+    )
+    assert arbitrage_2.profit_amount == 53560707941943273628

tests/test_arbitrages.py

@@ -1,9 +1,11 @@
-from mev_inspect.arbitrages import get_arbitrages
+from typing import List
+
+from mev_inspect.arbitrages import get_arbitrages, _get_all_routes
+from mev_inspect.schemas.swaps import Swap
 from mev_inspect.classifiers.specs.uniswap import (
     UNISWAP_V2_PAIR_ABI_NAME,
     UNISWAP_V3_POOL_ABI_NAME,
 )
-from mev_inspect.schemas.swaps import Swap
 
 
 def test_two_pool_arbitrage(get_transaction_hashes, get_addresses):
@@ -17,10 +19,11 @@ def test_two_pool_arbitrage(get_transaction_hashes, get_addresses):
         unrelated_pool_address,
         first_token_address,
         second_token_address,
-    ] = get_addresses(6)
+        third_token_address,
+    ] = get_addresses(7)
 
     first_token_in_amount = 10
-    first_token_out_amount = 10
+    first_token_out_amount = 11
     second_token_amount = 15
 
     arb_swaps = [
@@ -62,7 +65,7 @@ def test_two_pool_arbitrage(get_transaction_hashes, get_addresses):
         to_address=account_address,
         token_in_address=second_token_address,
         token_in_amount=first_token_in_amount,
-        token_out_address=first_token_address,
+        token_out_address=third_token_address,
         token_out_amount=first_token_out_amount,
     )
 
@@ -100,7 +103,7 @@ def test_three_pool_arbitrage(get_transaction_hashes, get_addresses):
     ] = get_addresses(7)
 
     first_token_in_amount = 10
-    first_token_out_amount = 10
+    first_token_out_amount = 11
     second_token_amount = 15
     third_token_amount = 40
 
@@ -158,3 +161,70 @@ def test_three_pool_arbitrage(get_transaction_hashes, get_addresses):
     assert arbitrage.start_amount == first_token_in_amount
     assert arbitrage.end_amount == first_token_out_amount
     assert arbitrage.profit_amount == first_token_out_amount - first_token_in_amount
+
+
+def test_get_all_routes():
+    # A -> B, B -> A
+    start_swap = create_generic_swap("0xa", "0xb")
+    end_swap = create_generic_swap("0xb", "0xa")
+    routes = _get_all_routes(start_swap, end_swap, [])
+    assert len(routes) == 1
+
+    # A->B, B->C, C->A
+    start_swap = create_generic_swap("0xa", "0xb")
+    other_swaps = [create_generic_swap("0xb", "0xc")]
+    end_swap = create_generic_swap("0xc", "0xa")
+    routes = _get_all_routes(start_swap, end_swap, other_swaps)
+    assert len(routes) == 1
+
+    # A->B, B->C, C->A + A->D
+    other_swaps.append(create_generic_swap("0xa", "0xd"))
+    routes = _get_all_routes(start_swap, end_swap, other_swaps)
+    assert len(routes) == 1
+
+    # A->B, B->C, C->A + A->D B->E
+    other_swaps.append(create_generic_swap("0xb", "0xe"))
+    routes = _get_all_routes(start_swap, end_swap, other_swaps)
+    assert len(routes) == 1
+
+    # A->B, B->A, B->C, C->A
+    other_swaps = [create_generic_swap("0xb", "0xa"), create_generic_swap("0xb", "0xc")]
+    routes = _get_all_routes(start_swap, end_swap, other_swaps)
+    assert len(routes) == 1
+    expect_simple_route = [["0xa", "0xb"], ["0xb", "0xc"], ["0xc", "0xa"]]
+    assert len(routes[0]) == len(expect_simple_route)
+    for i in range(len(expect_simple_route)):
+        assert expect_simple_route[i][0] == routes[0][i].token_in_address
+        assert expect_simple_route[i][1] == routes[0][i].token_out_address
+
+    # A->B, B->C, C->D, D->A, B->D
+    end_swap = create_generic_swap("0xd", "0xa")
+    other_swaps = [
+        create_generic_swap("0xb", "0xc"),
+        create_generic_swap("0xc", "0xd"),
+        create_generic_swap("0xb", "0xd"),
+    ]
+    routes = _get_all_routes(start_swap, end_swap, other_swaps)
+    assert len(routes) == 2
+
+
+def create_generic_swap(
+    tok_a: str = "0xa",
+    tok_b: str = "0xb",
+    amount_a_in: int = 1,
+    amount_b_out: int = 1,
+    trace_address: List[int] = [],
+):
+    return Swap(
+        abi_name=UNISWAP_V3_POOL_ABI_NAME,
+        transaction_hash="0xfake",
+        block_number=0,
+        trace_address=trace_address,
+        pool_address="0xfake",
+        from_address="0xfake",
+        to_address="0xfake",
+        token_in_address=tok_a,
+        token_in_amount=amount_a_in,
+        token_out_address=tok_b,
+        token_out_amount=amount_b_out,
+    )

tests/test_compound.py

@@ -1,6 +1,6 @@
 from mev_inspect.compound_liquidations import get_compound_liquidations
 from mev_inspect.schemas.liquidations import Liquidation
-from mev_inspect.schemas.classified_traces import Protocol
+from mev_inspect.schemas.traces import Protocol
 from mev_inspect.classifiers.trace import TraceClassifier
 from tests.utils import load_test_block, load_comp_markets, load_cream_markets
 

tests/test_swaps.py

@@ -4,7 +4,7 @@ from mev_inspect.classifiers.specs.uniswap import (
     UNISWAP_V2_PAIR_ABI_NAME,
     UNISWAP_V3_POOL_ABI_NAME,
 )
-from mev_inspect.schemas.classified_traces import Protocol
+from mev_inspect.schemas.traces import Protocol
 
 from .helpers import (
     make_unknown_trace,

tests/test_traces.py

@@ -1,6 +1,6 @@
 from typing import List
 
-from mev_inspect.schemas.classified_traces import ClassifiedTrace
+from mev_inspect.schemas.traces import ClassifiedTrace
 from mev_inspect.traces import is_child_trace_address, get_child_traces
 
 from .helpers import make_many_unknown_traces