feat: remove need for pg within mev-inspect env (#344)

* instructions for running without kubernetes ('monolithic mode')

* added docker instructions

* chore: remove pgsql as hard dependency

* chore: update deps

* docs: updated docs to remove local pg engine for docker install

* docs: reword docs

* ci: update poetry source

* fix: refactor tests for mypy

* fix: search miner for eth2

* feat: improve eth2 miner fn

* refactor: unnecessary comma

* test: add miner generation tests

---------

Co-authored-by: pintail <you@example.com>

.github/workflows/github-actions.yml

@@ -21,7 +21,7 @@ jobs:
       - name: Bootstrap poetry
         shell: bash
         run: |
-          curl -sL https://raw.githubusercontent.com/python-poetry/poetry/master/install-poetry.py \
+          curl -sSL https://install.python-poetry.org \
             | python - -y 
 
       - name: Update PATH

.gitignore

@@ -25,3 +25,6 @@ cache
 
 # pycharm
 .idea
+
+.env
+.python-version

MONOLITHIC.md

@@ -0,0 +1,126 @@
+
+# Running mev-inspect-py without kubernetes ('monolithic mode')
+
+Running mev-inspect-py outside of kubernetes can be useful for debug purposes. In this case, the steps for installation are:
+1. Install dependencies (pyenv, poetry, postgres)
+1. Set up python virtual environment using matching python version (3.9.x) and install required python modules using poetry
+1. Create postgres database
+1. Run database migrations
+
+The database credentials and archive node address used by mev-inspect-py need to be loaded into environment variables (both for database migrations and to run mev-inspect-py).
+
+## Ubuntu install instructions
+
+So, starting from a clean Ubuntu 22.04 installation, the prerequisites for pyenv, psycopg2 (python3-dev libpq-dev) can be installed with
+
+`sudo apt install -y make build-essential git libssl-dev zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev libncursesw5-dev xz-utils tk-dev liblzma-dev python3-dev libpq-dev`
+
+### pyenv
+Install pyenv using the web installer
+
+`curl https://pyenv.run | bash`
+
+and add the following to `~/.bashrc` (if running locally) or `~/.profile` (if running over ssh).
+
+```
+export PYENV_ROOT="$HOME/.pyenv"
+command -v pyenv >/dev/null || export PATH="$PYENV_ROOT/bin:$PATH"
+eval "$(pyenv init -)"
+```
+
+Then update the current shell by running `source ~/.bashrc` or `source ~/.profile` as appropriate.
+
+### Poetry
+
+Install Poetry using the web installer
+
+`curl -sSL https://install.python-poetry.org | python3 -`
+
+add the following to `~/.bashrc` (if running locally) or `~/.profile` (if running over ssh)
+
+`export PATH="/home/user/.local/bin:$PATH"`
+
+If running over ssh you should also add the following to `~/.profile` to prevent [Poetry errors](https://github.com/python-poetry/poetry/issues/1917) from a lack of active keyring:
+
+`export PYTHON_KEYRING_BACKEND=keyring.backends.null.Keyring`
+
+Again update current shell by running `source ~/.bashrc` or `source ~/.profile` as appropriate.
+
+### postgres
+We have tested two alternatives for postgres - installing locally or as a container.
+
+#### Option 1: Installing locally
+
+To install locally from a clean Ubuntu 22.04 installation, run:
+`sudo apt install postgresql postgresql-contrib`
+
+Note: You may need to reconfigure your pg-hba.conf to allow local access.
+
+#### Option 2: Installing docker
+
+To avoid interfering with your local postgres instance, you may prefer to run postgres within a docker container.
+For docker installation instructions, please refer to https://docs.docker.com/engine/install/ubuntu/
+
+### mev-inspect-py
+
+With all dependencies now installed, clone the mev-inspec-py repo
+```
+git clone https://github.com/flashbots/mev-inspect-py.git
+cd mev-inspect-py
+```
+We now install the required pythn version and use Poetry to install the required python modules into a virtual environment. 
+
+```
+pyenv install 3.9.16
+pyenv local 3.9.16
+poetry env use 3.9.16
+poetry install
+```
+
+### Create database
+mev-inspect-py outputs to a postgres database, so we need to set this up. There are various ways of doing this, two options are presented here.
+
+#### Option 1 — Run postgres locally
+```
+sudo -u postgres psql
+\password
+postgres
+create database mev_inspect;
+\q
+```
+
+#### Option 2 — Use postgres docker image
+To avoid interfering with your local postgres instance, you may prefer to run postgres within a docker container. First ensure that postgres is not currently running to ensure port `5432` is available:
+`sudo systemctl stop postgresql`
+and then start a containerised postgres instance:
+`sudo docker run -d -p 5432:5432 -e POSTGRES_USER=postgres -e POSTGRES_PASSWORD=postgres -e POSTGRES_DB=mev_inspect postgres`
+
+### Environment variables
+We will need to set a few environment variables to use mev-inspect-py. **These will be required every time mev-inspect-py runs**, so again you may wish to add these to your `~/.bashrc` and/or `~/.profile` as appropriate. Note that you need to substitute the correct URL for your archive node below if you are not running Erigon locally.
+```
+export POSTGRES_USER=postgres
+export POSTGRES_PASSWORD=postgres
+export POSTGRES_HOST=localhost
+export RPC_URL="http://127.0.0.1:8545"
+```
+### Database migrations
+Finally run the database migrations and fetch price information:
+
+```
+poetry run alembic upgrade head
+poetry run fetch-all-prices
+```
+
+## Usage instructions
+The same functionality available through kubernetes can be run in 'monolithic mode', but the relevant functions now need to be invoked by Poetry directly. So to inspect a single block, run for example:
+
+`poetry run inspect-block 16379706`
+
+Or to inspect a range of blocks:
+
+`poetry run inspect-many-blocks 16379606 16379706`
+
+Or to run the test suite:
+
+`poetry run pytest tests`
+

README.md

@@ -66,6 +66,10 @@ And load prices data
 ./mev prices fetch-all
 ```
 
+## Monolithic (non-kubernetes) install instructions
+
+For an alternative means of running mev-inspect-py for smaller set-ups or debug purposes see the [monolithic install instructions](MONOLITHIC.md).
+
 ## Usage
 
 ### Inspect a single block

mev_inspect/block.py

@@ -34,8 +34,7 @@ async def create_from_block_number(
         _find_or_fetch_block_traces(w3, block_number, trace_db_session),
         _find_or_fetch_base_fee_per_gas(w3, block_number, trace_db_session),
     )
-
-    miner_address = _get_miner_address_from_traces(traces)
+    miner_address = await _find_or_fetch_miner_address(w3, block_number, traces)
 
     return Block(
         block_number=block_number,
@@ -180,11 +179,27 @@ def _find_base_fee_per_gas(
         return base_fee
 
 
+async def _find_or_fetch_miner_address(
+    w3,
+    block_number: int,
+    traces: List[Trace],
+) -> Optional[str]:
+    # eth1 blocks
+    miner_address = _get_miner_address_from_traces(traces)
+    if miner_address is not None:
+        return miner_address
+    return await _fetch_miner_eth2(w3, block_number)
+
+
+async def _fetch_miner_eth2(w3, block_number: int) -> Optional[str]:
+    block_json = await w3.eth.get_block(block_number)
+    return block_json["miner"]
+
+
 def _get_miner_address_from_traces(traces: List[Trace]) -> Optional[str]:
     for trace in traces:
         if trace.type == TraceType.reward:
             return trace.action["author"]
-
     return None
 
 

mev_inspect/schemas/blocks.py

@@ -13,7 +13,7 @@ class CallResult(CamelModel):
     gas_used: int
 
     @validator("gas_used", pre=True)
-    def maybe_hex_to_int(v):
+    def maybe_hex_to_int(cls, v):
         if isinstance(v, str):
             return hex_to_int(v)
         return v
@@ -27,7 +27,7 @@ class CallAction(Web3Model):
     gas: int
 
     @validator("value", "gas", pre=True)
-    def maybe_hex_to_int(v):
+    def maybe_hex_to_int(cls, v):
         if isinstance(v, str):
             return hex_to_int(v)
         return v

mev_inspect/schemas/receipts.py

@@ -24,7 +24,7 @@ class Receipt(CamelModel):
         "cumulative_gas_used",
         pre=True,
     )
-    def maybe_hex_to_int(v):
+    def maybe_hex_to_int(cls, v):
         if isinstance(v, str):
             return hex_to_int(v)
         return v

pyproject.toml

@@ -10,12 +10,13 @@ web3 = "^5.23.0"
 pydantic = "^1.8.2"
 hexbytes = "^0.2.1"
 click = "^8.0.1"
-psycopg2 = "^2.9.1"
+psycopg2-binary = "^2.9.7"
 aiohttp = "^3.8.0"
 dramatiq = {extras = ["redis"], version = "^1.12.1"}
 pycoingecko = "^2.2.0"
 boto3 = "^1.20.48"
 aiohttp-retry = "^2.4.6"
+pyyaml = "^6.0.1"
 
 [tool.poetry.dev-dependencies]
 pre-commit = "^2.13.0"
@@ -30,6 +31,7 @@ alembic = "^1.6.5"
 CProfileV = "^1.0.7"
 regex = "^2021.10.8"
 pytest-profiling = "^1.7.0"
+sqlalchemy = "^1.4.23"
 
 [build-system]
 requires = ["poetry-core>=1.0.0"]
@@ -82,3 +84,6 @@ filter_files = true
 known_first_party = "mev_inspect"
 known_third_party = "alembic"
 py_version=39
+
+[pytest]
+asyncio_mode = "auto"

tests/test_block.py

@@ -0,0 +1,65 @@
+from unittest.mock import MagicMock, patch
+
+import pytest
+
+from mev_inspect.block import _find_or_fetch_miner_address
+from tests.utils import load_test_block
+
+
+@pytest.fixture
+def mocked_web3():
+    with patch("mev_inspect.block.Web3") as mock_web3:
+        yield mock_web3
+
+
+@pytest.mark.asyncio
+# pylint: disable=redefined-outer-name
+async def test_eth1_block_miner(mocked_web3):
+    # Create a mock Web3 instance
+    mock_web3_instance = mocked_web3.return_value
+
+    # Set up the mock for web3.eth.get_block
+    mock_eth = mock_web3_instance.eth
+    mock_eth.get_block.return_value = {
+        "miner": "0x4a536c1f6a5d5a9c1aeca9f6d04fbbf5f0d8f4e3"
+    }
+
+    # Load a sample block and remove the miner
+    block_number = 10921991
+    block = load_test_block(block_number)
+    block.miner = None
+
+    # Test that the miner is fetched
+    miner_address = await _find_or_fetch_miner_address(
+        w3=mock_web3_instance, traces=block.traces, block_number=block_number
+    )  # Use 'await'
+
+    # this is within the traces object
+    assert miner_address == "0x52bc44d5378309ee2abf1539bf71de1b7d7be3b5"
+
+
+@pytest.mark.asyncio
+# pylint: disable=redefined-outer-name
+async def test_eth2_block_miner(mocked_web3):
+    # Create a mock Web3 instance
+    mock_web3_instance = mocked_web3.return_value
+
+    # Create a coroutine function to mock w3.eth.get_block
+    # pylint: disable=unused-argument
+    async def mock_get_block(block_number):
+        return {"miner": "0x4a536c1f6a5d5a9c1aeca9f6d04fbbf5f0d8f4e3"}
+
+    # Mock w3.eth.get_block with the coroutine function
+    mock_web3_instance.eth.get_block = MagicMock(side_effect=mock_get_block)
+
+    # Load a sample block and remove the miner
+    block_number = 10921990
+    block = load_test_block(block_number)
+    block.miner = None
+
+    # Test that the miner is fetched
+    miner_address = await _find_or_fetch_miner_address(
+        w3=mock_web3_instance, traces=block.traces, block_number=block_number
+    )  # Use 'await'
+
+    assert miner_address == "0x4a536c1f6a5d5a9c1aeca9f6d04fbbf5f0d8f4e3"

tests/utils.py

@@ -2,8 +2,6 @@ import json
 import os
 from typing import Dict, List
 
-from pydantic import parse_file_as
-
 from mev_inspect.schemas.blocks import Block
 from mev_inspect.schemas.sandwiches import Sandwich
 
@@ -14,7 +12,10 @@ TEST_SANDWICHES_DIRECTORY = os.path.join(THIS_FILE_DIRECTORY, "sandwiches")
 
 def load_test_sandwiches(block_number: int) -> List[Sandwich]:
     sandwiches_path = f"{TEST_SANDWICHES_DIRECTORY}/{block_number}.json"
-    return parse_file_as(List[Sandwich], sandwiches_path)
+
+    with open(sandwiches_path, "r") as file:
+        sandwiches_data = json.load(file)
+    return [Sandwich(**sandwich) for sandwich in sandwiches_data]
 
 
 def load_test_block(block_number: int) -> Block: