Compare commits

...

92 commits

Author SHA1 Message Date
Andre Miras
03f93c494d
Fix missing curl binary and update CI (#45)
Fix parquet file download error because of a missing curl binary.
This was happening on the `eth_getBalance` test for instance.
Note that flood swallowed the exception making the error harder to spot.

Update the CI to also test against eth_getBalance which triggers a
parquet file download.
Also removes the redundant node2, this was a workaround as flood used to
crash with only one node.

Also unpin the patch version of Python and address a small DRY on the
Vegeta version.
2023-08-30 14:23:24 -07:00
Zane Peycke
eb88711615
Update README.md with report example (#44)
Add existing report example link in report generation section
2023-08-30 14:05:10 -07:00
sslivkoff
61644933f9 bump to 0.3.1 2023-07-28 16:09:54 -04:00
sslivkoff
c8a153a4a9 fix version checks for tagged commits 2023-07-28 16:09:35 -04:00
sslivkoff
dfa85e9779 bump to 0.3.0 2023-07-28 15:05:52 -04:00
sslivkoff
f1162e5592 clean up load test output 2023-07-28 14:52:14 -04:00
sslivkoff
453f5a4da4 update reports with more info 2023-07-28 14:35:28 -04:00
sslivkoff
18e182a147 fix single node report generation 2023-07-28 01:31:12 -04:00
sslivkoff
5b169dbbdb
add ToC to README.md 2023-07-27 14:33:19 -07:00
sslivkoff
7b22858f34 move user guide into readme 2023-07-27 17:24:07 -04:00
sslivkoff
f899efc9c9 update report generation pipeline for namespace refactor 2023-07-27 17:20:18 -04:00
sslivkoff
180545e986 remove spurious report 2023-07-27 16:53:29 -04:00
sslivkoff
a1a585f204
clean toplevel namespace (#39)
* clean toplevel namespace

* ruff checks

* update tests to new namespace refactor
2023-07-27 13:35:02 -07:00
sslivkoff
61104d47fb add docker notes 2023-07-27 16:06:09 +00:00
sslivkoff
8e2cdc2499
Improve docs (#38)
* improve docs

* add using --version or -V instead of just "version"
2023-07-27 08:02:48 -07:00
sslivkoff
67d3695242 fix ruff regression 2023-07-25 19:31:23 -07:00
Landon Gingerich
4d3776b266 fix Vegeta location 2023-07-25 19:31:23 -07:00
Landon Gingerich
0e0b58c0c2 use slim python image and multi-stage build 2023-07-25 19:31:23 -07:00
Andre Miras
9e698b0bff
Docker run from the CI, fixes #34 (#35)
Run a light test agaisnt the llama public RPC.
Note while this is working for now the GitHub runner IPs could get
blacklisted as people abuse it. Updating the CI to use an API key would
solve it.
2023-07-18 23:18:48 -07:00
sslivkoff
1c094f9d92 Merge branch 'main' of github.com:paradigm-operations/rpc_bench 2023-07-17 15:35:03 -07:00
sslivkoff
923f8838e6 add aggregation mode to print cli command 2023-07-17 15:34:46 -07:00
Andre Miras
dcf73c57f0
Drop the checkthechain clone (#32)
As checkthechain is now part of the `pyproject.toml`
2023-07-17 00:42:31 -07:00
Andre Miras
d42318d3ed
Add Docker build to the CI (#31) 2023-07-17 00:37:46 -07:00
Andre Miras
b86e1cc589
Fix polars dependency (#30)
The error was:
```
The conflict is caused by:
ERROR: ResolutionImpossible: for help visit
https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts
    paradigm-flood 0.3.0a0 depends on polars<0.19 and >=0.18.7
    checkthechain 0.3.9 depends on polars<0.18 and >=0.17.3
```
Refs:
https://github.com/paradigmxyz/flood/actions/runs/5571608157/jobs/10176802894
2023-07-17 00:30:55 -07:00
sslivkoff
c897403013 move metadata into SingleRunTestPayload instead of RunOutput 2023-07-16 20:57:19 -07:00
sslivkoff
3cbf6fcb26 save additional metadata alongside each test 2023-07-16 20:50:34 -07:00
sslivkoff
f5700e0d3e clean output formatting of deep check summaries 2023-07-16 18:05:11 -07:00
sslivkoff
1ba1b28d64 flesh put deep check metrics for failed requests 2023-07-16 17:19:57 -07:00
sslivkoff
0167d65ada change --nodes to a positional argument in update command 2023-07-16 16:08:38 -07:00
sslivkoff
6f70dafe62 use orjson for encoding/decoding of large json 2023-07-16 15:54:22 -07:00
sslivkoff
4bc0663f21 add eth_feeHistory load test 2023-07-16 14:28:53 -07:00
sslivkoff
ca80e9cbd5 improve cli arg help messages 2023-07-16 14:11:55 -07:00
sslivkoff
d3f6db6fa2 use timestamp as default random seed 2023-07-16 14:05:20 -07:00
sslivkoff
6cf83fd5d7 specify --work-tree alongside --git-dir in update command 2023-07-16 12:37:03 -07:00
sslivkoff
048808d163 completely avoid generating test clientside when running remote tests 2023-07-16 12:26:48 -07:00
sslivkoff
15c7f3b590 allow generating xxl block ranges using only xl sample 2023-07-16 11:23:21 -07:00
sslivkoff
d7b36bc4a0 update to toolcli 0.6.16 2023-07-16 11:00:29 -07:00
sslivkoff
7cd1ac07c1 fix cli passthrough of single vegeta args 2023-07-16 10:45:35 -07:00
sslivkoff
b499c375fb implement remote version of update 2023-07-15 19:52:09 -07:00
sslivkoff
c9c9a97163 change vegeta_kwargs to vegeta_args 2023-07-15 19:45:27 -07:00
sslivkoff
8734a749d2 fix check on null git dir 2023-07-15 17:21:59 -07:00
sslivkoff
d5c8a7cbfc fix pass vegeta_kwargs through call chain 2023-07-15 17:21:03 -07:00
sslivkoff
b40ab09e84 add update command 2023-07-15 17:19:21 -07:00
sslivkoff
a454760d42 follow symlinks of installation path 2023-07-15 16:00:59 -07:00
sslivkoff
50ce90a9cd expand installation metadata and add update process 2023-07-15 15:47:02 -07:00
Andre Miras
3f52adb037
fix pyproject.toml dependencies (#25)
1) adding `ipykernel` fixed:
```
jupyter_client.kernelspec.NoSuchKernel: No such kernel named python3
```

2) adding `ipython_genutils` fixed:
```
ModuleNotFoundError: No module named 'ipython_genutils'
```

3) bumping the `nbconvert` version fixed:
```
ImportError: cannot import name 'contextfilter' from 'jinja2'
(/home/flood/.local/lib/python3.11/site-packages/jinja2/__init__.py)
```

This was tested within a Docker container, built with:
```
docker build --tag paradigmxyz/flood:latest .
```
Ran with:
```
docker run --rm --volume ${PWD}/output:/output paradigmxyz/flood:latest \
eth_getBlockByNumber NODE:8545 --output /output/
```
2023-07-16 00:11:01 +03:00
sslivkoff
6342f74dfa use custom version command to include commit hash and remotes 2023-07-15 10:59:31 -07:00
sslivkoff
6b91c743b4 Merge branch 'main' of github.com:paradigm-operations/rpc_bench 2023-07-15 10:46:44 -07:00
sslivkoff
d55af3f944 have run() return test outputs for python calls 2023-07-15 10:42:09 -07:00
sslivkoff
08741652eb add commit hash to installation.flood_version 2023-07-15 09:24:02 -07:00
Andre Miras
da1382c3b4
Add vegeta PATH in Docker, fix #24 (#26)
The results.json was not generated because vegeta was not found.
For some reason the error was caught and silenced which made it harder
to debug.
2023-07-15 14:28:22 +03:00
sslivkoff
68e285665e add initial log load test parameterizations 2023-07-14 18:33:18 -07:00
sslivkoff
9c6e32705d ruff linelength lints 2023-07-14 16:04:00 -07:00
sslivkoff
e3256e1463 bump to latest ctc for debug traces 2023-07-14 16:03:09 -07:00
sslivkoff
f17215fca7 add debug_traceX methods to equality test 2023-07-14 15:39:44 -07:00
sslivkoff
9a92f3c2a8 add various stateDiff and vmTrace calls to equality tests 2023-07-14 13:08:04 -07:00
sslivkoff
572c3f0c9d clip equality test diff summaries that are >100 lines long 2023-07-14 13:07:32 -07:00
sslivkoff
7a0e7ec406 bump to toolcli 0.6.15 2023-07-14 12:42:56 -07:00
sslivkoff
cf288b0a7b show client version on multiple lines for clarity 2023-07-14 01:11:40 -07:00
sslivkoff
d0e4dd6918 fix noqa format for github version of ruff 2023-07-14 00:40:24 -07:00
sslivkoff
cdc736ec70 fix deep check tests 2023-07-14 00:26:19 -07:00
sslivkoff
d0ab7e175b clean output messages 2023-07-13 21:05:19 -07:00
sslivkoff
fd64682707 tweaks for local testing 2023-07-13 20:47:58 -07:00
sslivkoff
af14a3fd8f bump version to 0.3.0a for testing 2023-07-13 20:09:02 -07:00
sslivkoff
fc0ed5a940 refactor of many data structures and pipelines 2023-07-13 20:08:45 -07:00
sslivkoff
2a0dcd582d add version checks to remote tests 2023-07-11 17:50:00 -07:00
sslivkoff
4a535a1660 add separate figures+tables for success+fail when doing --deep-check 2023-06-27 17:10:53 -07:00
sslivkoff
100412fa49 use print_timestamped() function for DRY 2023-06-27 17:05:02 -07:00
sslivkoff
de4a846bed upgrade to ctc 0.3.7 2023-06-27 12:39:47 -07:00
sslivkoff
b1ceecafff bump to 0.2.5 2023-06-27 00:15:05 -07:00
sslivkoff
d67b6d2136 add eth_feeHistory to equality test 2023-06-27 00:14:36 -07:00
sslivkoff
8ea6d4c7f2 bump to 0.2.4 2023-06-21 17:49:51 -07:00
sslivkoff
5b195625f7 add nbconvert to dependencies 2023-06-21 17:44:36 -07:00
sslivkoff
fbb6b092b1 fix for #7 2023-06-21 17:40:59 -07:00
sslivkoff
b9ae6b26c0
Merge pull request #14 from paradigmxyz/matt/filter-by-name
feat: filter test by test name argument
2023-06-21 17:33:40 -07:00
sslivkoff
16eeb0f202 formatting 2023-06-21 17:30:39 -07:00
sslivkoff
608eefe3cf add --deep-check cli arg for deeper levels of load test validation 2023-06-21 16:18:56 -07:00
sslivkoff
9777865146 add print cli subcommand for summarizing previous tests 2023-06-21 13:58:19 -07:00
sslivkoff
d1aab83bec fix raw output multiprocess buffer overloading issue 2023-06-21 12:45:27 -07:00
Matthias Seitz
ba1cb01dfe feat: filter test by test name argument 2023-06-21 15:14:36 +02:00
sslivkoff
eabbcd7a36 add raw output saving option to remote runs 2023-06-20 12:17:39 -07:00
sslivkoff
8b3de92156 replicate vegeta's reporting pipeline to allow for customization 2023-06-19 23:30:48 -07:00
sslivkoff
643ab9c92e add stub for multi test generators 2023-06-19 21:51:39 -07:00
sslivkoff
29b618d8cb add utils to save, load, and manipulate raw vegeta gob data 2023-06-19 21:51:12 -07:00
sslivkoff
c974d8c198 upgrade to toolstr 0.9.7 2023-06-17 18:49:54 -07:00
sslivkoff
fc2dfcf431 split + refactor runner code and exclude raw test from results payload 2023-06-15 17:02:26 -07:00
sslivkoff
018ae92c9c bump to 0.2.3 2023-06-15 10:45:17 -07:00
sslivkoff
bbd7fe398d upgrade to ctc 0.3.6 2023-06-15 10:44:50 -07:00
sslivkoff
3a89bba9e4 bump to 0.2.2 2023-06-14 23:39:32 -07:00
sslivkoff
35000e9feb expand equality test outputs to show all diffs 2023-06-14 23:36:58 -07:00
sslivkoff
8fad6b4ed2
Merge pull request #16 from JonasPf/allow_url_parameters
only split name and url on the first '='
2023-06-13 10:02:11 -07:00
Jonas Pfannschmidt
23d244d983 only split name and url on the first '='
this allows for providers that use url parameters, e.g.
node1=https://provider.com/eth?authKey=xxx
2023-06-08 12:11:52 +01:00
64 changed files with 3143 additions and 1294 deletions

4
.gitignore vendored