Compare commits

..

150 commits

Author SHA1 Message Date
Andre Miras
03f93c494d
Fix missing curl binary and update CI (#45)
Fix parquet file download error because of a missing curl binary.
This was happening on the `eth_getBalance` test for instance.
Note that flood swallowed the exception making the error harder to spot.

Update the CI to also test against eth_getBalance which triggers a
parquet file download.
Also removes the redundant node2, this was a workaround as flood used to
crash with only one node.

Also unpin the patch version of Python and address a small DRY on the
Vegeta version.
2023-08-30 14:23:24 -07:00
Zane Peycke
eb88711615
Update README.md with report example (#44)
Add existing report example link in report generation section
2023-08-30 14:05:10 -07:00
sslivkoff
61644933f9 bump to 0.3.1 2023-07-28 16:09:54 -04:00
sslivkoff
c8a153a4a9 fix version checks for tagged commits 2023-07-28 16:09:35 -04:00
sslivkoff
dfa85e9779 bump to 0.3.0 2023-07-28 15:05:52 -04:00
sslivkoff
f1162e5592 clean up load test output 2023-07-28 14:52:14 -04:00
sslivkoff
453f5a4da4 update reports with more info 2023-07-28 14:35:28 -04:00
sslivkoff
18e182a147 fix single node report generation 2023-07-28 01:31:12 -04:00
sslivkoff
5b169dbbdb
add ToC to README.md 2023-07-27 14:33:19 -07:00
sslivkoff
7b22858f34 move user guide into readme 2023-07-27 17:24:07 -04:00
sslivkoff
f899efc9c9 update report generation pipeline for namespace refactor 2023-07-27 17:20:18 -04:00
sslivkoff
180545e986 remove spurious report 2023-07-27 16:53:29 -04:00
sslivkoff
a1a585f204
clean toplevel namespace (#39)
* clean toplevel namespace

* ruff checks

* update tests to new namespace refactor
2023-07-27 13:35:02 -07:00
sslivkoff
61104d47fb add docker notes 2023-07-27 16:06:09 +00:00
sslivkoff
8e2cdc2499
Improve docs (#38)
* improve docs

* add using --version or -V instead of just "version"
2023-07-27 08:02:48 -07:00
sslivkoff
67d3695242 fix ruff regression 2023-07-25 19:31:23 -07:00
Landon Gingerich
4d3776b266 fix Vegeta location 2023-07-25 19:31:23 -07:00
Landon Gingerich
0e0b58c0c2 use slim python image and multi-stage build 2023-07-25 19:31:23 -07:00
Andre Miras
9e698b0bff
Docker run from the CI, fixes #34 (#35)
Run a light test agaisnt the llama public RPC.
Note while this is working for now the GitHub runner IPs could get
blacklisted as people abuse it. Updating the CI to use an API key would
solve it.
2023-07-18 23:18:48 -07:00
sslivkoff
1c094f9d92 Merge branch 'main' of github.com:paradigm-operations/rpc_bench 2023-07-17 15:35:03 -07:00
sslivkoff
923f8838e6 add aggregation mode to print cli command 2023-07-17 15:34:46 -07:00
Andre Miras
dcf73c57f0
Drop the checkthechain clone (#32)
As checkthechain is now part of the `pyproject.toml`
2023-07-17 00:42:31 -07:00
Andre Miras
d42318d3ed
Add Docker build to the CI (#31) 2023-07-17 00:37:46 -07:00
Andre Miras
b86e1cc589
Fix polars dependency (#30)
The error was:
```
The conflict is caused by:
ERROR: ResolutionImpossible: for help visit
https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts
    paradigm-flood 0.3.0a0 depends on polars<0.19 and >=0.18.7
    checkthechain 0.3.9 depends on polars<0.18 and >=0.17.3
```
Refs:
https://github.com/paradigmxyz/flood/actions/runs/5571608157/jobs/10176802894
2023-07-17 00:30:55 -07:00
sslivkoff
c897403013 move metadata into SingleRunTestPayload instead of RunOutput 2023-07-16 20:57:19 -07:00
sslivkoff
3cbf6fcb26 save additional metadata alongside each test 2023-07-16 20:50:34 -07:00
sslivkoff
f5700e0d3e clean output formatting of deep check summaries 2023-07-16 18:05:11 -07:00
sslivkoff
1ba1b28d64 flesh put deep check metrics for failed requests 2023-07-16 17:19:57 -07:00
sslivkoff
0167d65ada change --nodes to a positional argument in update command 2023-07-16 16:08:38 -07:00
sslivkoff
6f70dafe62 use orjson for encoding/decoding of large json 2023-07-16 15:54:22 -07:00
sslivkoff
4bc0663f21 add eth_feeHistory load test 2023-07-16 14:28:53 -07:00
sslivkoff
ca80e9cbd5 improve cli arg help messages 2023-07-16 14:11:55 -07:00
sslivkoff
d3f6db6fa2 use timestamp as default random seed 2023-07-16 14:05:20 -07:00
sslivkoff
6cf83fd5d7 specify --work-tree alongside --git-dir in update command 2023-07-16 12:37:03 -07:00
sslivkoff
048808d163 completely avoid generating test clientside when running remote tests 2023-07-16 12:26:48 -07:00
sslivkoff
15c7f3b590 allow generating xxl block ranges using only xl sample 2023-07-16 11:23:21 -07:00
sslivkoff
d7b36bc4a0 update to toolcli 0.6.16 2023-07-16 11:00:29 -07:00
sslivkoff
7cd1ac07c1 fix cli passthrough of single vegeta args 2023-07-16 10:45:35 -07:00
sslivkoff
b499c375fb implement remote version of update 2023-07-15 19:52:09 -07:00
sslivkoff
c9c9a97163 change vegeta_kwargs to vegeta_args 2023-07-15 19:45:27 -07:00
sslivkoff
8734a749d2 fix check on null git dir 2023-07-15 17:21:59 -07:00
sslivkoff
d5c8a7cbfc fix pass vegeta_kwargs through call chain 2023-07-15 17:21:03 -07:00
sslivkoff
b40ab09e84 add update command 2023-07-15 17:19:21 -07:00
sslivkoff
a454760d42 follow symlinks of installation path 2023-07-15 16:00:59 -07:00
sslivkoff
50ce90a9cd expand installation metadata and add update process 2023-07-15 15:47:02 -07:00
Andre Miras
3f52adb037
fix pyproject.toml dependencies (#25)
1) adding `ipykernel` fixed:
```
jupyter_client.kernelspec.NoSuchKernel: No such kernel named python3
```

2) adding `ipython_genutils` fixed:
```
ModuleNotFoundError: No module named 'ipython_genutils'
```

3) bumping the `nbconvert` version fixed:
```
ImportError: cannot import name 'contextfilter' from 'jinja2'
(/home/flood/.local/lib/python3.11/site-packages/jinja2/__init__.py)
```

This was tested within a Docker container, built with:
```
docker build --tag paradigmxyz/flood:latest .
```
Ran with:
```
docker run --rm --volume ${PWD}/output:/output paradigmxyz/flood:latest \
eth_getBlockByNumber NODE:8545 --output /output/
```
2023-07-16 00:11:01 +03:00
sslivkoff
6342f74dfa use custom version command to include commit hash and remotes 2023-07-15 10:59:31 -07:00
sslivkoff
6b91c743b4 Merge branch 'main' of github.com:paradigm-operations/rpc_bench 2023-07-15 10:46:44 -07:00
sslivkoff
d55af3f944 have run() return test outputs for python calls 2023-07-15 10:42:09 -07:00
sslivkoff
08741652eb add commit hash to installation.flood_version 2023-07-15 09:24:02 -07:00
Andre Miras
da1382c3b4
Add vegeta PATH in Docker, fix #24 (#26)
The results.json was not generated because vegeta was not found.
For some reason the error was caught and silenced which made it harder
to debug.
2023-07-15 14:28:22 +03:00
sslivkoff
68e285665e add initial log load test parameterizations 2023-07-14 18:33:18 -07:00
sslivkoff
9c6e32705d ruff linelength lints 2023-07-14 16:04:00 -07:00
sslivkoff
e3256e1463 bump to latest ctc for debug traces 2023-07-14 16:03:09 -07:00
sslivkoff
f17215fca7 add debug_traceX methods to equality test 2023-07-14 15:39:44 -07:00
sslivkoff
9a92f3c2a8 add various stateDiff and vmTrace calls to equality tests 2023-07-14 13:08:04 -07:00
sslivkoff
572c3f0c9d clip equality test diff summaries that are >100 lines long 2023-07-14 13:07:32 -07:00
sslivkoff
7a0e7ec406 bump to toolcli 0.6.15 2023-07-14 12:42:56 -07:00
sslivkoff
cf288b0a7b show client version on multiple lines for clarity 2023-07-14 01:11:40 -07:00
sslivkoff
d0e4dd6918 fix noqa format for github version of ruff 2023-07-14 00:40:24 -07:00
sslivkoff
cdc736ec70 fix deep check tests 2023-07-14 00:26:19 -07:00
sslivkoff
d0ab7e175b clean output messages 2023-07-13 21:05:19 -07:00
sslivkoff
fd64682707 tweaks for local testing 2023-07-13 20:47:58 -07:00
sslivkoff
af14a3fd8f bump version to 0.3.0a for testing 2023-07-13 20:09:02 -07:00
sslivkoff
fc0ed5a940 refactor of many data structures and pipelines 2023-07-13 20:08:45 -07:00
sslivkoff
2a0dcd582d add version checks to remote tests 2023-07-11 17:50:00 -07:00
sslivkoff
4a535a1660 add separate figures+tables for success+fail when doing --deep-check 2023-06-27 17:10:53 -07:00
sslivkoff
100412fa49 use print_timestamped() function for DRY 2023-06-27 17:05:02 -07:00
sslivkoff
de4a846bed upgrade to ctc 0.3.7 2023-06-27 12:39:47 -07:00
sslivkoff
b1ceecafff bump to 0.2.5 2023-06-27 00:15:05 -07:00
sslivkoff
d67b6d2136 add eth_feeHistory to equality test 2023-06-27 00:14:36 -07:00
sslivkoff
8ea6d4c7f2 bump to 0.2.4 2023-06-21 17:49:51 -07:00
sslivkoff
5b195625f7 add nbconvert to dependencies 2023-06-21 17:44:36 -07:00
sslivkoff
fbb6b092b1 fix for #7 2023-06-21 17:40:59 -07:00
sslivkoff
b9ae6b26c0
Merge pull request #14 from paradigmxyz/matt/filter-by-name
feat: filter test by test name argument
2023-06-21 17:33:40 -07:00
sslivkoff
16eeb0f202 formatting 2023-06-21 17:30:39 -07:00
sslivkoff
608eefe3cf add --deep-check cli arg for deeper levels of load test validation 2023-06-21 16:18:56 -07:00
sslivkoff
9777865146 add print cli subcommand for summarizing previous tests 2023-06-21 13:58:19 -07:00
sslivkoff
d1aab83bec fix raw output multiprocess buffer overloading issue 2023-06-21 12:45:27 -07:00
Matthias Seitz
ba1cb01dfe feat: filter test by test name argument 2023-06-21 15:14:36 +02:00
sslivkoff
eabbcd7a36 add raw output saving option to remote runs 2023-06-20 12:17:39 -07:00
sslivkoff
8b3de92156 replicate vegeta's reporting pipeline to allow for customization 2023-06-19 23:30:48 -07:00
sslivkoff
643ab9c92e add stub for multi test generators 2023-06-19 21:51:39 -07:00
sslivkoff
29b618d8cb add utils to save, load, and manipulate raw vegeta gob data 2023-06-19 21:51:12 -07:00
sslivkoff
c974d8c198 upgrade to toolstr 0.9.7 2023-06-17 18:49:54 -07:00
sslivkoff
fc2dfcf431 split + refactor runner code and exclude raw test from results payload 2023-06-15 17:02:26 -07:00
sslivkoff
018ae92c9c bump to 0.2.3 2023-06-15 10:45:17 -07:00
sslivkoff
bbd7fe398d upgrade to ctc 0.3.6 2023-06-15 10:44:50 -07:00
sslivkoff
3a89bba9e4 bump to 0.2.2 2023-06-14 23:39:32 -07:00
sslivkoff
35000e9feb expand equality test outputs to show all diffs 2023-06-14 23:36:58 -07:00
sslivkoff
8fad6b4ed2
Merge pull request #16 from JonasPf/allow_url_parameters
only split name and url on the first '='
2023-06-13 10:02:11 -07:00
Jonas Pfannschmidt
23d244d983 only split name and url on the first '='
this allows for providers that use url parameters, e.g.
node1=https://provider.com/eth?authKey=xxx
2023-06-08 12:11:52 +01:00
sslivkoff
651044eea9 add example scripts in /examples 2023-06-06 14:21:17 -07:00
sslivkoff
09b75f22d0
Update tests.yml 2023-06-06 14:10:09 -07:00
sslivkoff
d295faa0fa skip node tests if env vars unset 2023-06-06 14:04:49 -07:00
sslivkoff
e1036395e7 apply ruff linting 2023-06-06 13:44:44 -07:00
Georgios Konstantopoulos
b957910ca1 docs: add tg 2023-06-06 10:16:53 -07:00
Georgios Konstantopoulos
0fd5c75cb8
ci: add tests/lint & issue/pr templates (#11)
* ci: scaffold

* add issue/pr templates

* pyproject: add ruff config

* docs: add badge + tg

* chore: fix pip install

* docs: add cover
2023-06-06 10:04:40 -07:00
sslivkoff
36e0fba27e bump to 0.2.1 2023-06-06 08:07:28 -07:00
sslivkoff
7430530cf7 fix report notebook variable name 2023-06-05 21:51:27 -07:00
sslivkoff
3e2d3a2bb6 bump to 0.2.0 2023-06-05 18:37:35 -07:00
sslivkoff
68003a83ef add note about python3 version 2023-06-05 18:37:07 -07:00
sslivkoff
ba19d5f73b clean dependencies and simplify install instructions 2023-06-05 18:02:41 -07:00
sslivkoff
ea2ebab80d run black repo wide 2023-06-04 16:16:10 -07:00
sslivkoff
6690e77bee add unit tests for load tests 2023-06-04 14:29:54 -07:00
sslivkoff
21486b8c5f add testing rails and unit test for equality test 2023-06-04 13:53:00 -07:00
sslivkoff
687ebc1d85 revise README.md 2023-06-04 13:47:08 -07:00
sslivkoff
6311527407 add LICENSE files and CONTRIBUTING.md 2023-06-04 13:22:48 -07:00
sslivkoff
815b72ead5 change display of remote node name 2023-06-04 13:10:15 -07:00
sslivkoff
eb1ecaaf5f add more color profiles and pin profiles to known clients 2023-06-04 13:05:09 -07:00
sslivkoff
20b8cd2b4f fix node parsing for equality test and prepare for network chain_id parsing 2023-06-04 09:09:11 -07:00
sslivkoff
756f3add06 parallelize web3_clientVersion calls to nodes 2023-06-03 23:55:48 -07:00
sslivkoff
5f08c290bf use web3_clientVersion during parsing to get client info 2023-06-03 16:05:17 -07:00
sslivkoff
9028a8160a add tests for eth_call, state diff tracing, and vm tracing 2023-06-03 14:43:01 -07:00
sslivkoff
6f72d231f0 clean up namespace 2023-06-03 13:47:12 -07:00
sslivkoff
80becea8ff lazy import for better startup time 2023-06-03 13:20:57 -07:00
sslivkoff
0d2763f8cc add rng_utils and propagate random_seed throughout 2023-06-03 13:09:28 -07:00
sslivkoff
4e6f2aa0ec add json dumps of equality test 2023-06-03 11:10:42 -07:00
sslivkoff
e0da2b8295 adjust report figure legend locations 2023-06-03 10:52:42 -07:00
sslivkoff
61ad830417 fix network specification for eth_get_logs 2023-06-02 23:40:15 -07:00
sslivkoff
609cf10304 add global start and end times to test output 2023-06-02 23:32:06 -07:00
sslivkoff
02da57d432 fix eoa column 2023-06-02 22:47:26 -07:00
sslivkoff
fec286ffcc upgrade to toolcli 0.6.14 to fix help doc piping 2023-06-02 22:14:26 -07:00
sslivkoff
af343711be add placeholder link to example report in README.md 2023-06-02 20:27:46 -07:00
sslivkoff
d988de22df fix http vs https prefix in parsing test 2023-06-02 20:18:45 -07:00
sslivkoff
37b20540aa use auto-downloading of raw evm data samples in background when needed 2023-06-02 20:06:50 -07:00
sslivkoff
89819421e1 add black formatter settings to pyproject.toml 2023-06-02 19:41:33 -07:00
sslivkoff
5834d0638e implement concurrent tests for remote nodes 2023-06-02 19:23:34 -07:00
sslivkoff
07255df2df increase verbosity of equality test outputs 2023-06-02 17:32:31 -07:00
sslivkoff
b932348e37 fix numpy vs native int typing 2023-06-02 16:47:33 -07:00
sslivkoff
8e4b3d0e2b trace tests 2023-06-02 16:29:24 -07:00
sslivkoff
28a6104061 add unified raw object generation pipeline and multinetwork handling 2023-06-02 16:29:12 -07:00
sslivkoff
b4afbeafc7 use new generation pipeline for object generation 2023-06-02 14:37:11 -07:00
sslivkoff
81cfb48705 Merge branch 'main' of github.com:paradigm-operations/rpc_bench 2023-06-02 08:37:19 -07:00
sslivkoff
6fb341bc24
Update README.md 2023-06-02 08:35:08 -07:00
sslivkoff
73b4594eed add cd subcommand 2023-06-02 00:16:10 -07:00
sslivkoff
135790112e add samples ls subcommand 2023-06-01 23:11:35 -07:00
sslivkoff
8d51d9526f add verbose output to flood samples collect 2023-06-01 21:27:06 -07:00
sslivkoff
1911d5c943 reorg tests dir and remove old tests 2023-06-01 21:19:42 -07:00
sslivkoff
544471df96 add raw samples subcommands 2023-06-01 21:16:14 -07:00
sslivkoff
90668f74f6 add pdp to depdendencies 2023-06-01 21:02:36 -07:00
sslivkoff
b7c98b222a add utils for raw_data_sources 2023-06-01 17:53:19 -07:00
sslivkoff
3484367da3
Update README.md 2023-06-01 11:06:49 -07:00
sslivkoff
231fdeb347 change code directory to flood 2023-06-01 17:43:51 +00:00
sslivkoff
9ac7a60ccc rename to flood 2023-06-01 17:40:40 +00:00
sslivkoff
0211a15f95
Merge pull request #4 from paradigmxyz/matt/randomize-equality-tests
feat: use random blocks for equality tests
2023-06-01 10:29:45 -07:00
sslivkoff
d4ff8fa0bf
Merge pull request #1 from paradigmxyz/matt/fix-instructions
docs: use repository url
2023-06-01 10:02:46 -07:00
sslivkoff
2522d3a2d3
Merge pull request #5 from paradigmxyz/matt/add-deps
chore(deps): add missing packages to dependencies
2023-06-01 10:02:16 -07:00
Matthias Seitz
5e7f2438a9 chore(deps): add missing packages to dependencies 2023-05-30 14:36:50 +02:00
Matthias Seitz
b223965ed6 docs: use repository url 2023-05-16 13:26:11 +02:00
129 changed files with 8483 additions and 4817 deletions

4
.gitignore vendored

Binary file not shown.

After

Width:  |  Height:  |  Size: 2.9 MiB

82
examples/report.sh Executable file

Some files were not shown because too many files have changed in this diff Show more