Compare commits

..

164 commits

Author SHA1 Message Date
sslivkoff
e8c046fe7c fix defillama stablecoins schema 2025-10-03 15:32:01 -07:00
sslivkoff
ef08c9df48 add retries and graceful skips to defillama.fees 2025-10-03 15:31:52 -07:00
sslivkoff
1a37adcffa update kalshi collectors 2025-09-26 19:07:09 -07:00
sslivkoff
75c8076b2b fix chunk size check in path generation 2025-09-26 19:06:59 -07:00
sslivkoff
be88e0c500 update defillama schema 2025-09-26 19:06:42 -07:00
sslivkoff
02922990fe fix divide by zero edgecase 2025-09-26 19:06:29 -07:00
sslivkoff
5c77376e4f update timezone checks 2025-09-26 19:06:21 -07:00
sslivkoff
977e921c1b bump to 0.3.1 2025-08-07 23:12:28 -07:00
sslivkoff
094764d6dd
Update README.md 2025-08-07 23:11:35 -07:00
sslivkoff
7eca8a3a38 improve ls verbosity levels 2025-08-06 16:08:41 -07:00
sslivkoff
a62deea1c2 further refinement of Table api and toplevel api 2025-08-06 15:59:21 -07:00
sslivkoff
5af4654e53 add print_table_info as toplevel function 2025-08-06 15:03:35 -07:00
sslivkoff
e0dad1f8bf clean up python api and add cli<-->python equivalence 2025-08-06 14:51:47 -07:00
sslivkoff
232230db25 revise toplevel absorb.__init__ functions 2025-08-06 13:44:11 -07:00
sslivkoff
8baa6c0c9b update readme 2025-08-06 13:28:47 -07:00
sslivkoff
2085a821c4 improve narration of bucket uploads/downloads 2025-08-01 15:30:21 -07:00
sslivkoff
eb98a17463 add better bucket setup 2025-08-01 15:29:58 -07:00
sslivkoff
b3bf46d587 clean verbose output of abs collect 2025-08-01 12:03:25 -07:00
sslivkoff
75e5bc7c2d validate that when write_range=overwrite_all there is at most 1 data file 2025-08-01 01:20:14 -07:00
sslivkoff
82b40a7458 clear old data after collection when write_range=overwrite_all 2025-08-01 01:14:26 -07:00
sslivkoff
1259dad121 chunk_glob --> data_glob 2025-08-01 01:00:36 -07:00
sslivkoff
bd73b70350 clean output of abs info 2025-08-01 00:57:01 -07:00
sslivkoff
72c16f5a5d make query eager unless lazy=True 2025-08-01 00:44:46 -07:00
sslivkoff
7c2309723a standardize allium timestamp column 2025-08-01 00:12:55 -07:00
sslivkoff
72aba68d9f check if rclone_remote is set 2025-08-01 00:12:28 -07:00
sslivkoff
425f4bc78d add in-repo checks for git tracking 2025-08-01 00:12:10 -07:00
sslivkoff
9d3d918784 change ls formatting to trim overflow of each source line when nonverbose 2025-07-29 23:49:34 -07:00
sslivkoff
0c4be15dfd more index unifying 2025-07-29 22:58:28 -07:00
sslivkoff
8aebb9fc74 refine the flowchart of update time, latency, and other temporal table properties 2025-07-29 22:58:15 -07:00
sslivkoff
29ded7d8e2 add --provider arg to upload and download 2025-07-28 15:04:18 -07:00
sslivkoff
fee84ff010 unify all tables on same timestamp name, precision, and tz 2025-07-28 15:04:08 -07:00
sslivkoff
8ba4956b1e separate dynamic table properties into special methods 2025-07-27 20:59:46 -07:00
sslivkoff
128496f0d9 make function names more specific 2025-07-27 17:57:50 -07:00
sslivkoff
1060882324 clean bucket pathing up 2025-07-27 17:45:52 -07:00
sslivkoff
d3ab784089 clean up path functions 2025-07-27 17:25:18 -07:00
sslivkoff
46050d40cb organize config ops by module 2025-07-27 13:33:34 -07:00
sslivkoff
1c7ab33945 ChunkData --> ChunkResult 2025-07-27 12:44:54 -07:00
sslivkoff
a5510bcf55 refactor indexing to use much cleaner abstractions 2025-07-27 12:11:23 -07:00
sslivkoff
8dd7d1cfd7 when parsing table strs, check config before using general search 2025-07-26 10:06:58 -07:00
sslivkoff
6d189de5ed improve validation for missing metadata 2025-07-26 09:51:52 -07:00
sslivkoff
a417abeef8 unify add and remove of cli/python interfaces 2025-07-26 09:43:29 -07:00
sslivkoff
b1fda4b826 add bucket scanning directly to query() 2025-07-26 08:08:31 -07:00
sslivkoff
054210e2a2 bump to 0.3.0 2025-07-25 23:50:27 -07:00
sslivkoff
c12632b367 add [all] option for installations 2025-07-25 23:50:12 -07:00
sslivkoff
a124c45de7 split types module into cleaner set of submodules 2025-07-25 23:45:45 -07:00
sslivkoff
159380bb15 datasets --> catalog 2025-07-25 23:41:38 -07:00
sslivkoff
5d3e4a6d07 add config path to setup output 2025-07-25 23:28:09 -07:00
sslivkoff
1122ffd60e change key bucket-->bucket_name 2025-07-25 16:31:43 -07:00
sslivkoff
c19446e059 enable querying buckets directly 2025-07-25 16:24:58 -07:00
sslivkoff
9e91228a02 add config validation check for dict subkeys 2025-07-25 16:15:43 -07:00
sslivkoff
745377414a fix FRED timestamp datatype 2025-07-25 10:28:57 -07:00
sslivkoff
d704bee71a allow preview to preview uncollected datasets 2025-07-25 10:28:38 -07:00
sslivkoff
3a98654acc simplify table reference parameter flow 2025-07-25 10:22:50 -07:00
sslivkoff
5a01cb5ee6 implement query() which collects uncollected before querying and can update 2025-07-25 09:56:44 -07:00
sslivkoff
4cd4e9de14 add more descriptive error handling and explicit message for --confirm 2025-07-25 09:46:20 -07:00
sslivkoff
3e001b3840 add name validation to table instantiation 2025-07-25 01:05:00 -07:00
sslivkoff
8351aeb1d6 support dune and snowflake backends for sql queries 2025-07-25 00:30:32 -07:00
sslivkoff
652795853a add cli sql subcommand 2025-07-25 00:16:30 -07:00
sslivkoff
0da111d266 add ability to upload and download tables from buckets 2025-07-24 23:32:06 -07:00
sslivkoff
7ae6353d4a print config in abs setup 2025-07-24 23:19:43 -07:00
sslivkoff
a453483926 add name collision check when tracking new tables 2025-07-24 00:36:08 -07:00
sslivkoff
6be5030400 allow info recipe 2025-07-24 00:30:18 -07:00
sslivkoff
96cb9c1c31 allow info source subcommand 2025-07-23 23:36:40 -07:00
sslivkoff
5409ff03af name --> id 2025-07-23 23:03:01 -07:00
sslivkoff
516e12bd71 add enable and disable git operations 2025-07-23 22:58:14 -07:00
sslivkoff
d7764d50ce add --verbose to info 2025-07-23 22:35:00 -07:00
sslivkoff
14e148894e add url to abs info output 2025-07-23 14:32:43 -07:00
sslivkoff
302dc13ddf implement auto schema fetching for snowflake tables 2025-07-23 14:30:53 -07:00
sslivkoff
dd2a5163f5 add url and description metadata to tables 2025-07-23 14:08:42 -07:00
sslivkoff
9a6f4fd016 add proper checks around whether index_type is None for certain operations 2025-07-23 12:16:17 -07:00
sslivkoff
e18e3e1f32 add allium.stablecoin_supply table 2025-07-22 18:17:22 -07:00
sslivkoff
03bc7b75ba update snowflake flow 2025-07-22 18:17:02 -07:00
sslivkoff
dcc8424d2c improve metavar name of cli args 2025-07-22 15:46:59 -07:00
sslivkoff
e7d8fa777b remove redundant path manipulations 2025-07-22 15:34:30 -07:00
sslivkoff
17f0acb945 introduce full_name() where optimal 2025-07-22 14:35:10 -07:00
sslivkoff
8864492f2c put all metadata changes under version control 2025-07-22 14:27:37 -07:00
sslivkoff
a0f357adae add background git tracking of table metadatas 2025-07-22 00:53:11 -07:00
sslivkoff
e9968b9bbd fix dune schema 2025-07-22 00:12:42 -07:00
sslivkoff
7ed879ec58 enable Table.collect_chunk to write files to shortcircuit dataframes 2025-07-22 00:04:12 -07:00
sslivkoff
2618adc90c add validation checks to collected chunks 2025-07-21 23:42:57 -07:00
sslivkoff
4034cbe8ae add warnings for missing packages/credentials when adding new table 2025-07-21 23:33:51 -07:00
sslivkoff
f2db762e2a add --delete option to abs remove 2025-07-21 23:21:45 -07:00
sslivkoff
cb14fd22e7 add global --absorb-root arg and set_absorb_root() function 2025-07-21 23:12:34 -07:00
sslivkoff
c51eaf1315 add Table.description field 2025-07-21 23:12:02 -07:00
sslivkoff
a6463dd47e refactor Table by splitting out creation + naming functions 2025-07-21 22:16:57 -07:00
sslivkoff
1149c85bd0 get_dir_path --> get_table_path 2025-07-21 21:42:40 -07:00
sslivkoff
19ce5c27b6 update spice version 2025-07-21 21:42:17 -07:00
sslivkoff
46779592cb add credential checks prior to data collection 2025-07-21 13:26:50 -07:00
sslivkoff
53702757b1 add proper req's to dune datasets 2025-07-21 10:22:43 -07:00
sslivkoff
7aa77db9f3 clean up metadata path formulation 2025-07-21 00:25:31 -07:00
sslivkoff
54f389b5c3 add placeholder available ranges for binance markets 2025-07-20 20:37:49 -07:00
sslivkoff
abcb78808c fix specific instance name instead of class name in tracked ls 2025-07-20 20:37:36 -07:00
sslivkoff
2631b9e5db improve dynamic name parsing 2025-07-20 20:29:43 -07:00
sslivkoff
12f5751c0d add better error handling 2025-07-20 20:29:27 -07:00
sslivkoff
1d5d15eafb use placeholder for binance available range 2025-07-20 20:28:43 -07:00
sslivkoff
48e79c9eb7 simplify much of the table parsing logic 2025-07-20 17:43:31 -07:00
sslivkoff
919cf25e9b enable interactive data previews 2025-07-20 14:24:18 -07:00
sslivkoff
1e4d63f563 fix slow test by hardcoding schema instead of using dynamic 2025-07-20 13:56:12 -07:00
sslivkoff
8ff79a1ee1 refactor TableDict handling and flow 2025-07-20 13:37:26 -07:00
sslivkoff
13fba0ac5e separate validation logic into separate module 2025-07-20 12:42:36 -07:00
sslivkoff
e9e28fe780 load table parameters from filesystem instead of regeneration 2025-07-20 11:58:25 -07:00
sslivkoff
15ce34d64d add config regeneration to setup 2025-07-19 13:25:55 -07:00
sslivkoff
6199a194a6 add validity checks for tracked metadata 2025-07-19 00:09:51 -07:00
sslivkoff
c920d8a9bf enforce that parameter_types are strict types rather than annotations 2025-07-19 00:09:33 -07:00
sslivkoff
0398162960 enforce TableDict parameters are JSON-serializable 2025-07-19 00:09:33 -07:00
sslivkoff
d503db01f9 use better table name template matching 2025-07-18 23:09:50 -07:00
sslivkoff
83d97defbf add more data integrity checks 2025-07-18 22:59:06 -07:00
sslivkoff
b18635244f clean up binance names 2025-07-18 22:04:31 -07:00
sslivkoff
2d1b513fc9 refactor default parameter system 2025-07-18 22:02:17 -07:00
sslivkoff
b8041b9d35 add setup and validate subcommands 2025-07-18 21:18:13 -07:00
sslivkoff
7a75efb0cc comments in Table for clarity 2025-07-18 17:24:35 -07:00
sslivkoff
8ff36118a0 track table metadata in filesystem 2025-07-18 16:55:30 -07:00
sslivkoff
5db2df2e0f track table versioning 2025-07-18 16:55:06 -07:00
sslivkoff
af22856247 remove struct fields from kalshi metadata 2025-07-18 16:54:46 -07:00
sslivkoff
22c2a36501 rename chain_ids.py --> chains.py 2025-07-18 13:44:09 -07:00
sslivkoff
1cdb1db7af fix cd hidden arguments 2025-07-18 13:18:13 -07:00
sslivkoff
b7b6c8bed4 add abs preview 2025-07-16 17:54:09 -07:00
sslivkoff
8b38174917 TrackedTable --> TableDict 2025-07-16 16:58:21 -07:00
sslivkoff
320e7b37b7 fix cd setup instructions 2025-07-16 16:48:08 -07:00
sslivkoff
8ebcbbe433 improve ls outputs 2025-07-16 16:40:08 -07:00
sslivkoff
65b186c4b7 add tables doc 2025-07-16 15:58:38 -07:00
sslivkoff
2eb4d323d7 make annotations consistent 2025-07-15 22:02:56 -07:00
sslivkoff
210791934c add initial message about number of datasets being collected 2025-07-15 15:43:16 -07:00
sslivkoff
093d5a1e0f add more automation of dateset updates 2025-07-15 13:23:40 -07:00
sslivkoff
149bf96e57 rearrange dependencies 2025-07-15 12:26:23 -07:00
sslivkoff
2a44511e28 add package installation checks 2025-07-13 08:50:24 -07:00
sslivkoff
9a1a90a187 add status info to info subcommand 2025-07-12 22:05:27 -07:00
sslivkoff
f8124ef538 add tests for get_schema() 2025-07-12 21:45:30 -07:00
sslivkoff
ae1cb63113 clean toplevel module imports 2025-07-12 21:22:56 -07:00
sslivkoff
94252352a4 import polars as needed 2025-07-12 21:22:44 -07:00
sslivkoff
24507797aa run experimental ty type checker 2025-07-12 20:15:39 -07:00
sslivkoff
c1a81e2664 remove flipside as data source 2025-07-12 19:47:53 -07:00
sslivkoff
5e2fc501eb simply name formulation logic 2025-07-12 10:04:39 -07:00
sslivkoff
5db9fb8bae fix class searching for source-specific parsing 2025-07-11 20:17:07 -07:00
sslivkoff
5aa39dcd18 use individual class in dateset parser 2025-07-11 19:28:17 -07:00
sslivkoff
234d516db9 fix range diff tests 2025-07-11 19:27:44 -07:00
sslivkoff
e6334e9a5a add snake camel conversion tests 2025-07-11 19:18:08 -07:00
sslivkoff
9fc5a6e69f sort ls entries 2025-07-11 19:08:17 -07:00
sslivkoff
a4c716df07 modularize dataset parsing 2025-07-11 18:26:21 -07:00
sslivkoff
081b3ca444 update l2beat schema 2025-07-11 18:26:12 -07:00
sslivkoff
a33a9b1f01 add source argument to ls 2025-07-11 09:57:46 -07:00
sslivkoff
2bc8c8f46a add --collected argument to add 2025-07-11 09:41:29 -07:00
sslivkoff
c5236b7e8b formatting 2025-07-10 21:26:19 -07:00
sslivkoff
2129e6598e in coverage_to_list, also convert lists of ranges to simple lists 2025-07-10 21:26:06 -07:00
sslivkoff
4c12a2b7ca fix error message variables 2025-07-10 21:25:23 -07:00
sslivkoff
2d4fbdec00 add cd subcommand 2025-07-10 21:25:08 -07:00
sslivkoff
0dbcd51c5a add color to tracebacks if avaialable 2025-07-10 12:44:12 -07:00
sslivkoff
ce1971b15b get debugger dependency in proper group 2025-07-09 23:28:09 -07:00
sslivkoff
87e1736187 git ignore pytestdebug.log 2025-07-09 23:06:40 -07:00
sslivkoff
73432326e4 clean up types for mypy 2025-07-09 21:30:18 -07:00
sslivkoff
79e5438066 add optional deps to pyproject.toml 2025-07-09 18:11:50 -07:00
sslivkoff
ea60121b77 change help subcommand to info 2025-07-09 16:30:11 -07:00
sslivkoff
b2a4150317 add mypy as dev dependency 2025-07-09 16:29:59 -07:00
sslivkoff
381ced522f change TRUCK_ROOT to ABSORB_ROOT 2025-07-09 14:02:23 -07:00
sslivkoff
b88c196276 update missing env var warning 2025-07-09 13:51:02 -07:00
sslivkoff
cf6c0bcee9 update pyproject.toml for abs executable and venv config 2025-07-09 13:50:16 -07:00
sslivkoff
e3ef4827a0 short circuit collected table getter if root DNE 2025-07-07 16:37:14 -07:00
sslivkoff
90495dfc57 update min python version to satisfy uv dependency resolver 2025-07-07 16:36:46 -07:00
sslivkoff
3f98f48dfe ignore uv versioning files 2025-07-07 16:36:26 -07:00
sslivkoff
0c547a919d add collected untracked datasets to ls 2025-06-22 17:40:41 -07:00
sslivkoff
5f29b6bd19 use index_type=None for coingecko.categories 2025-06-22 17:37:15 -07:00
sslivkoff
22f000428a for overwrite_all datasets use None as default available range 2025-06-22 17:35:26 -07:00
sslivkoff
d17002d09e
accomodate non-temporally indexed overwrite_all datasets (#1) 2025-06-12 10:09:29 +02:00
sslivkoff
eb8e19a8b7 add pair column to binance tables 2025-06-01 10:31:44 -07:00
141 changed files with 9906 additions and 5980 deletions

4
.gitignore vendored

Some files were not shown because too many files have changed in this diff Show more