dunebuggy
dunebuggy copied to clipboard
rysuds
Lightweight Python Client for Dune Analytics
Dunebuggy
Installation
pip install dunebuggy
Getting started
Retrieving a public query
To retrieve a query, all we'll need is the query_id for the public query we're interested in. In the below example we can take a look at the popular "Custom NFT Floor Tracker" query by @smaroo (The query_id below can be found in the URL).
from dunebuggy import Dune
dune = Dune()
query = dune.fetch_query(83579)
query here is a DuneQuery object, we can get the pandas DataFrame for the query output bf calling df on the object
print(query.df.head())
Floor (Approx)
Time Interval
0
0.122649
2021-06-01T00:00:00+00:00
1
0.130000
2021-06-02T00:00:00+00:00
2
0.193455
2021-06-03T00:00:00+00:00
3
0.189000
2021-06-04T00:00:00+00:00
4
0.189930
2021-06-05T00:00:00+00:00
We can also take a look at some basic information about the returned query with query.info
print(query.info)
{'name': 'Custom NFT Floor Tracker',
'author': '@smaroo',
'length': 264,
'query_id': 83579,
'result_id': UUID('e5aef8a0-1453-44d1-a27b-f576ea2b3ba2'),
'job_id': UUID('ec680fa9-217f-44c5-b223-56730cd07473'),
'columns': ['Time Interval', 'Floor (Approx)']}
Some queries in Dune are "parameterized", meaning that the author exposes certain variables for the user to enter custom values. The example query (83579) happens to be parameterized, we can verify this by inspecting query.parameters
print(query.parameters)
[QueryParameter(key='Enter NFT Contract Address', type='text', value='xc3f733ca98e0dad0386979eb96fb1722a1a05e69', enumOptions=None),
QueryParameter(key='Floor Time Interval', type='enum', value='Day', enumOptions=['Day', 'Hour']),
QueryParameter(key='Start Date', type='datetime', value='2021-06-01 00:00:00', enumOptions=None)]
If you'd like to run this query with your own custom parameters, all we'll need to do is take the parameters from from the initial query, change the values to what we want, and re-fetch the query.
Below we are replacing the old NFT contract address param with a new one (the contract address for BAYC)
params = query.parameters
# Replacing with contract address for BAYC
params[0].value = 'xBC4CA0EdA7647A8aB7C2061c2E118A18a936f13D'
custom_query = dune.fetch_query(83579, parameters=params)
Note: You can also create a fresh set of parameters by importing QueryParameter from dunebuggy.models.query and adding the values to the new object.
from dunebuggy.models.query import QueryParameter
param_to_change = QueryParameter(
key='Enter NFT Contract Address',
value='xBC4CA0EdA7647A8aB7C2061c2E118A18a936f13D',
)
params[0] = param_to_change
custom_query = dune.fetch_query(83579, parameters=params)
print(custom_query.info)
{'name': 'Custom NFT Floor Tracker',
'author': '@smaroo',
'length': 265,
'query_id': 83579,
'result_id': UUID('42a3c13d-5fbd-42bd-86c0-acc9adcdc803'),
'job_id': UUID('9051ebe7-862f-46d0-9999-b4645659ca56'),
'columns': ['Time Interval', 'Floor (Approx)']}
Note that the result_id and job_id here are different, this is because we ran the query with our changed params
print(custom_query.parameters)
[QueryParameter(key='Enter NFT Contract Address', type='text', value='xBC4CA0EdA7647A8aB7C2061c2E118A18a936f13D', enumOptions=None),
QueryParameter(key='Floor Time Interval', type='enum', value='Day', enumOptions=['Day', 'Hour']),
QueryParameter(key='Start Date', type='datetime', value='2021-06-01 00:00:00', enumOptions=None)]
print(custom_query.df.head())
Floor (Approx)
Time Interval
0
0.8000
2021-06-01T00:00:00+00:00
1
0.8518
2021-06-02T00:00:00+00:00
2
0.8260
2021-06-03T00:00:00+00:00
3
0.7400
2021-06-04T00:00:00+00:00
4
0.8499
2021-06-05T00:00:00+00:00
Creating a new query
dunebuggy also allows you to create a new using an existing Dune.com account.To login just need to pass in your username/password into the Dune object.
You can verify your login by inspecting your Dune user_id
import os
username = os.environ.get('DUNE_USERNAME')
password = os.environ.get('DUNE_PASSWORD')
dune = Dune(username=username, password=password)
# print(dune.user_id)
To create a query now, all we need to do is pass in a name, query_string and dataset_id
We can construct the SQL query by using a raw sql string
query_string = "select * from ethereum.transactions\nLIMIT 100\n"
Or we could use a fancy ORM-style library like pypika
from pypika import Database, Query
ethereum = Database('ethereum')
q = Query.from_(ethereum.transactions).select('*').limit(100)
query_string = q.get_sql(quote_char=None)
print(query_string)
'SELECT * FROM ethereum.transactions LIMIT 100'
Dune requires us to specify a dataset_id for each of their supported blockchain datasets upon query creation. The currently supported datasets are the following:
Blockchain Dataset
Id
ETHEREUM
4
XDAI
6
POLYGON
7
OPTIMISM_1
8
OPTIMISM_2
10
BINANCE
9
SOLANA
1
We can access these integer codes via the DatasetId enum
from dunebuggy.models.constants import DatasetId
created_query = dune.create_query("My Query's Name", query_string, DatasetId.ETHEREUM)
Our created query can be accessed like any other, you can also log into your Dune account as see it there as well!
print(created_query.df.head())
access_list
block_hash
block_number
block_time
data
from
gas_limit
gas_price
gas_used
hash
index
max_fee_per_gas
max_priority_fee_per_gas
nonce
priority_fee_per_gas
success
to
type
value
0
None
\x887c665b0c52ccace092d817e984e2e828ef59079295...
47287
2015-08-07T08:50:01+00:00
None
\xdb312d1d6a2ccc64dd94a3892928bac82b4e8c15
21000
100000000000
21000
\xd3e6a2fc34066d20bb83020b1ee95b9dc7919fd242bd...
0
None
None
0
None
None
\x34bb6978c5a1ad68777ad388c6787df53903430c
None
1000000000000000000
1
None
\x4869e218b0a8f5784f16193ac66cbf35c4510ace0c9b...
48698
2015-08-07T15:29:53+00:00
None
\x48040276e9c17ddbe5c8d2976245dcd0235efa43
90000
57550496008
21000
\x8ba39f908731171fe96ee4e700e71d170ef8e651fac7...
0
None
None
0
None
None
\xd8d0549637b65d58e7fb6cbdd11530b399d1ddac
None
100000000000000000000
2
None
\xab9491b62b16bd928b281a83db82483584c22aeebc0d...
49051
2015-08-07T17:03:48+00:00
None
\x8686578c4f7c75246f548299d6ffdac3b67b5cd1
90000
57178423039
21000
\x57f8ba638903d6335e211eb470159587c73316788880...
0
None
None
0
None
None
\x87abffa6b80f712c852a9558120ba6611f0b5e46
None
45150000000000000000
3
None
\x1f9adc2190701ca3085b28252e4f1f467d980f763dad...
49174
2015-08-07T17:41:03+00:00
None
\x18e4ce47483b53040adbab35172c01ef64506e0c
90000
58589751415
21000
\xb8280da44f8d35011c3f431f7d1a82213477a4e742de...
2
None
None
0
None
None
\xfb26ae2d3621829472555fbd11bb2a324b7a5c57
None
10000000000000000000
4
None
\xf1f392fd197a149afe9f8843d7ba759d1a9f79d1ef62...
49938
2015-08-07T21:06:21+00:00
None
\xc6bf5b6558f2ee21f2e43d9ff9b5408a0cb89413
90000
71214529679
21000
\x538e1664c12c55287c98dc5dd248f60c642cbbbd7a18...
0
None
None
4
None
None
\x33a3f479f6c3e7f91128348490d1f7e8d2a0fab5
None
5000000000000000000
Saving to CSV
To save a query to a CSV, we can take advantage of the to_csv method on our df
created_query.df.to_csv('my_test_data.csv')
Roadmap
- [ ] Cleanup punding TODO comments
- [ ] Add support for embedding Dune graphs/ plotting w/ Dune style colors
- [ ] Add tests (lol)
- [ ] Add support for query updating
- [ ] Investigate whether dashboard support makes sense?
- [ ] Investigate whether there is a max row limit for data returned, if so, query in batches?
- [ ] Better formatting for certain returned columns (links etc..)
- [ ] Add Documentation (Sphinx or something else)
Notes
This project was inspired by the itzemstar's duneanalytics repo
README image is from the IAN Symbols dataset
Installation
pip install dunebuggy
Getting started
Retrieving a public query
To retrieve a query, all we'll need is the query_id for the public query we're interested in. In the below example we can take a look at the popular "Custom NFT Floor Tracker" query by @smaroo (The query_id below can be found in the URL).
from dunebuggy import Dune
dune = Dune()
query = dune.fetch_query(83579)
query here is a DuneQuery object, we can get the pandas DataFrame for the query output bf calling df on the object
print(query.df.head())
| Floor (Approx) | Time Interval | |
|---|---|---|
| 0 | 0.122649 | 2021-06-01T00:00:00+00:00 |
| 1 | 0.130000 | 2021-06-02T00:00:00+00:00 |
| 2 | 0.193455 | 2021-06-03T00:00:00+00:00 |
| 3 | 0.189000 | 2021-06-04T00:00:00+00:00 |
| 4 | 0.189930 | 2021-06-05T00:00:00+00:00 |
We can also take a look at some basic information about the returned query with query.info
print(query.info)
{'name': 'Custom NFT Floor Tracker',
'author': '@smaroo',
'length': 264,
'query_id': 83579,
'result_id': UUID('e5aef8a0-1453-44d1-a27b-f576ea2b3ba2'),
'job_id': UUID('ec680fa9-217f-44c5-b223-56730cd07473'),
'columns': ['Time Interval', 'Floor (Approx)']}
Some queries in Dune are "parameterized", meaning that the author exposes certain variables for the user to enter custom values. The example query (83579) happens to be parameterized, we can verify this by inspecting query.parameters
print(query.parameters)
[QueryParameter(key='Enter NFT Contract Address', type='text', value='xc3f733ca98e0dad0386979eb96fb1722a1a05e69', enumOptions=None),
QueryParameter(key='Floor Time Interval', type='enum', value='Day', enumOptions=['Day', 'Hour']),
QueryParameter(key='Start Date', type='datetime', value='2021-06-01 00:00:00', enumOptions=None)]
If you'd like to run this query with your own custom parameters, all we'll need to do is take the parameters from from the initial query, change the values to what we want, and re-fetch the query.
Below we are replacing the old NFT contract address param with a new one (the contract address for BAYC)
params = query.parameters
# Replacing with contract address for BAYC
params[0].value = 'xBC4CA0EdA7647A8aB7C2061c2E118A18a936f13D'
custom_query = dune.fetch_query(83579, parameters=params)
Note: You can also create a fresh set of parameters by importing QueryParameter from dunebuggy.models.query and adding the values to the new object.
from dunebuggy.models.query import QueryParameter
param_to_change = QueryParameter(
key='Enter NFT Contract Address',
value='xBC4CA0EdA7647A8aB7C2061c2E118A18a936f13D',
)
params[0] = param_to_change
custom_query = dune.fetch_query(83579, parameters=params)
print(custom_query.info)
{'name': 'Custom NFT Floor Tracker',
'author': '@smaroo',
'length': 265,
'query_id': 83579,
'result_id': UUID('42a3c13d-5fbd-42bd-86c0-acc9adcdc803'),
'job_id': UUID('9051ebe7-862f-46d0-9999-b4645659ca56'),
'columns': ['Time Interval', 'Floor (Approx)']}
Note that the result_id and job_id here are different, this is because we ran the query with our changed params
print(custom_query.parameters)
[QueryParameter(key='Enter NFT Contract Address', type='text', value='xBC4CA0EdA7647A8aB7C2061c2E118A18a936f13D', enumOptions=None),
QueryParameter(key='Floor Time Interval', type='enum', value='Day', enumOptions=['Day', 'Hour']),
QueryParameter(key='Start Date', type='datetime', value='2021-06-01 00:00:00', enumOptions=None)]
print(custom_query.df.head())
| Floor (Approx) | Time Interval | |
|---|---|---|
| 0 | 0.8000 | 2021-06-01T00:00:00+00:00 |
| 1 | 0.8518 | 2021-06-02T00:00:00+00:00 |
| 2 | 0.8260 | 2021-06-03T00:00:00+00:00 |
| 3 | 0.7400 | 2021-06-04T00:00:00+00:00 |
| 4 | 0.8499 | 2021-06-05T00:00:00+00:00 |
Creating a new query
dunebuggy also allows you to create a new using an existing Dune.com account.To login just need to pass in your username/password into the Dune object.
You can verify your login by inspecting your Dune user_id
import os
username = os.environ.get('DUNE_USERNAME')
password = os.environ.get('DUNE_PASSWORD')
dune = Dune(username=username, password=password)
# print(dune.user_id)
To create a query now, all we need to do is pass in a name, query_string and dataset_id
We can construct the SQL query by using a raw sql string
query_string = "select * from ethereum.transactions\nLIMIT 100\n"
Or we could use a fancy ORM-style library like pypika
from pypika import Database, Query
ethereum = Database('ethereum')
q = Query.from_(ethereum.transactions).select('*').limit(100)
query_string = q.get_sql(quote_char=None)
print(query_string)
'SELECT * FROM ethereum.transactions LIMIT 100'
Dune requires us to specify a dataset_id for each of their supported blockchain datasets upon query creation. The currently supported datasets are the following:
| Blockchain Dataset | Id |
|---|---|
| ETHEREUM | 4 |
| XDAI | 6 |
| POLYGON | 7 |
| OPTIMISM_1 | 8 |
| OPTIMISM_2 | 10 |
| BINANCE | 9 |
| SOLANA | 1 |
We can access these integer codes via the DatasetId enum
from dunebuggy.models.constants import DatasetId
created_query = dune.create_query("My Query's Name", query_string, DatasetId.ETHEREUM)
Our created query can be accessed like any other, you can also log into your Dune account as see it there as well!
print(created_query.df.head())
| access_list | block_hash | block_number | block_time | data | from | gas_limit | gas_price | gas_used | hash | index | max_fee_per_gas | max_priority_fee_per_gas | nonce | priority_fee_per_gas | success | to | type | value | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | None | \x887c665b0c52ccace092d817e984e2e828ef59079295... | 47287 | 2015-08-07T08:50:01+00:00 | None | \xdb312d1d6a2ccc64dd94a3892928bac82b4e8c15 | 21000 | 100000000000 | 21000 | \xd3e6a2fc34066d20bb83020b1ee95b9dc7919fd242bd... | 0 | None | None | 0 | None | None | \x34bb6978c5a1ad68777ad388c6787df53903430c | None | 1000000000000000000 |
| 1 | None | \x4869e218b0a8f5784f16193ac66cbf35c4510ace0c9b... | 48698 | 2015-08-07T15:29:53+00:00 | None | \x48040276e9c17ddbe5c8d2976245dcd0235efa43 | 90000 | 57550496008 | 21000 | \x8ba39f908731171fe96ee4e700e71d170ef8e651fac7... | 0 | None | None | 0 | None | None | \xd8d0549637b65d58e7fb6cbdd11530b399d1ddac | None | 100000000000000000000 |
| 2 | None | \xab9491b62b16bd928b281a83db82483584c22aeebc0d... | 49051 | 2015-08-07T17:03:48+00:00 | None | \x8686578c4f7c75246f548299d6ffdac3b67b5cd1 | 90000 | 57178423039 | 21000 | \x57f8ba638903d6335e211eb470159587c73316788880... | 0 | None | None | 0 | None | None | \x87abffa6b80f712c852a9558120ba6611f0b5e46 | None | 45150000000000000000 |
| 3 | None | \x1f9adc2190701ca3085b28252e4f1f467d980f763dad... | 49174 | 2015-08-07T17:41:03+00:00 | None | \x18e4ce47483b53040adbab35172c01ef64506e0c | 90000 | 58589751415 | 21000 | \xb8280da44f8d35011c3f431f7d1a82213477a4e742de... | 2 | None | None | 0 | None | None | \xfb26ae2d3621829472555fbd11bb2a324b7a5c57 | None | 10000000000000000000 |
| 4 | None | \xf1f392fd197a149afe9f8843d7ba759d1a9f79d1ef62... | 49938 | 2015-08-07T21:06:21+00:00 | None | \xc6bf5b6558f2ee21f2e43d9ff9b5408a0cb89413 | 90000 | 71214529679 | 21000 | \x538e1664c12c55287c98dc5dd248f60c642cbbbd7a18... | 0 | None | None | 4 | None | None | \x33a3f479f6c3e7f91128348490d1f7e8d2a0fab5 | None | 5000000000000000000 |
Saving to CSV
To save a query to a CSV, we can take advantage of the to_csv method on our df
created_query.df.to_csv('my_test_data.csv')
Roadmap
- [ ] Cleanup punding TODO comments
- [ ] Add support for embedding Dune graphs/ plotting w/ Dune style colors
- [ ] Add tests (lol)
- [ ] Add support for query updating
- [ ] Investigate whether dashboard support makes sense?
- [ ] Investigate whether there is a max row limit for data returned, if so, query in batches?
- [ ] Better formatting for certain returned columns (links etc..)
- [ ] Add Documentation (Sphinx or something else)
Notes
This project was inspired by the itzemstar's duneanalytics repo
README image is from the IAN Symbols dataset
rysuds