-
Notifications
You must be signed in to change notification settings - Fork 1.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: a new ChainIndexer to index tipsets, messages and events #12421
base: master
Are you sure you want to change the base?
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please update the PR title to match https://github.com/filecoin-project/lotus/blob/master/CONTRIBUTING.md#pr-title-conventions
node/config/def.go
Outdated
EthTxHashMappingLifetimeDays: 0, | ||
EthTraceFilterMaxResults: 500, | ||
EthBlkCacheSize: 500, | ||
EnableEthRPC: true, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These changes will be reverted as the final outcome of that discussion.
gcRetentionDays := si.gcRetentionEpochs / (builtin.EpochsInDay) | ||
if gcRetentionDays < 1 { | ||
log.Infof("skipping gc of eth hashes as retention days is less than 1") | ||
return | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
oh, so if you set it anything less than 2880 then you're going to have hashes building up? maybe setting it to less than 2880 should be an error on startup if this is necessary, alternatively convert this to minutes and delete more fine-grained?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If we're deferring support then then I think it should be an error to set it to anything 0>x<2880
, this is going to be a surprise for anyone not info logging and watching their logs
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@rvagg This is done. Have added better docs for this param and we now throw an error if it's less than 2880 with an informative error message.
Co-authored-by: Rod Vagg <[email protected]>
…s enabled (#12463) * auto repair events * make jen * fix leaky abstraction
For #12453.
Subsumes #12388 to implement the
ChainIndexer
.TODO
ChainIndexer
(tests are at feat: tests for the chain indexer #12521)Migration
Index "migration" will effectively involve re-indexing the chain state here. This is because the data in the old Indices isn't reliable (we're already aware of multiple bugs in the old Indices which motivated this PR in the first place) and the old Indices have never been gc'd/kept in sync with the chain state on nodes with splitstore enabled.
For Snapshot synced nodes with Splitstore enabled
ReConcileEmptyIndex
to true and setting theMaxReconcileTipsets
as needed.For archival nodes
If archival nodes want to backfill the index as part of daemon startup, they can always bootup the node with
ReConcileEmptyIndex
set to true andMaxReconcileTipsets
set to infinity. This will drastically slow down the node startup but will ensure that the node starts up with it's entire history indexed.The other option for users is to use the
Backfill
RPC API with a range of height they want backfilled. However, we need to handle race conditions here where backfilling races with new tipsets coming in and need to ensure that backfilling does not end up overwriting updates caused by new tipsets being indexed.Sqllite Query Plans
For looking up
MsgInfo
reverted != 0
and only one withreverted=0
for a givenmessage_cid
so no need to index byreverted
as well here.For looking up an eth tx hash
For checking if a tipset exists
For getting the minimum no-reverted height for reconciliation
reverted != 0
and only one entry withreverted=0
, this should be allright.Looking up a
message_id
to insert an eventLooking up events at a specific height
Looking up events in a height range
Looking up events for a specific tipset