feat: migration("re-indexing"), backfilling and diasgnostics tooling for the ChainIndexer

[aarshkshah1992]

ChainIndexer Migration and Diagnostics Tooling

This PR implements the "migration" (really re-indexing / backfilling), and diagnostics tooling for the ChainIndexer implemented in PR #12450, and is part of the work for #12453. This tooling takes the form of both RPC APIs on the daemon and lotus-shed CLI commands.

Re-indexing Process

The re-indexing tool enables clients to index their entire existing ChainState in the ChainIndexer. This process is necessary due to the removal of the existing MsgIndex, EthTxIndex, and EventIndex from Lotus.

Why Re-index Instead of Migrate?

We've chosen to re-index rather than migrate data from existing indices for two primary reasons:

1. Known issues: The existing indices have multiple known problems, and migrating could perpetuate incorrect index entries.
2. Lack of garbage collection: Existing indices contain many entries for which the corresponding tipset messages/events no longer exist in the ChainStore due to splitstore GC.

Instead, we're re-indexing the Chainstore/Chainstate on the node into the ChainIndexer. This ensures that all re-indexed entries have gone through the indexing logic of the new ChainIndexer and that the Index is in sync/reflects the actual contents of the Chainstore/Chainstate post re-indexing.

Diagnostics Tooling

This PR introduces diagnostic tools for detecting corrupt Index entries at specific epochs or epoch ranges.

While this PR implements functionality for optionally backfilling missing Index entries, it does not yet include the capability to "repair" corrupted Indexed entries. The repair functionality will be introduced in a subsequent PR. This approach allows us to first gather and analyze user reports, helping us understand the types and causes of corrupted Indexed entries(and if all they exist in the new ChainIndexer) before implementing repair mechanisms.

Core API

The fundamental building block for this tooling is the following RPC API:

type IndexValidation struct {
	TipsetKey string
	Height    uint64

	TotalMessages  uint64
	TotalEvents    uint64
	EventsReverted bool

	Backfilled bool
}

func (si *SqliteIndexer) ChainValidateIndex(ctx context.Context, epoch abi.ChainEpoch, backfill bool) (*types.IndexValidation, error)

This API has the following features:

• Optionally backfills the Index with a tipset on the canonical chain for the given epoch if it is absent in the Index
• Returns some aggregated stats for an indexed entry for diagnostics/inspection
• Reports errors/corrupted indexed entries at the given epoch. Forms of Index corruption that can be diagnosed includes:
  • Presence of multiple non-reverted tipsets at the given epoch
  • Complete absence of a non-reverted tipset at the given epoch that does contain reverted tipsets
  • Mismatch between the Chainstore state and the Indexed entries (tipset messages/events)
  • Incorrect Indexing of null rounds at the given epoch

lotus-shed CLI tooling

The lotus-shed CLI tooling for both re-indexing/backfilling and diagnostics can then invoke this RPC API over epoch ranges. The corresponding lotus-shed backfill index [from, to] and lotus-shed inspect index [from, to] can then backfill/inspect/diagnose the Index for the given epoch ranges.

TODO

• automated tests

[github-actions]

Please update the PR title to match https://github.com/filecoin-project/lotus/blob/master/CONTRIBUTING.md#pr-title-conventions

[aarshkshah1992]

Fill out the other fields here using SQL queries.

[aarshkshah1992]

@rvagg Would be great to have a first round of review here when you get the time.

[github-actions]

Please update the PR title to match https://github.com/filecoin-project/lotus/blob/master/CONTRIBUTING.md#pr-title-conventions

[github-actions]

Please update the PR title to match https://github.com/filecoin-project/lotus/blob/master/CONTRIBUTING.md#pr-title-conventions

[aarshkshah1992]

Link will have updated docs for the ChainIndexer RPC once we merge this to master.

[aarshkshah1992]

Okay, starting work on the automated tests as it is the last remaining dev task for this workstream.

[BigLep]

Great docs updates - thanks.

I read the updated docs and left some comments. I also went through everything yesterday and resolved any conversation that I saw had been incorporated. I think anything still open is still relevant.

The big feedback idea/item from this revision is whether we want to write the chainindexer sqllite state to a different directory since that simplifies the migration and rollback story.

I can meet in office on 2024-09-25 Pacific morning if that helps for closing anything out as I don't want to drag this out on you.

[BigLep]

Suggested change

	2. **Backup Existing Index Databases**
	- Before restarting your Lotus node, back up the directory containing your existing index databases (`MsgIndex`, `EthTxIndex`, and `EventIndex`).
	- These databases are located in the `{$LOTUS_PATH/sqlite}` directory.
	- Use the following command to copy the entire directory:
	```bash
	cp -r $LOTUS_PATH/sqlite {destination_path}
	```
	- **Note: If you have configured a custom directory path for the Index databases using the `Events.DatabasePath` config option, replace `{$LOTUS_PATH/sqlite}` with your custom path.**
	- These backups are essential for potential rollbacks, even though they are not used in the migration process.
	3. **Remove Old Index Files**
	- After creating backups, remove the `{$LOTUS_PATH/sqlite}` directory (*or your custom index database path*) using the following command:
	```bash
	rm -rf $LOTUS_PATH/sqlite
	```
	- **Warning: Please ensure and validate that you have made backups of your existing index databases before removing the directory.**
	2. **Backup and Remove Existing Index Databases**
	- The backup is accomplished by moving the legacy `{$LOTUS_PATH/sqlite}` directory so the existing index databases (`MsgIndex`, `EthTxIndex`, and `EventIndex`) don't get overwritten.
	- Use the following command to mv the entire directory:
	```bash
	mv $LOTUS_PATH/sqlite $LOTUS_PATH/sqlite-pre-chainindexer
	```
	- **Note: If you had configured a custom directory path for the Index databases using the `Events.DatabasePath` config option, replace `{$LOTUS_PATH/sqlite}` with your custom path.**
	- These old indexes are essential for potential rollbacks, even though they are not used in the migration process.

I was realizing we can hit two birds with one stone here: moving accomplishes the backup and the "removal".

That said, we could simplify this further if chaindindexer didn't use $LOTUS_PATH/sqlite. What if instead we wrote to $LOTUS_PATH/chainindex ? (I don't have access to a full lotus node to come up with other ideas. ). I assume we don't want to write under chain even though these indices are related to the chain. ChatGPT tells me there is a indices directory on full nodes, but I don't know what that holds

[aarshkshah1992]

This is done.

[BigLep]

Suggested change

	2. Remove the current `${LOTUS_PATH}/sqlite` directory and replace it with the backup taken in the "**Backup Existing Index Databases**" section of the [Migration Guide](#migration-guide).
	2. Replace the current `${LOTUS_PATH}/sqlite` with the backup taken in the "**Backup Existing Index Databases**" section of the [Migration Guide](#migration-guide).
	```bash
	mv $LOTUS_PATH/sqlite-pre-chainindexer $LOTUS_PATH/sqlite
	```

[BigLep]

Related to previous discussion, this step goes away if we just write chainindexer sql state to somewhere else.

[aarshkshah1992]

Would love to know what @rvagg thinks about this. Writing the chainindexer sql state to another directory makes sense to me as it does simplify migration ops.

[BigLep]

@aarshkshah1992 : @rvagg and I spoke verbally on 2024-09-25 (my time) and he was supportive of this idea of writing chainindexer sqllite in a different directory.

[aarshkshah1992]

@BigLep This was a great idea and is now done. These steps look much cleaner now. Please take a look.

[BigLep]

Inline this into your documentation above about "error'?

[BigLep]

Since this is docs about IndexValidation, should these comments be moved to where IndexValidation is defined?

[aarshkshah1992]

@BigLep Have addressed your second round of review on the RPC user doc.

[BigLep]

@aarshkshah1992 : @rvagg and I spoke verbally on 2024-09-25 (my time) and he was supportive of this idea of writing chainindexer sqllite in a different directory.

[BigLep]

From a docs regard, things look good to me. I left a couple of small suggestions.