Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rkyv 0.8 migration #575

Merged
merged 91 commits into from
Sep 23, 2024
Merged

Rkyv 0.8 migration #575

merged 91 commits into from
Sep 23, 2024

Commits on Sep 18, 2024

  1. update to rkyv 0.8.7

    bunnie committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    56e5d53 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    0be556a View commit details
    Browse the repository at this point in the history
  3. dependency updates

    bunnie committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    11e112d View commit details
    Browse the repository at this point in the history
  4. mods to lib

    bunnie committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    e004ce2 View commit details
    Browse the repository at this point in the history
  5. a pass at modifying string.

    Turns out, this is the wrong approach. It's still useful code
    because it's a direct method for passing strings that doesn't
    rely on rkyv (so it could potentially be more efficient for
    very efficiency-oriented applications).
    
    But I think in the end, after writing this and seeing the errors
    that come from it, I think we should just abandon String<N> altogether
    because it doesn't get us anything.
    bunnie committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    3dd4e2a View commit details
    Browse the repository at this point in the history

Commits on Sep 22, 2024

  1. bump rkyv in planning

    bunnie committed Sep 22, 2024
    Configuration menu
    Copy the full SHA
    a506e7c View commit details
    Browse the repository at this point in the history
  2. remove xous-ipc string

    this triggers a whole lotta changes....
    bunnie committed Sep 22, 2024
    Configuration menu
    Copy the full SHA
    6d954d6 View commit details
    Browse the repository at this point in the history
  3. clean up a warning in loader crate

    minor cleanup because the warning is distracting from chasing
    down all the other warnings
    bunnie committed Sep 22, 2024
    Configuration menu
    Copy the full SHA
    423bfe9 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    6470e1c View commit details
    Browse the repository at this point in the history
  5. regex search/replace instances of xous_ipc::String to String

    split into a couple commits for ease of committing
    bunnie committed Sep 22, 2024
    Configuration menu
    Copy the full SHA
    a1bf70b View commit details
    Browse the repository at this point in the history
  6. regex search/replace instances of xous_ipc::String to String

    split into multiple commits for easier committing
    bunnie committed Sep 22, 2024
    Configuration menu
    Copy the full SHA
    6d7a8fd View commit details
    Browse the repository at this point in the history
  7. regex search/replace instances of xous_ipc::String to String

    split into multiple commits for easier committing
    bunnie committed Sep 22, 2024
    Configuration menu
    Copy the full SHA
    52ffd88 View commit details
    Browse the repository at this point in the history
  8. regex search/replace instances of xous_ipc::String to String

    split into multiple commits for easier committing
    bunnie committed Sep 22, 2024
    Configuration menu
    Copy the full SHA
    916c98f View commit details
    Browse the repository at this point in the history
  9. regex search/replace instances of xous_ipc::String to String

    core xous services cluster
    bunnie committed Sep 22, 2024
    Configuration menu
    Copy the full SHA
    1d9316f View commit details
    Browse the repository at this point in the history
  10. regex search/replace instances of xous_ipc::String to String

    more regex results
    bunnie committed Sep 22, 2024
    Configuration menu
    Copy the full SHA
    f2fbc7f View commit details
    Browse the repository at this point in the history
  11. regex search/replace instances of xous_ipc::String to String

    codegenerator fixes
    bunnie committed Sep 22, 2024
    Configuration menu
    Copy the full SHA
    849ae5f View commit details
    Browse the repository at this point in the history
  12. regex search/replace instances of xous_ipc::String to String

    includes some light touch-up to the API to clean up a consequence
    of String not being copy-able anymore
    bunnie committed Sep 22, 2024
    Configuration menu
    Copy the full SHA
    3eb61d3 View commit details
    Browse the repository at this point in the history
  13. Configuration menu
    Copy the full SHA
    c4f67cc View commit details
    Browse the repository at this point in the history
  14. Configuration menu
    Copy the full SHA
    0ef93e3 View commit details
    Browse the repository at this point in the history
  15. Configuration menu
    Copy the full SHA
    15722a3 View commit details
    Browse the repository at this point in the history
  16. regex xous_ipc::String to String + cleanup idioms

    some idioms are improved with String, this cleans up a bit of code
    in addition to the search/replace
    bunnie committed Sep 22, 2024
    Configuration menu
    Copy the full SHA
    d2f13d9 View commit details
    Browse the repository at this point in the history
  17. Configuration menu
    Copy the full SHA
    1b0fa6e View commit details
    Browse the repository at this point in the history
  18. cleanup a typecast ambiguity due to new API

    Rust can't tell how to cast some u32_le into u32/usize automatically
    due to confusion with the num_derive trait, so we have to explicitly
    specify a chain of typecasts to clean up the conflict with the
    new rkyv
    bunnie committed Sep 22, 2024
    Configuration menu
    Copy the full SHA
    0fbca1c View commit details
    Browse the repository at this point in the history
  19. Configuration menu
    Copy the full SHA
    b514d31 View commit details
    Browse the repository at this point in the history
  20. Configuration menu
    Copy the full SHA
    1fd59f0 View commit details
    Browse the repository at this point in the history
  21. security-relevant: clear_volatile() API missing on String

    xous_ipc::<String> had a clear-volatile() API for zeroizing
    sensitive strings.
    
    This is now missing in String, so it has been manually implemented
    here by casting the String to a slice-u8 and then iterating
    through it using raw pointers and volatile writes with a compiler
    barrier to hopefully avoid all of the landmines I know of for
    zeroizing a string...
    
    commit is called out separately because this one is one of the
    more "dangerous" security-related changes in the API migration.
    bunnie committed Sep 22, 2024
    Configuration menu
    Copy the full SHA
    f1d2633 View commit details
    Browse the repository at this point in the history
  22. Configuration menu
    Copy the full SHA
    f8719d7 View commit details
    Browse the repository at this point in the history
  23. manual cleanup Copy derive: gam crate

    There isn't a simple regex to deal with the loss of Copy when
    deprecating xous_ipc, so this has to be a manual pass.
    
    Losing the copy trait means (maybe) some loss of efficiency
    in the API because we have to clone some structures instead
    of copying them, although, I think actually what might have
    been happening under the hood anyways is the copy was implemented
    as a clone-under-the-hood for the xous-ipc.
    
    Either way, this happens in the high level management structures
    for modals, so it's not the end of the world to incur a clone
    on the description of the structures -- we'd have to pay
    more attention if this happened e.g. at the pixel shuffling
    level, but a few UI strings at human-interaction rates
    isn't going to kill us.
    bunnie committed Sep 22, 2024
    Configuration menu
    Copy the full SHA
    58527ea View commit details
    Browse the repository at this point in the history
  24. review-needed: modify the alert modal prompt loop

    The original loop takes advantage of the copy-trait that is
    no longer available. I think the intent of the loop was to
    mutate a field within the copy through each iteration so you
    can update the prompt to handle field validations that failed
    without re-allocating the ManagedPromtWithTextResponse structure.
    
    This has been rewritten to alloc that each iteration to avoid
    the borrow error, and instead the `prompt` itself is a string
    that's initialized with the initial prompt message and is updated
    on each iteration if the form validation fails.
    
    @gsora git seems to say you wrote this code, maybe I could use
    your help to review this patch.
    bunnie committed Sep 22, 2024
    Configuration menu
    Copy the full SHA
    dfd7272 View commit details
    Browse the repository at this point in the history
  25. more cleanup of copy trait

    bunnie committed Sep 22, 2024
    Configuration menu
    Copy the full SHA
    af35f6e View commit details
    Browse the repository at this point in the history
  26. bunnie-review: fix borrows in modals main loop

    note for more review by bunnie down the road to make sure
    that I have not changed any behaviors accidentally in fixing
    the loss of Copy on a core structure
    bunnie committed Sep 22, 2024
    Configuration menu
    Copy the full SHA
    f6f7cbd View commit details
    Browse the repository at this point in the history
  27. format cleanup & copy remove

    a bunch of code has to be re-formatted because spacing changed
    and we have a strict formatting rule
    bunnie committed Sep 22, 2024
    Configuration menu
    Copy the full SHA
    0cdbfdf View commit details
    Browse the repository at this point in the history
  28. formatting changes that resulted from the regex search

    a bunch of lines are shorter or different length now which
    triggers a mandatory reformat. this patch cleans that up,
    so that going forward the manual rework patches are clearer and
    not tied up with formatting patches.
    bunnie committed Sep 22, 2024
    Configuration menu
    Copy the full SHA
    34e6f46 View commit details
    Browse the repository at this point in the history
  29. removed sized strings from pddb interchange structures

    I think this is safe to do because no strings can be directly
    serialized into the PDDB's structures. The place to check for
    this is inside the backend/ directory. For example, `DictName`
    is where the name field is serialized into a structure that
    goes onto the disk. It's explicitly length-checked.
    
    What this means is that we could have calls into the pddb crate
    with invalid length strings, however, if you check the lib
    implementation, all of the APIs also pre-flight the length of
    arguments to ensure they fit into the destination string.
    
    So mostly what we're losing is the constraints on the intermediate
    IPC representation, but the length is still checked on either side.
    bunnie committed Sep 22, 2024
    Configuration menu
    Copy the full SHA
    be227ad View commit details
    Browse the repository at this point in the history
  30. Configuration menu
    Copy the full SHA
    1b5eacd View commit details
    Browse the repository at this point in the history
  31. more cleanup on usb crate

    bunnie committed Sep 22, 2024
    Configuration menu
    Copy the full SHA
    6e4eec9 View commit details
    Browse the repository at this point in the history
  32. ah yes. This was the kind of xous-ipc abuse I was worried I would hav…

    …e done.
    
    And I did it! and it's now fixed.
    
    I was using xous-ipc String to make a fixed-length string in a
    structure that's not strictly an IPC structure.
    
    The solution to this is to just implement a fixed length string
    directly.
    bunnie committed Sep 22, 2024
    Configuration menu
    Copy the full SHA
    7141a02 View commit details
    Browse the repository at this point in the history
  33. Configuration menu
    Copy the full SHA
    8399239 View commit details
    Browse the repository at this point in the history
  34. more typecast cleanup

    bunnie committed Sep 22, 2024
    Configuration menu
    Copy the full SHA
    4eb504b View commit details
    Browse the repository at this point in the history
  35. Configuration menu
    Copy the full SHA
    15aaf55 View commit details
    Browse the repository at this point in the history
  36. Configuration menu
    Copy the full SHA
    e423bfd View commit details
    Browse the repository at this point in the history
  37. Configuration menu
    Copy the full SHA
    95a31a3 View commit details
    Browse the repository at this point in the history
  38. cleanup replace, and make into_inner/from_inner methods

    the into/from inner methods are useful for using the Buffer
    abstraction as a way to invoke rkyv on a structure to serialize
    it and then manipulate the serialized data directly
    bunnie committed Sep 22, 2024
    Configuration menu
    Copy the full SHA
    1bd8be2 View commit details
    Browse the repository at this point in the history
  39. cleanup borrow issues in com

    bunnie committed Sep 22, 2024
    Configuration menu
    Copy the full SHA
    ef71885 View commit details
    Browse the repository at this point in the history
  40. fixup API to not rely on copy, instead use borrow

    also some more regex fixup
    bunnie committed Sep 22, 2024
    Configuration menu
    Copy the full SHA
    8424d19 View commit details
    Browse the repository at this point in the history
  41. final cleanup on shellchat xous-ipc abstractions

    the implementation relied on a quirk of xous-ipc where the message
    ID is 0 for string send/receives. This cleans it up so that the API
    is transparent and consistent with other API calls in the system.
    bunnie committed Sep 22, 2024
    Configuration menu
    Copy the full SHA
    3c0140b View commit details
    Browse the repository at this point in the history
  42. more xous_ipc cleanup

    bunnie committed Sep 22, 2024
    Configuration menu
    Copy the full SHA
    5cad1a3 View commit details
    Browse the repository at this point in the history
  43. change xous-ipc api to 0.10.0

    bunnie committed Sep 22, 2024
    Configuration menu
    Copy the full SHA
    4c7c931 View commit details
    Browse the repository at this point in the history
  44. clarify some variable names

    bunnie committed Sep 22, 2024
    Configuration menu
    Copy the full SHA
    ada196d View commit details
    Browse the repository at this point in the history
  45. Configuration menu
    Copy the full SHA
    840964d View commit details
    Browse the repository at this point in the history
  46. Configuration menu
    Copy the full SHA
    d78cbff View commit details
    Browse the repository at this point in the history
  47. Configuration menu
    Copy the full SHA
    65e3a03 View commit details
    Browse the repository at this point in the history
  48. Configuration menu
    Copy the full SHA
    89bc7c1 View commit details
    Browse the repository at this point in the history
  49. attempt to update rkyv use in TLS library

    @nworbnhoj would love it if you could have a look at
    the implementation and let me know if I've ported this correctly.
    
    I think what was going on is you were using rkyv to serialize
    a structure to the PDDB. I've just updated the ser/de to use
    the latest `alloc` API (which is much simpler than the API used
    in xous-ipc because it is no-alloc).
    
    I'll try and run some tests that involve the TLS crate too down
    the road under the theory that it would suss out any porting
    errors.
    bunnie committed Sep 22, 2024
    1 Configuration menu
    Copy the full SHA
    44b9041 View commit details
    Browse the repository at this point in the history
  50. Configuration menu
    Copy the full SHA
    5973e48 View commit details
    Browse the repository at this point in the history
  51. Configuration menu
    Copy the full SHA
    2f86042 View commit details
    Browse the repository at this point in the history
  52. Configuration menu
    Copy the full SHA
    a3933a7 View commit details
    Browse the repository at this point in the history
  53. Configuration menu
    Copy the full SHA
    888cd22 View commit details
    Browse the repository at this point in the history
  54. Configuration menu
    Copy the full SHA
    7d69e16 View commit details
    Browse the repository at this point in the history
  55. update formatting on the published crates

    not doing a reformat of the entire project to match
    nightly formatting changes, but capturing the published
    crates so we can maybe save a version bump later on to get
    the project clean.
    bunnie committed Sep 22, 2024
    Configuration menu
    Copy the full SHA
    bb72095 View commit details
    Browse the repository at this point in the history
  56. commit the version bump

    bunnie committed Sep 22, 2024
    Configuration menu
    Copy the full SHA
    75954de View commit details
    Browse the repository at this point in the history
  57. update verifier pins

    bunnie committed Sep 22, 2024
    Configuration menu
    Copy the full SHA
    e01326c View commit details
    Browse the repository at this point in the history
  58. lose the local patches

    bunnie committed Sep 22, 2024
    Configuration menu
    Copy the full SHA
    d39ca54 View commit details
    Browse the repository at this point in the history
  59. commit bumped version pins

    bunnie committed Sep 22, 2024
    Configuration menu
    Copy the full SHA
    838bd58 View commit details
    Browse the repository at this point in the history

Commits on Sep 23, 2024

  1. Configuration menu
    Copy the full SHA
    cbe9aa0 View commit details
    Browse the repository at this point in the history
  2. port getrandom to use flatipc

    bunnie committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    489c614 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    e9ab865 View commit details
    Browse the repository at this point in the history
  4. update test on cram side too

    bunnie committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    9ebb61c View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    ec43c46 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    cdc8411 View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    e5bd1cf View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    f4a4dfb View commit details
    Browse the repository at this point in the history
  9. Configuration menu
    Copy the full SHA
    d581bd4 View commit details
    Browse the repository at this point in the history
  10. cleanup warning

    bunnie committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    4239176 View commit details
    Browse the repository at this point in the history
  11. Configuration menu
    Copy the full SHA
    553f98f View commit details
    Browse the repository at this point in the history
  12. Configuration menu
    Copy the full SHA
    7ea1400 View commit details
    Browse the repository at this point in the history
  13. Configuration menu
    Copy the full SHA
    1ec6613 View commit details
    Browse the repository at this point in the history
  14. cleanup chat-test

    bunnie committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    3ce6baa View commit details
    Browse the repository at this point in the history
  15. Configuration menu
    Copy the full SHA
    c699e06 View commit details
    Browse the repository at this point in the history
  16. mtxchat port to xous-ipc 0.10

    bunnie committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    048e8d8 View commit details
    Browse the repository at this point in the history
  17. length of a record stored is the pos field for rkyv-to-pddb

    prior use of `rkyv` reserves two bytes to record `pos` pre-pended
    to the rkyv record
    
    the new rkyv works without having to do that because it indexes
    the rkyv data off the end of the stored records. Thus, all you
    need to know is the total length of the byte slice that stores
    the serialized data. This is available as the return argument
    when you read a PDDB key, so we don't have to explicitly store it
    anymore.
    bunnie committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    3a9a7eb View commit details
    Browse the repository at this point in the history
  18. Configuration menu
    Copy the full SHA
    e61a2e3 View commit details
    Browse the repository at this point in the history
  19. add bounds checking to a TextView string length

    Before xous-ipc's fixed-length string guaranteed this thing
    fit into a single page. Now, we can send potentially unbounded
    amounts of text between servers.
    
    I don't think it makes sense to send more than a few k of text to
    render on the screen at once, given our screen sizes, so I've added
    a patch that truncates the text to the previous length that things
    would truncate to.
    
    The behavior is the same as before, except that you will
    over-allocate memory on the heap in the sending process
    as you shove more string data into a TextView than can be rendered.
    Previously, the TextView would just truncate the data and discard
    the data.
    
    The truncation happens at the point where you try to hand the
    record off to another process, so, one could in theory try
    to make this recoverable by adding a layer that splits the
    textview into more manageable chunks or pages the data, but that
    is an exercise for another day.
    bunnie committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    921be1c View commit details
    Browse the repository at this point in the history
  20. cleanup bulk read on PDDB

    we don't have to pass a "size" field anymore; since we're end-offset
    from pos, just pass pos and use that to chain records.
    bunnie committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    8cd1cef View commit details
    Browse the repository at this point in the history
  21. fail, don't panic on serialization failure

    this allows us to handle the error. probably should propagate
    on to xous-ipc
    bunnie committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    2e9b19d View commit details
    Browse the repository at this point in the history
  22. add release notes

    bunnie committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    047ffe3 View commit details
    Browse the repository at this point in the history
  23. change rkyv behavior in xous-ipc from panic to failure

    this allows for more graceful error handling in case of
    serialization failures
    bunnie committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    0ecfa80 View commit details
    Browse the repository at this point in the history
  24. absorb new xous-ipc version

    bunnie committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    f494da2 View commit details
    Browse the repository at this point in the history
  25. update xous-ipc pins

    bunnie committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    9c2ee16 View commit details
    Browse the repository at this point in the history
  26. Configuration menu
    Copy the full SHA
    67e0004 View commit details
    Browse the repository at this point in the history
  27. update to use new flat-ipc API

    bunnie committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    e68fde8 View commit details
    Browse the repository at this point in the history