Implementing the Audio Playback/Download on the chat

[panoramix360]

Summary

This is a work-in-progress PR that implements using the react-native-video library, the playback of audio on the mobile application. It uses the audioOnly prop to load audio using the library.

Features implemented

• Play/Pause of the media
• Progress with Cursor
• Update the progress/time when the audio is playing

Work to be done

• Download the audio

Ticket Link

mattermost/mattermost#26290

Checklist

• Added or updated unit tests (required for all new features)
• Has UI changes
• Includes text changes and localization file updates
• Have tested against the 5 core themes to ensure consistency between them.

Device Information

This PR was tested on:

• For now, just Android will test on iOS shortly

Screenshots

Android

iOS

Release Note

Added the possibility to play and download audios on the chat

[mattermost-build]

Hello @panoramix360,

Thanks for your pull request! A Core Committer will review your pull request soon. For code contributions, you can learn more about the review process here.

[mattermost-build]

This PR has been automatically labelled "stale" because it hasn't had recent activity.
A core team member will check in on the status of the PR to help with questions.
Thank you for your contribution!

[panoramix360]

@larkox

Added the possibility for download/preview of the audio file.

But I'm not so sure if this is the right code for it, I don't know if it's already implemented, I did the same way that was implemented on the Documents File.

So if any reviewer can help me clean this code or make it better, I'm available to fix it.

I'll test on iOS and open the PR for review.

Thanks!

[larkox]

@panoramix360 I gave it a quick glance, and I don't see any big problems in reusing the code on the document file. In order to have things cleaner and get better reusability, I recommend you to create a custom hook.

Something like useFileDownload (or a better name, I am horrible at naming things XD) where you keep and manage all the different states and callbacks needed. That way you can use it in both components.

How does that sound? Is there any other question?

[panoramix360]

@larkox makes sense!

I'll create it :D

[panoramix360]

@larkox,

Hook created and migrated in the document_file and audio_file!

Thanks for the tip! Can you take a look now?

I'm having some trouble running on iOS, I don't know what it could be, I managed to do the npm install without problems and pod install as well.

note: Removed stale file '/Users/LucasReis/Library/Developer/Xcode/DerivedData/Mattermost-gbiqdcuxzxywxvfzujpwidajdzki/Build/Intermediates.noindex/Pods.build/Debug-iphonesimulator/JitsiWebRTC.build/Script-46EB2E0002EF80.sh'


--- xcodebuild: WARNING: Using the first of multiple matching destinations:
{ platform:iOS Simulator, id:1EA71F74-860F-4DC5-B8D6-3143E22A2366, OS:17.4, name:iPhone SE (3rd generation) }
{ platform:iOS Simulator, id:1EA71F74-860F-4DC5-B8D6-3143E22A2366, OS:17.4, name:iPhone SE (3rd generation) }
** BUILD FAILED **


The following build commands failed:
	CompileC /Users/LucasReis/Library/Developer/Xcode/DerivedData/Mattermost-gbiqdcuxzxywxvfzujpwidajdzki/Build/Intermediates.noindex/Pods.build/Debug-iphonesimulator/FlipperKit.build/Objects-normal/arm64/FlipperPlatformWebSocket.o /Users/LucasReis/projects/mattermost-mobile/ios/Pods/FlipperKit/iOS/FlipperKit/FlipperPlatformWebSocket.mm normal arm64 objective-c++ com.apple.compilers.llvm.clang.1_0.compiler (in target 'FlipperKit' from project 'Pods')
(1 failure)

Any ideas? Maybe updating my branch with main?

[larkox]

Great work! Some changes around the hook.

[larkox]

Any ideas? Maybe updating my branch with main?

Yes. Update your branch with main, run npm clean, and npm i again. That should do it.

[panoramix360]

hey @larkox just updated with the main branch and now it builds the application but after trying to run it says:

3CA4978B6CC")
success Successfully built the app
2024-05-21 22:48:16.304 xcodebuild[32994:8107447]  DVTPlugInQuery: Requested but did not find extension point with identifier 'Xcode.InterfaceBuilderBuildSupport.PlatformDefinition'. This is programmer error; code should only request extension points that are defined by itself or its dependencies.
error Failed to get the target build directory.

Weird, I'm trying to investigate it, thanks!

EDIT: I was able to run it using Xcode directly, I don't know what may be wrong with the npm run ios but ok, hehe

[panoramix360]

I found a bug on the iOS version, try to fix it now.

The player is not behaving like in Android, strange, after playing the audio it doesn't pause it anymore.

[panoramix360]

Fixed, I need to fix it in a not-so-great way.

Let me know what you think @larkox, it seems to be a known issue that was solved in new versions:
TheWidlarzGroup/react-native-video#3644

[mattermost-build]

This PR has been automatically labelled "stale" because it hasn't had recent activity.
A core team member will check in on the status of the PR to help with questions.
Thank you for your contribution!

[larkox]

@panoramix360 There is a thing failing in the tests, related to the en.json file missing the new line at the end of the file. Can you review it?

Once we have the CI all green, we can create a build and send to UX to do the first review on the "front-end" side.

[panoramix360]

@larkox done!

Thank you!

[larkox]

@abhijit-singh Here should be the builds for this:
https://community.mattermost.com/core/pl/4txx9r7j13djj8a6h3yzpe7wzy
https://community.mattermost.com/core/pl/m3mjca7enfnddx65egq7xxw7he

[abhijit-singh]

Thanks @panoramix360 and @larkox. I still see a few issues, more prominently on Android. Attaching recordings of the test on both Android and iOS. The first file in the recording is recorded and sent from iOS (M4A format) whereas the file sent second is recorded and sent from Android (AAC format).

Also thanks for taking the time to add in the scroll functionality! It seems a bit laggy right now. Is there any way to optimize that? Might be okay doing that as a follow up even.

Android recording:
https://github.com/user-attachments/assets/b8c4bc97-cc20-43c1-af32-9a489641f4bb

List of issues observed on Android:

1. The second file in the attached recording never shows the duration. It plays normally and the duration increments as expected when played, but it does not show the duration upfront. I can see the duration in iOS though.
2. If you tap on the message containing the audio to go to the thread view, the duration flickers to a long string for a second.
3. Also in the thread view, neither files show the actual duration in the default view and instead show 00:00. It should be visible ideally even before the user clicks on play.
4. On Android, I was not able to use the scroll functionality for the AAC file which was recorded and shared from that same device. I was able to scroll on both files on iOS though.

iOS recording:
https://github.com/user-attachments/assets/5cd36611-5ee9-4de7-86c4-5b2352d56705

1. While most things are working as expected, it takes significantly longer to load the duration here. Especially when you go into the thread view, it takes a few seconds before the duration of the file loads.
2. The scroll dot can seemingly be dragged off the screen which also makes the duration value negative. Ideally dragging beyond the scroll line or the file container should not be allowed. If we can't prevent that, we should try to bring it back to the start immediately even if someone drags it away. This seems to be happening on Android as well.

[panoramix360]

Thanks for the feedback @abhijit-singh!

I'll take a look at those and fix it accordingly, I think overall it will be a polished and nice experience if we manage to solve all of this.

Some of your issues are probably something related to the library we are using to load the audios, but I'll take a look at open issues there, like the duration issue on Android. The experience on iOS seems to be better, no idea why, will investigate it as well.

Thank you!

[panoramix360]

@abhijit-singh

Fixed the load duration issues, in fact it was a bug on the code itself. I hope that the experience is better now!

I clamped the cursor draggin as well to only be able to be dragged inside the progress itself, so this is a more nicer experience for the user, sorry about that.

Now I'm working to make the dragging of the cursor with less stuttering, I noticed that the experience is not so good in a real device. I think this is the only thing missing!

Thanks for your feedback, if you want you can already test the other issues.

Can you provide more details on this one?

On Android, I was not able to use the scroll functionality for the AAC file which was recorded and shared from that same device. I was able to scroll on both files on iOS though.

I was not able to replicate it.