sync GVRaudio to video - Google Cardboard

Hello,
I'm using the Samsung Gear VR Framework to develop a VR 360 video playback app. The video's I will use all have surround sound that need to be lipsync to the video.
It would be best to add to audio to the mp4 video file, but how would you then get that audio so you can rotate it using headtracking? Android mediaplayer will play the audio, but it will not rotate the field when the user rotates his head. The Samsung GVR framework relies on GVRAudioEngine (cardboard / google vr api) classes to playback audio with use of the headtracking (binaural). However, that only works for separate audio files, not linked to video. I'm in the dark on how to sync such audio to a video. The API doens't expose any functions to get/set playback position to allow me to make my own sync engine (I'm even not sure that this would work)
Any pointers are welcome!
Thanks,
Vincent

Related

mp4 video play back on PC

Hi Folks,
I just took for the first time a video clip with my Magician today, and had the Camera settings to
Capture Mode: Video
Capture Format: MPEG-4
Capture Size: Large
it produced a nice video clip file with in the .mp4 format (lets not talk about the quality)
I copied this file over to my SD card, and from there to my PC to play here.
It played by default using Apple Quick Time (7.0.3), no probs.
I tried to open it with WMP10 on my PC, and only got the sound, not video.
I was assuming, that a video clip taken with a Microsoft product (Magician WM2003) would play by default using the Microsoft pendant on the PC (WMP10), and MS does not rely on a 3rd party software (QT).
Any ideas what I'm doing wrong, or if I'm just missing the right codec on my PC to play back the file ?
Thanks
torklemm said:
Hi Folks,
I just took for the first time a video clip with my Magician today, and had the Camera settings to
Capture Mode: Video
Capture Format: MPEG-4
Capture Size: Large
it produced a nice video clip file with in the .mp4 format (lets not talk about the quality)
I copied this file over to my SD card, and from there to my PC to play here.
It played by default using Apple Quick Time (7.0.3), no probs.
I tried to open it with WMP10 on my PC, and only got the sound, not video.
I was assuming, that a video clip taken with a Microsoft product (Magician WM2003) would play by default using the Microsoft pendant on the PC (WMP10), and MS does not rely on a 3rd party software (QT).
Any ideas what I'm doing wrong, or if I'm just missing the right codec on my PC to play back the file ?
Thanks
Click to expand...
Click to collapse
Correct me if i'm wrong but i do believe the camera program on modern HTC devices (like the Magician) is made by HTC itself and not by MS. So it's possible it uses different kind of codecs.
MS isn't allowed to ship a lot of codec with WMP. Only some basic and MS-made codecs. So you have to rely on third party apps/codecs to play the .mp4 files.
Hello Torklemm
You can use the VLC Player to Play almost any type of media including the mp4 , 3gp etc etc .
here is the link to download it - h**p://www.download.com/VLC-Media-Player/3000-2194_4-10267151.html
You can also install the Nero Digital MPEG 4 AVC h.264 dshow decoder , after i installed this , my wmp ver 10 was able to play the Mp4 video files.
Cheers.
Sean_Dillon said:
Hello Torklemm
You can use the VLC Player to Play almost any type of media including the mp4 , 3gp etc etc .
here is the link to download it - h**p://www.download.com/VLC-Media-Player/3000-2194_4-10267151.html
You can also install the Nero Digital MPEG 4 AVC h.264 dshow decoder , after i installed this , my wmp ver 10 was able to play the Mp4 video files.
Cheers.
Click to expand...
Click to collapse
very useful decoder!
but only motion decoding not audio track.
i see only motion without sound.
Hmmm , That is Strange Vigor_ , I thought the audio recorded in the video format of Magician was either mp3 , wma or wav . . . all 3 being suported by wmp. Will have to check this Thanx.
It's none of those. The audio is AMR Narrowband, whatever the blazes that is! I've been having the same problem converting it. AMR doesn't seem to be supported by much.
mp4 format
You should use a plugin or converter program. For plugin Ulead video suite v9.0 is the best solution.For convert purpose I offer IMTOO program.Another way is using avi format instead of mp4
Where do you get that Nero Digital dshow decoder? A Link would be nice.
Re: mp4 format
bucat said:
You should use a plugin or converter program. For plugin Ulead video suite v9.0 is the best solution.For convert purpose I offer IMTOO program.Another way is using avi format instead of mp4
Click to expand...
Click to collapse
The same with me. I use ImTOO 3GP Video Converter 2.1.55.1107b, good stuff.
Where do you get that Nero Digital dshow decoder?
The File Is Attached To The Post , Please Log-in To View and Download it.
How to do it for free, though it's a ton of work
Okay, I've successfully done it now. My biggest problem was that imTOO 3gpp costs money, and I just couldn't justify spending the money on it.
There's probably a few different ways to do this, but here's how I've done it:
1) Separate the Video stream from the audio stream
I used Microsoft Graphedit for this. It comes in a Microsoft SDK, but if you google it, you can find it.
When you open graphedit and select render the file, you'll get a message saying that part of the stream isn't recognised. That's the AMR audio. That's step 2, so ignore. Assuming you've got an xvid mpeg 4 codec on your machine already (and I suspect also a 3ivx codec), a graph will show up starting with your video file and ending with a video renderer. Click on the video renderer, delete it, and click on the xvid decoder and delete it. Then add a filter called file writer to the video stream coming out of the 3ivx media splitter.
Run that and you have an xvid mpeg-4 video stream with no audio.
2) Separate the amr audio stream
I used Miksoft 3gp decoder for this. For some reason I couldn't do video with this tool, so I just clicked the "convert audio only" option. It gave me a PCM .avi file.
3) Mix the two together
I have used two programs for this, both locatable through google and both freeware. Virtualdub and TMPGENC.
Virtualdub is handy if you have to rotate the video stream. I don't think you can use virtualdub to finalize the video because the .avi sound file kicks up an error for me. But the video I wanted to convert was filmed on my JAM with it held lengthwise (aka 320x240), so when the file is normally viewed, you see it 240x320 and have to tilt your head.
I prefer to use TMPGENC and end up with an MPEG-1 file. Here's why...
You input the xvid video file as the video source, the .avi pcm audio as the audio source, under advanced, select "Full screen (keep aspect ratio)" instead of the "full screen" that is presented there, and then let her run. You end up with a video in mpeg-1 that is formated to 352x240 (NTSC vcr resolution), but with black stripes above and below so that the image isn't stretched or distorted, like it would be if you just went with "full screen". You can then convert that into an avi file with whatever software you want.
It's a ton of work to do, but for me, it was a better solution than spending another $30US on something I'd use minimally. I also do a fair bit of video work, so I had all the other tools and codecs handy.
Good luck!
Hi
I use apples Quicktime - pro can also convert to AVI's.
BR
Daniel
The way I did it was to get the K-lite codec pack. Google it and don't download any versions from places that try to charge you for it, it's free. Also be aware of dodgy sites that may include spyware/malware with it. I've been stung a couple of times.
Then nearly any freeware converter will work. You can also play them in WMP.
Errrr...yes, that would seem to be a "faster" way of doing it...
screen capture
hmmm, need help...can't seem to capture images (using the screen capture button) when playing mp4 files on the vlc player...i've imported the image on adobe photoshop but it was distorted...help?
this is for info in relation to the original post
I too could not play my mp4 videos on my PC with sound and tried all the various players and plugins but this link from last year worked
http://forum.xda-developers.com/viewtopic.php?t=12308&highlight=mp4

[Q] Video and audio going out of sync with BS Player

I was using BS Player with my Samsung Galaxy S2 and I didn't have any issues with it. Audio/video sync was fine and I was able to use the video speed up feature to blast through those programmes that are a bit slow.
I've now moved over to the Nexus 4 and I am having issues. There's three options for the decoding mode:
Software decoding mode
Hardware decoding mode (BS player engine)
System decoding mode (System player)
I believe the default setting is System decoding mode. When I use this mode, the video speed up feature does work but the audio/video sync goes really badly out of sync (whether I use normal speed or whether it is sped up). If I enable the option to "drop frames when going out of sync" then it comes back in sync every few seconds but it would still go out of sync within a second so it was equally annoying.
If I use the BS player engine, when I play a video, it tells me that it was not able to use the decoding mode and then reverts to the software decoding mode so I've set it to the software decoding mode as the default. This keeps the audio/video in sync but the video speed up feature no longer works.
Has anyone managed to get things with with BS Player? Is this not working like my SGS2 because it does not have some hardware or built in codec support?
TIA

Bitstreaming with MX Player (or other)

I'm not sure if this is the right forum to post this in, or instead the MX Player forums, but I keep reading persistent claims that people are successfully bitstreaming lossless (TrueHD and DTS-HD-MA) audio to their AVR via the MX Player app on NVidia Shield.
My library consists of uncompressed Blu Ray rips made using MakeMKV. I take the lossless audio codec whenever available.
For the life of me, I can't seem to get proper playback. If I enable HW+, video plays with no audio. If I disable HW+ in favor of "HW", I get neither audio or video.
I haven't downloaded or installed any extra codecs specifically for MX Player yet... currently, within settings, it says I have "Use ARMv7 type custom codec".
Anybody having success with this care to expound on what settings are making this work?

Media player that outputs lossless native sample rate to Chromecast Audio?

Does anybody know of a music player app that will output hi-res FLAC files to Chromecast Audio without down converting to 48khz besides the built in LG Music app? Using the LG app I can play a 24 bit 96khz file and send it to the Chromecast Audio connected via optical cable to my Sony AV amp and the display panel shows that the file is playing 'LINEAR PCM [96kHz]' so I know that the V20 hardware is capable of passing the hi res files to the Chromecast. However, the native LG Music player is pretty awful a far as features and playlist support. I can use the built in Android screen casting feature to send audio and then use any app (including my preferred Poweramp alpha), but then all audio is downconverted to 48kHz.
Any other decent apps besides LG Music that offer casting?

[SM-G9750] [SOFTWARE BUG] Video playback being throttled or not utilizing CPU / GPU?

For owners of the SM-G9750 (S10+ CN / HK model), I'm trying to get some verification here on a possible software issue. I'm trying to play a 4K video file locally (specifically a 4K HDR H.265 / HEVC MKV @ 24fps video, but I believe this bug may apply to all UHD video playback - I'm not sure as I haven't done enough testing) and it seems to be dropping frames and stuttering throughout. I cannot get it to play smoothly despite trying all of the following:
Setting the phone to Performance mode.
Trying different players - in this case, the default video player, MX Player Pro and VLC.
Moving the video player app to Game Launcher and optimizing them for performance.
Forced 2D GPU Rendering in Developer Options.
Changing display resolution from WQHD to FHD.
Moving the video file from SD card to internal storage.
Disabling / uninstalling all apps which use Accessibility Services.
Clearing cache.
Factory reset.
None of these worked. ---UNTIL--- I tried using the screen recorder, from the Game Launcher settings, whilst watching the video. Playback was then perfect! No dropped frames, completely smooth, like it should be. The other alternative was to use software decoding but then colours would appear muted and battery life takes a huge hit.
So what's the deal here? Is the CPU or GPU not kicking in / being throttled whenever I launch a video player app? I'm assuming this is the case as the screen recorder seems to boost playback performance once I activate it. I'm on the June update and I'm not sure if this bug was introduced recently (I never tested 4K playback until now).
Owners of this particular model, could you please confirm if you are having this issue? Try watching a 4K video with lots of action and camera panning movements - that's where you can detect dropped frames or stuttering easily. I could provide the file I'm using for reference but it's a clip from a movie and I'm not sure if I'm allowed to share it here.
Okay since no one has replied and moderators on the Samsung forums are busy removing my posts - try this clip for example, it's 4K HDR encoded in H.265 / HEVC, I don't know if that makes a difference:
http://www.mediafire.com/file/m83ftn7n8d8le2s/Plane_Crash_Scene.mp4/file
Stutters in the default video player app for me. One thing to note is that it's smooth in portrait mode, but the moment you rotate into landscape fullscreen, it starts dropping frames. Now, turn on the screen recorder from within game tools and the stutters are mostly eliminated. What gives?
Another way to compare would be with a video player that lets you switch between hardware and software decoding, like MX Player. Watch the clip in HW mode and then switch to SW - it's choppier in HW mode for whatever reason. Using SW is not an option as the colours are not displayed properly and battery drain is much higher.
Regarding the clip - if it's against the rules to post such content, let me know and I'll remove it. There's a whole YouTube channel full of this stuff, I don't want to link to those as the problem I'm encountering is related to local video playback.
One last try before I sign off Here's another example if anyone has YouTube Vanced (google vanced.app for those unaware). I'm suggesting Vanced because the original YouTube app doesn't let you view at 4K resolution.
Firstly, open Vanced and tap the account icon in the upper right. Go to settings, Vanced settings, Video settings and then toggle Resolution Checks to ON. Refresh and restart the app.
Now open Game Launcher, tap the three dots in the upper right > Add apps and add YouTube Vanced to the list. We're doing this for the screen recording function which I'll mention later.
Once that's done, here's a 4K 60fps clip for you to try. It should be easier to perceive any hiccups with the higher frame rate. And of course, please set the resolution to 2160p (tap the three dots in the upper right) and watch in landscape orientation, since this is what the test is all about:
https://www.youtube.com/watch?v=LXb3EKWsInQ
Panning shots work best. For reference, see the one of the step trail at 2:42 in the video. Does it look like it's stuttering or dropping frames intermittently?
Now, swipe up from the bottom of the screen to show the navigation bar (you need the navigation bar enabled to use the recording function) and tap on Game Tools, which is the leftmost button. Tap Record.
While the screen is recording, are the stutters now gone? To my eyes, the video is definitely smoother while the recorder is active. Appreciate any feedback from those willing to try (and confirm if I'm insane or not!) - cheers!

Categories

Resources