Slashdot is powered by your submissions, so send in your scoop

 


Forgot your password?
Close
typodupeerror
×
Graphics Software

Notes On The Future of Video on Linux 126

Posted by timothy from the must-be-visible-and-moving dept.
Dina's Dream points out two interesting articles currently running on LinuxPower, and linked from Gnotices (GNOME news site) as well. "The first article is a really good summary of the current state of affairs of video under Linux and the direction we should take. Questions are bounced back between a few very knowledgeable people, including GStreamer developers, SGI people and Alan Cox. The second article is a set of lessons learned by Chris Pirazzi while working at SGI. Chris was involved in a lot of Video API programming at Silicon Graphics, and raises a few very good points based on his experience. All people even remotely working on video drivers or software should read these points and take them to heart."
This discussion has been archived. No new comments can be posted.

Notes On The Future of Video on Linux More

Notes On The Future of Video on Linux

Comments Filter:

  • Awesome (Score:1, Interesting)

    by Anonymous Coward

    I love this philosophy. Cut the crap, focus
    on what's important, and you end up with the
    right facilities to build higher-layer stuff
    on top of later.

  • DivX ;-) (Score:2, Offtopic)

    by Apreche ( 239272 )
    Yeah, there are lots of DivX players for linux. I've seen them everywhere. But how come none of them seem to work with Mandrake? I tried everything. The #1 problem with linux is no longer hardware support, since Mandrake supports all the hardware in my computer, even my printer, automatically.

    The #1 problem is ease of software installation and configuration. Sure rpms are great, until you get some that depend on something else. Which depends on somethign else, which dependson the rpm you were trying to install in the first place. Not to mention when you try to install something from source, and it wont compile. I don't have the time or the patience to figure out what is wrong with someone else's code.

    I know how to code yes. But when I donwload a DivX player and it fails to make, make install. I have no clue how to fix it. That's why I'm downloading a DivX player, and not writing my own. So far Tribes 2 server, and Netscape/Mozilla are the only linux programs that I've seen (there are probably more) that have windows style installations. This means it can be done. Do it more often. Like all the time. Then maybe I wont have to reboot my computer to watch movies.
    • Really? One of my stations runs XINE under 'drake 8.1 juuuust fine.
    • While I agree that getting DivX working in linux is a hack job, it can be done. I urge you to think about giving a little more credit to the open source, no funding projects that even allow the possibility of DivX to work. Sometimes these people code it in their spare time with little or no monies. Usually I try to appreciate any "free lunch", even if it's not a sirloin steak.

    • Re:DivX ;-) (Score:5, Informative)

      by Alan Cox ( 27532 ) on Wednesday February 27, 2002 @03:25PM (#3079275) Homepage
      DivX is waste of energy and programmer time. Its so patent trapped that one yawn from the movie industry to the patent holders and it'll be gone except in very limited form

      Something like VP3.2 holds more promise (and hey I got the lib to actually -compile- on a non windows/mac box two days ago)

      Now combine VP3.2 video with Ogg audio and you have a credible media format. Add xhtml navigation and you have something really cool

      • Re:DivX ;-) (Score:3, Interesting)

        by MartinG ( 52587 )
        Interesting. I hadn't seen vp3.2 before.
        I notice their license is derived from the Mozilla Public License 1.1
        Can anyone comment on whether that is good or bad news? eg. will the gstreamer folks be able to include vp3.2 codec support and distribute it as a plugin?
        • VP3 is distributed under a proprietary, non-free license that, whilst it purports to be open, meets neither the Open Source nor Free Software definitions.


          The problem lies herein [vp3.com]:

          (e) Notwithstanding Sections 2.1 (a), (b), and (c) above, no license
          is granted to You, under any intellectual property rights including patent
          rights, to modify the code in such a way as to create or accept data that is
          incompatible with data produced or accepted by the Original Code. By way of
          example but not limitation, a Modification that adds support for other
          compression data such as MPEG-1 or MPEG-2 would be permissible, but only if the
          resulting Larger Work continues to support playback of VP3.2 data.
          Modifications that provide only playback or encode support are also permissible.
          However, a Modification that adds support for encoding or playback of any non-
          VP3.2 compatible files or bitstreams without complementary support for VP3.2
          encoding or playback would not be permissible, and no license is granted for
          such Modification(s).

          Basically, this is denying users the right to modify the source code to produce binaries that produce a stream incompatible with the original software. It may sound good to some, but I urge developers to think twice before releasing modifications or compiled versions of the VP3 codec because, even if unintended, a compiler bug or error in your modifications to the software could mean that the stream your modified VP3 codec produces is unintentionally incompatible with the VP3 specification, opening you to legal procedure from On2 Technologies, the proprietors of the codec.


          The VP3 codec licensing terms are not only not Open Source, they are a threat to developers, contribuors and distributers of VP3 both in source code and compiled form. Please contact On2 Technologies and try to convince them to update their license to remove this dangerous clause, and spread the word to your friends!

      • Why is VP3 different with respect to patents? On2 has patents, they seem doing a good thing and try to let people use their IP. But they could be bought in a heartbeat and work that you and others put into it would be gone.

        What they should do is give the patent to a conservancy, like collabnet's [collab.net] or the Knowledge Conservancy [cmu.edu].
    • I absolutely agree with you in regards to the creation of custom installers. RPMS are wonderful when they work. But getting stuck in dependency hell is worse than gum surgery.

      It's great people are donating all this time, but if Linux is ever to suceed on the desktop outside its niche audieneces, than this simple problem must be addressed. Plus in terms of downloading, I do not mind waiting for a large download with all the dependacies included, rather than searching the net like a lunatic just so I can get this app to work. By the time I eventually give up, I'm ready to go back to Windows.

      Oh yeah - video editing on Linux would be fantastic. (waves wand - poof - there it is)
      • I do not mind waiting for a large download with all the dependacies included, rather than searching the net like a lunatic just so I can get this app to work

        Then you'd just end up in "DLL Hell". Each program would install it's own slightly customized version of a lib breaking another program which would in turn be broken by another install with it's own custom libs. Be carefull what you wish for. There's no easy answer. Besides, compiling a program yourself will yield better performance.

        ---
        Extra! Extra! Read all about it [slashdot.org]! Slashdot editors censor dissenters [slashdot.org].

        • Re:DivX ;-) (Score:1, Troll)

          by Foxman98 ( 37487 )
          You know what. Bullshit. I'm sorry - I see this comment all the time and in all the time that I've been using windows I have *never* had a problem with my .dlls. This includes a win2k machine in heavy use every day, 7 days a week, for 2 years without a reload. And you know what? It ran just fine...

          Ok, don't get me wrong. I love linux just as much as everyone else here does, I use it all day long at work, by choice. But I must say that installing software, in general, is a pain in the ass. Am I blaming anyone? no. Is microsoft the answer? no. But they sure as hell do have the installation part down pat.

          • Re:DivX ;-) (Score:2, Informative)

            by bloo9298 ( 258454 )

            IIRC, Win2k and recent installation generators partially avoid the DLL hell problem by encouraging applications to install their own DLLs in their own separate installation directory. This negates many of the advantages of DLLs.

            DLL hell has been a genuine problem for a considerable time (>2 years), even if you haven't seen it in Win2k in the past 2 years. Try setting up a Win95 machine and install some old software if you are feeling masochistic!

    • Funny.
      I also reboot my computer to watch movies, but in the opposite direction.
      On my computer, the only things I do is to play games (mainly quake3 and total annihilation) and watch movies and music.
      I usally run windows, but when I want to watch a movie, I just can't face the crappy windows media player. (Yes, I tried others. bsplayer, microdvd, etc. no one satisfied me.)
      Then I boot linux to watch it on MPlayer [mplayerhq.hu], by far the best player i have seen.
      By the way, MPlayer is only avaiable as source code, but I never had any problem compiling it.
      • Same experience here. MPlayer [mplayerhq.hu] is by far the best player I know, no matter the platform. Apreche, I'd seriously advise giving it a try -- it works great and it's very well documented. I've never had a problem with it, and I even think it's one of the most underrated open source programs. And if for whatever reason you can't seem to make it compile right, toss an email my way (balin[]rne.eu.org with [ares@ie] in gap) and I'll see if I can help. :)
        • Does anyone know how, in Mandrake 8.1, one can make konqueror open say, a .avi file, with mplayer *but* without opening a new instance of mplayer. For exmample, I have a creed video playing right now, if I were to click on a different video, konq will open up a new instance of mplayer which doesn't work. Any hints?
      • Really. Installing this from source really is quite easy. Just do

        ./configure [options]

        make

        make install

        And on my matrox card I get hardware scaling and other good stuff.

    • Re:DivX ;-) (Score:3, Informative)

      by LinuxGeek8 ( 184023 )
      You should check out the PLF distro (not PLD).
      It contains non-free rpms for mandrake, installable with urpmi/rpmdrake.
      Only mplayer is not available in binary rpm, just source.rpm.

      PLF is actually built on a current cooker distro (cooker-mandrake-devel), but there is someone recompiling them against mdk 8.1.
      Just try them.

    • Re:DivX ;-) (Score:2, Informative)

      by Blasphemy ( 78348 )
      Yes, DivX isn't easy under Linux, but this is because it's a windoze format. It was written under windoze for windoze, supposedly using hacked microsoft code.

      DivX for Linux is still in the development stage. There are no mature players, and until there are, it will not be super simple to install.

      That being said, you should take a look at mplayer (see freshmeat.net). Mplayer has step by step instructions on how to compile and install avifile, libdvdread and other tools that come together to make a really great player. Yes, you do have to ./configure make make install at least two packages, but the resulting player is worth it. In addition to the many formats supported, you get a player that is very well optimized for your cpu and graphics card (well, only Matrox cards are really well done right now, but ATI and Nvidia support are coming).

      Give it a try, it works.
    • Switch to debian ;-)
      Ok, so it may take many long hours of work to get it to support your hardware, but then you can just apt-get install and get all the dependencies as well.
      Or use one of many graphical frontends -- it beats the hell out of windows installation.
    • I, as much as anyone else, hate rpm dependency hell.

      However, I don't think we'd have such a vibrant open source scene without shared libraries like zlib and libjpeg.

      If you are releasing a stable version of your code try to depend on as much as possible, the version of whatever libraries you need which came with the latest RedHat.

      If people did this and RedHat had an extension to rpm...

      `libpng not found, insert redhat cd 1 to install'

      I think dependency problems would be much reduced.

      Other than that, you want to live on the bleedin' edge, you're gonna have to do some work (or use Debian :)).

      Or we go XBox-like and statically link everything :)

      • Re:DivX ;-) (Score:3, Informative)

        by volsung ( 378 )
        Or we could do what they do in Windows land, which is bundle every dependent library with the program and install them if they aren't already there. This is easy in a world of shrink-wrapped CD installs (where you have space to burn), but is a good deal more cumbersome with downloaded applications.

        urpmi (in Mandrake) already does exactly what you request from rpm. It resolves all the dependencies and will either download the RPMS from an online source or ask you to insert your installation CDs. Hopefully Redhat will snarf it into their next distro. (That would actually be quite appropriate given how much Mandrake has snarfed from RedHat. Both distros would benefit from some cross-pollenation.)

    • The #1 problem is ease of software installation and configuration. Sure rpms are great, until you get some that depend on something else.

      Or you could just get apt-get and let it sort out the dependencies in your rpms for you automatically. The Freshrpms apt archive (for Red Hat 7.2) has packages for
      • Xine
      • Ogle
      • MPlayer (source packages only as Mplayer isn't Open Source)
      • Drip (a DivX encoder)
      • Transcode (another DivX encoder)

      And all you need to install them is type apt-get installl (whatever) to fetch the rpms, whatever dependencies they need to install, and install the packages.
    • Dude, have you ever tried out MPlayer?
      http://www.mplayerhq.hu/
      It support *nearly all* video formats available today!

  • The Future Is Here for me already (Score:5, Informative)

    by anewsome ( 58 ) on Wednesday February 27, 2002 @03:06PM (#3079153)
    My pvr production is running slick as can be, now that I have ditched all the capture card madness (WinTV, Buz, etc). All I need now is a $30 firewire card and my trusty Canopus ADVC-100 and I capture full frame 720x480 video in better than DVD quality. If you are considering doing a lot of capture/encoding of anything, I highly recommend this setup. dvgrab running under linux seems to be very stable and the captured video is virtually identical to the source.

  • SSSCA (Score:4, Insightful)

    by Anonymous Coward on Wednesday February 27, 2002 @03:06PM (#3079155)
    Well, if SSSCA passes, there won't be any video on linux.

    • Re:SSSCA (Score:2, Interesting)

      by Russ Steffen ( 263 )

      If the SSSCA passes there won't be any linux. Not in the US anyway.

    • Well, if SSSCA passes, there won't be any video on linux.

      What are they going to do, search the entire world to find and destroy machines/archives with linux video software? Because that's what it would take for there to be no "video on linux."

  • Complete MultiMedia Architecture (Score:3, Informative)

    by Anonymous Coward on Wednesday February 27, 2002 @03:11PM (#3079192)
    There is one group working on a complete multimedia network architecture. More information can be found unter www.networkmultimedia.org [networkmultimedia.org].

    Quote from the web-page:

    "The goal of this project is to design and implement a network-integrated multimedia infrastructure for Linux as well as other operating systems. Since many low-level parts of such a system already exist (like en-/decoders, de-/compression, multimedia networking APIs, CORBA, and others), our primary goal is the development of an architecture, which integrates these usually isolated modules. This unified architecture will offer a simple and easy to use interface for applications to integrate multimedia functionality. Therefore, it can be used as enabling technology for traditional multimedia applications, but also for ubiquitous computing and mobile computing.The result of our work will be made available as OpenSource."

    - Jarman

  • SDL? SDL. (Score:2, Informative)

    by donglekey ( 124433 )
    I read the article on video for linux, and what the author basically describes is SDL. I think that he means it to be a layer under what SDL is, so it wouldn't be portable, but the functionality is basically SDL.

    • Re:SDL? SDL. (Score:1, Informative)

      by Anonymous Coward

      SDL doesn't do video. It does still images, blitting, etc., and it can provide an encapsulation for OpenGL contexts, but it doesn't do Video. There's a callable smpeg player that can play mpeg movies from an SDL app, but that's much more simplistic than what he's talking about.

      Video, in this context, is movies, not computer graphics.
    • i think we need a more fundamental cross platform structure, OpenGL/OpenML/OpenAL combined with input and some other candy to form OpenXL or something of the sort? and maybe have real C++ bindings (do i have to call 3f?)
  • Are they still around? Is anyone continuing their work? And what of VirtualDub? Is there a Linux port out? Anyone know? It sucks to have to boot into Windows just to use premiere... (Yeah, yeah VMWare, but shhh :) ).

    • Re:Whatever happened to Broadcast 2000? (Score:5, Interesting)

      by qqtortqq ( 521284 ) <<mark> <at> <doodeman.org>> on Wednesday February 27, 2002 @03:30PM (#3079318)
      Its interesting you mention broadcast2000. I used to use it, but gave up due to crashes and things, but it was a neat program if they ever got the bugs out of it.

      From: http://heroinewarrior.com/bcast2000.php3 [heroinewarrior.com]

      After a long period of deliberation on the matter, Broadcast 2000 has been removed from public access due to excessive liability.

      We've already seen several organizations win lawsuits against GPL/ warranty free software writers because of damage that software caused to the organization. Several involved the RIAA vs mp3/p2p software writers. Several involved the MPAA vs media player authors. You might say that warranty exemption has become quite meaningless in today's economy.

      Fucking dmca....

      • The HeroineWarrior people have a wierd way of interpreting things, and are their writings are really paranoid.

        But, it is only a small loss in my book, since Broadcast 2000 was pretty awefull. They went all out optimizing it for Linux on Intel (and/or AMD) without first working out all the bugs or making an less optimized version in plain platform independent C that could be ported to other platforms (like Irix since O2s with video IO are pretty cheap).

        The other major flaw of Broadcast 2000 was that it kept trying to produce flashy effects without focussing on the basics (which are extremely important for making fancy effects work and blend into the project).

        Basically, it seemed to suffer from the by programmers for programmers problem, which is bad when one is trying to make software for artists.

        Kino (http://www.schirmacher.de/arne/kino/) looks superior. Sure, it doesn't have the same feature list, but from the descriptions, the workflow is more usefull. But, I haven't tried it since they work only with firewire style DV, and some of us prefer to capture video on a dedicated capture machine through high end capture cards from professional grade analog tape rather than stick firewire in a machine, so I can't use Kino either until they see the light.
  • That sure was slashdotted quickly, does anybody have a mirror? Somebody should write a program to refresh /. a lot and mirror pages in the new articles right when they are posted, before the /. effect hits, lol.
  • Appears they are slashdotted to hell and back.

    (sigh)

  • video focus (Score:4, Insightful)

    by paulbd ( 118132 ) on Wednesday February 27, 2002 @03:26PM (#3079282) Homepage
    the slashdot headline is more accurate than the article's actual title. the author's approach comes almost entirely from a consideration of video. if he was starting from a primary interest in audio, he would have talked about many different issues, and mentioned different kinds of solutions. gstreamer is a cool system, but it needs to be stressed over and over that gstreamer is an architecture for building applications. it does not offer any mechanisms for inter-application communication or synchronization. since most people want to do a lot more than write a particular plugin for gstreamer, gstreamer doesn't help us when the challenge is not providing an architecture for a single program, but one for multiple applications on the same (or even networked) system(s). when you want to run a cool video processor along with a really nice FX rack for audio, gstreamer can't help you unless the author of each component had decided to implement his stuff as a gstreamer plugin. since this cramps the GUI style rather considerably, its unlikely that many people will choose to do this. finally, i would note that although it has become customary to sync audio to video, this actually makes very little sense when the temporal resolution of audio (22000-96000 frames/sec) is vastly greater than video (20-30 frames/sec). its really just an artifact of the way technological development has happened, of who has the most power in the entertainment "content" business, and of the fact that we generally consider visual data more significant than acoustic data. we'd be in much better shape is the conventional approach was to sync a video stream to audio, since we could easily and uniformly take advantage of the much better clock resolution that audio devices provide.

    • Re:video focus (Score:5, Insightful)

      by vektor_sigma ( 178320 ) on Wednesday February 27, 2002 @05:31PM (#3080369) Homepage
      Paul: You mention that it makes little sense to sync audio to video and then state that this is what most applications do.

      On the contrary, syncing video output to the audio is much easier, since audio resampling is a big pain. It's much easier to just watch the soundcard buffer and decide approximately when to show the frame as you imply. You only run into big difficulties when you want to sync audio to the video (or more accurately, sync video blits and audio output to the vertical refresh rate of your output device).

      There are times when this is a very important. You imply that video framerates are between 20-30fps, which is quite small. For playback of video sources we need to handle framerates of 50fps and 59.94fps. At such a high framerate, the monitor refresh must track the video input in order to achieve smooth video. See the link to Dave Marsh's faq on judder I mention in the article.

      In these cases, you need to set the monitor to the correct refresh rate and then watch the error between the refresh and the sound card clock, and resample the audio when necessary: playing with the refresh rate on the fly would cause a monitor resync and disturb playback!

      If video cards let applications drive the refresh of the monitor (software genlock), we could run it based on the audio clock and get the advantages you describe. However given the current state of the hardware it's better to do these small resamplings. That said, actually doing this in linux is still infeasible without some updates to the APIs and drivers, which we are working on.

      -Billy Biggs
    • I have to disagree with your video-chasing-audio proposal. While it would be simpler so code something like that (ignoring issues of monitor sync), in real life this approach wouldn't work. In a real post-production environment video has to be very rigidly clocked, and when working with digital audio interfaces the audio has to be rigidly clocked as well. The video (in NTSC land) goes at 59.94 fps (fields-per-second) and audio generally is at 48K/sec. There really isn't any way of resampling the video, which means that you have to slave your video signal to your sync generator or generate your own sync and then resample the audio (if necessary) to get the proper 48K samples/second. There are a myriad of reasons why you can't do this in reverse, most of which should be fairly obvious (just try resampling the video in real time, and it's a lot easier to pass around a 59.94Hz sync pulse than a 48KHz one).


      • Resampling video real-time should be rather easy since we're not talking about reformatting the video or editting the frames, just changing the time they are displayed. But there are probably other reasons it wouldn't work that way, refresh rate, etc.
        I don't know how we encode audio/video data today, but shouldn't we add some metadata that shows which audio frame syncs up to which video frame for every few seconds of video? Something as simple as 2 frame numbers every 2 seconds shouldn't take up that much space. And resampling audio to sync up with video using the popular algorithms that don't change the pitch would kick ass, IMO.

    • Re:video focus (Score:3, Insightful)

      by Omega Hacker ( 6676 )
      First of all, the focus of the article was indeed multimedia, from the point of view of the average person who wants to use Linux to watch DVDs and videos off the net. These people really couldn't care less about perfect sample accuracy of their $10 sound card. Trying to force pro-audio qualities onto every machine is impossible, and highly counterproductive in the face of other more visible issues.

      GStreamer is indeed an infrastructure for building multimedia apps. Absolutely everything it does is under the auspices of the elements that it loads and connects together. Claiming that GStreamer has no means of inter-application communication is false, as there are already numerous elements both for communication with sound servers and other applications (network source/sinks, etc.). A set of Jack elements are being written, as well.

      GStreamer also has nothing to do with the GUI whatsoever. Elements are GObjects that have no GUI in them at all, they focus on doing what they're supposed to. Any GUI exists as a seperate entity on top of the processing pipeline as managed by GStreamer.

      If your worry is that LADSPA won't be used, don't. LADSPA plugins are fully supported with a shim under GStreamer.

      As for whether to sync to audio or video, you actually have it quite backwards. First of all, the most difficult situation is not with progressive video, but with field-based video (which both NTSC and PAL are, BTW), where the vertical rates are 50 or 59.95Hz. Compare this to a CRT, and you have significant problems finding a decent match between them.

      As for the "content" business somehow making video more important than audio, you're ignoring the fact that video *is* more important than audio when both are present together in the same stream. There are several order of magnitude more bits in the video than than the audio, anyway.

      Now, to the technical reasons it makes vastly more sense to absorb changes in the audio clock:

      1) 90+% of computers have sound cards with clocks that can be best described as "kinda sorta correct". They vary wildly around the real 44.1k baseline. If anyone notices this, there's not much they can do about it. More fundamentally, how are you going to maintain any kind of stream clock when the audio rate is changing during the presentation (as it will when temperature and other things change in your computer, since the clock is susceptable to this)?

      2) The quanta of video playout is much greater than that of audio, on the order of 500+ times larger. The goal of any good video playback is to synchronize the playout of a video frame (theoretical time unit) to a vertical refresh (physical time unit). If at any point those become desynchronized, you will have a discontinuity of at least one vertical retrace, or on average about one 75th of a second. Specifically, you'll have a video frame that is suddently displayed for between 50% and 100% longer than it's supposed to be, in the middle of a bunch of correct frames. This is *blatantly* noticable by anyone with a normal visual cortex.

      Audio, and the human ear, is much more forgiving. If the program simply drops or duplicates a sample every once in a while to maintain minimal drift between the two (video and audio) clocks, it will be altering 1/44,100th of the samples in that second. If done wrong, it will cause a click. Doing it "less wrong" to avoid clicks is trivial.

      So the decision is between locking to a highly variable audio clock (think 3.6 seconds per hour per 0.1% off) and having video that jerks and sputters whenever the theoretical and physical frame times disagree, or doing some resampling to the audio where necessary, with the possibility of some loss of quality that is undetectable by 99% of people watching videos on their computer with tinny speakers.

      Me, I don't like video-induced headaches, I'll resample the audio.
      • First of all, the focus of the article was indeed multimedia, from the point of view of the average person who wants to use Linux to watch DVDs and videos off the net. if "multimedia framework for linux" has come to mean "a system designed for and limited to consumer usage", then that's fine. i consider that a useful system, but an inadequate one at the same time. i think its possible to do better. GStreamer is indeed an infrastructure for building multimedia apps. Absolutely everything it does is under the auspices of the elements that it loads and connects together. Claiming that GStreamer has no means of inter-application communication is false, as there are already numerous elements both for communication with sound servers and other applications (network source/sinks, etc.). A set of Jack elements are being written, as well. such connections are not first class citizens in GStreamer's world. you are not using GStreamer to connect apps, you are using a GStreamer element that may obey GStreamer's semantics and design, but operates via a wholly different mechanism than the connections between GStreamer elements. the fact that GStreamer can talk to JACK, for example, isn't as notable as the fact that unless one of them drives the other, the two systems on each of the interconnect are not running synchronously with each other. this discussion has all been played on both LAD and jackit-devel. AFAIK, GStreamer does not currently allow elements to drive its graph execution in the most fundamental sense. GStreamer also has nothing to do with the GUI whatsoever. Elements are GObjects that have no GUI in them at all, they focus on doing what they're supposed to. Any GUI exists as a seperate entity on top of the processing pipeline as managed by GStreamer. my point was that if decide to write a really cool software audio "device", its likely that part of my "device" will be a really cool (G)UI too. if i write my device as a GStreamer element, then given that my element will execute in the process context of the GStreamer app that loads the element, what do i write the (G)UI with? this is right back to the issues we've discussed on LAD for ever about the incompatibilities of different GUI toolkits. As for whether to sync to audio or video, you actually have it quite backwards. First of all, the most difficult situation is not with progressive video, but with field-based video (which both NTSC and PAL are, BTW), where the vertical rates are 50 or 59.95Hz. Compare this to as i understand it, they are refresh rates, not frame rates. frame rates are lower than that. i may have this wrong - most of my understanding of video refresh/frame rates comes from protools, which is probably not the best place to start. As for the "content" business somehow making video more important than audio, you're ignoring the fact that video *is* more important than audio when both are present together in the same stream. There are several order of magnitude more bits in the video than than the audio, anyway. it depends on the work in question. there are some things for which the audio is vastly more significant, other things where the video is, and many where it can be hard to tell. i would agree you that for most "video" its obviously the video that dominates. if you want a great framework for video playback, then call it that, not "multimedia" which has broader connotations. 1) 90+% of computers have sound cards with clocks that can be best described as "kinda sorta correct". There are (at least) two versions of "kinda sorta correct":
        • runs with low jitter, but absolute frequency is wrong
        • absolute frequency is correct, but jitter is poor
        In the first case, there's not much we can do: the clock just isn't really suitable for use. In the second case, the much greater resolution/frequency of the audio clock means that we can have fairly high jitter and not really affect the video refresh timing we generate from it. 2) The quanta of video playout is much greater than that of audio, on the order of 500+ times larger. The goal of any good video playback is to synchronize the playout of a video frame (theoretical time unit) to a vertical refresh (physical time unit). If at any point those become desynchronized, you will have a discontinuity of at least one vertical retrace, or on average about one 75th of a second. Specifically, you'll have a video frame that is suddently displayed for between 50% and 100% longer than it's supposed to be, in the middle of a bunch of correct frames. This is *blatantly* noticable by anyone with a normal visual cortex. this isn't really the point. the idea of audio/video sync, as billy pointed out before, is to decide which video frame should be blitted at a given audio frame time. none of this has to interfere with the video refresh rate. its just a matter of picking the correct frame every time you do the refresh. no more or less, i think. you're going to only ever either duplicate or drop a single video frame. Audio, and the human ear, is much more forgiving. If the program simply drops or duplicates a sample every once in a while to maintain if only that was it. however, the video clock is updated at a very low rate - you can only notice out-of-syncness 20-50 times a second, which means that you will need to drop a lot more than a sample every second or so to correct for drift unless you actually resample the audio. So the decision is between locking to a highly variable audio clock (think 3.6 seconds per hour per 0.1% off) and having video that jerks the entire point of synchronization is that it doesn't matter what's going on with the clock - all the slaves follow the clock. if the audio is a little on the slow side, so is the video. and the video doesn't have to jerk for reasons described above. and sputters whenever the theoretical and physical frame times disagree, or doing some resampling to the audio where necessary, with the possibility of some loss of quality that is undetectable by 99% of people watching videos on their computer with tinny speakers. i don't want to see linux, an operating system with great promise in this area, end up with a consumer-oriented multimedia API, when one suitable for professional use will also serve consumers well. Me, I don't like video-induced headaches, I'll resample the audio. its a fair point. it would be a better point, i think, if video induced headaches were inevitable from having the video follow the audio. i don't think they are.

  • My experience with Video on Linux (Score:5, Informative)

    by lkaos ( 187507 ) <anthony@NOspaM.codemonkey.ws> on Wednesday February 27, 2002 @04:05PM (#3079617) Homepage Journal
    Video on linux has always been one of my favorite things.

    The biggest advance in v4l was essentially the XFree86 4.x release since it encorporated an Xvideo extension allowing for really nice video play back in Linux.

    There are a couple cards that are extremely well supported. I personally use an ATI AIW and the MPEG playback in incredible. In fact, I prefer MPEGs in Linux verses Windows simply because I think Linux does a better job at using the RGB conversion stuff at the hardware level than Windows does.

    Of course, the biggest foe to video in Linux has been the fact that many of the best accelerations (iDCT) are simply not supported because card makers fear releases 'techinical secrets.'

    Another problem is the split in video display APIs. Prior to the Xv extension being released, the Linux kernel had a video4linux API. The second version of that API is incredible as it has so many features that would allow for truely pluggable components.

    Unfortunately, all X stuff is implemented in user space so cards that have integrated display and video stuff end up supporting everything in user space and then providing a loopback mechanism to work with the kernel API. It's a little messy but hopefully everything will get worked out as things progress.

    Otherwise, hats off to all the hard work of the gatos project, the v4l2 project, and linuxvideo. If you haven't tried all the really cool stuff available for video on linux, you really should.
  • This isn't exactly the same topic, but a related issue and something I've been upset about for years.

    Namely: graphics drivers should be moved into the kernel, which should provide a very low level graphics API

    There are many people opposed to this as bloat in the kernel. But come on, there are so many things in the kernel that should be called bloat if gfx is bloat (like sound for instance).

    And of course this is related to video. I think these low level drivers would include support for TV/out, processing signals from TV cards, standardizing APIs.. etc.

    • Anyway, I have many reasons FOR this.
    • We have massive duplication of effort with many different projects developing their own low level drivers for gfx cards. this would eliminate that. Sure they could all have some card-specific code, but we wouldn't see nearly the level of duplicated effort we do now (think GGI-X-directfb-svgalib-list goes on)
    • It is my belief that vendors would be more motivated to create drivers if we had a simple and super standard low level API. 'gfx driver for linux' wouldn't mean 'write an xserver' or 'write an svgalib driver also'
    • This in my belief is what it would take to get linux gfx to a much higher state

    • There are many people opposed to this as bloat in the kernel. But come on, there are so many things in the kernel that should be called bloat if gfx is bloat (like sound for instance).

      First off, sound support in the kernel is extremely minimal. It just provides a method for userland programs to access the hardware, and other systems like ESD handle the higher-level stuff.

      Second, I've rarely had a sound card crash, but my nVidia graphics card has been known to seize on occasion, often because of minor conflict with the X driver. However, when this happens, I can still SSH into the machine and shut it down cleanly. If your idea were used, those crashes would take down my whole box, and that would suck the big one.

      • what would the diffrence be in running a process as root in user space crashing and overwriteing random parts of memory or a program running in kernel space and crashing and overwriteing random parts of memory.

        I think in the kernel would be great for a low level graphics access. really it just needs fb on steroids.

        • Since when did a process avoid the VM system just because it was running as root?

          Again, when my userspace graphics driver crashes, I can SSH in and take the system down cleanly. When a kernel driver crashes, you don't typically get that luxury. This is not a toy for me. I have to do real work on this machine, and the last thing I can afford is kernel panics while the drivers are perfected that make me to fsck my drives and cost me time.

          • I have never had a graphics card crash on me unless I was doing something that was well out of the ordanary. I am not saying that it doesn't happen, just that if the code is stable enough it should be in the kernel with the rest of the device driers.

            Of course the linux kernel is going to move to a more microkernel like hybrid than it is right now witch would put it into the possision to support the ability to run things like graphics drivers as kernel modules.

            With linux we get the best of both worlds. the speed and simplicity of a monolithic kernel and the modularity of a micro kernel.

    • You should try out the 2.4 version...

      It has drm included in the kernel.

    • I have been saying this for over a year now.... If modules like khttpd can be optional ... why the hell cant we have an optionalo intergrated video api. In my experience .. there are 2 types of an os ... ( user space ( i did comp sci so I know about monolythic ...) ) ... 1. desktop or 2. server ... its pretty obvious which one should have the vid api included at compile time :-)

  • dScaler (Score:1, Insightful)

    by Anonymous Coward
    For me, one major thing holding me back from using Linux as my home theater PC is the lack of a Linux port of dScaler [dscaler.org]. My own programming experience isn't enough for me to wade through the Linux video APIs to get the job done. Not to mention the fact that Linux APIs for video change every year or so (frame buffering, Xvideo, v4l, v4l2, etc).

  • a refreshing perspective (Score:4, Insightful)

    by FrostyWheaton ( 263146 ) <mark.frost@gmailMOSCOW.com minus city> on Wednesday February 27, 2002 @04:49PM (#3079984) Homepage
    This has to be the only article I have read in a long time that stresses the importance of doing things the right way, instead of the wrong way or, heaven forbid, the "Max Power" way.

    There are many core issues with video on any Unix that need to be hammered out now to ensure that things will go well both now and in the future.

    As the author mentions several times, adapting refresh rates to video frame rates and working with the monitor's vertical sync as well as audio sync etc, are all very important things that need to be implimented before Video for (insert favorite unix here) will become anything more than a glorified hack.

    The first logical step is to impliment what is needed to do things right, and to impliment them in the right(proper) way. the X-protocol should be fully implimented in Xfree, and the kernel should be extended to enable applications to be written which can make full use of the hardware, with minimal kludge-work.

    Then the focus moves to making the "killer-app" type media production tools and players. The power of Open Source is the ability to build on the work of others. However, stealing someone's hack to adapt refresh rates, and jamming it into your own code is not an optimal solution. Focus on doing things right the first time, anything less (especially when dealing with core issues) is just asking for untold headaches and frustration in x years, when we are kicking ourselves for not doing the right thing the first time
  • Some of the motivation for the design elements that Chris objects to is the desire to exploit very sophisticated hardware. Ideally you develop to one API and the application exploits very different hardware on heterogeneous systems. Mainstream PC cards are only just starting to get sophisticated about video so it doesn't on the face of things need some of this, but if you don't abstract then you are stuck with applications that either can never exploit better hardware or need to be ported by hand to use it.
  • Forgive me if I am wrong, but the upcoming OpenML libraries are going to change a lot of whats going on here. If the industry supports OpenML like is does OpenGL then software programmers like us :) will be able to focus on OUR CODE and not have to worry about low-level media calls and having to use several different cryprtic media libraries just for multiple platform support! The Jahshaka project is moving to OpenML, so you can be sure that you will see OpenML video cards hitting the market real soon... http://www.jahshaka.com for more info
  • As long as Linus keeps multimedia and extraneous "features" out of the kernel. As in Operating Systems Design and Implementation by Tanebaum, the author of Minix, I tend to think that compartmentalization is safer from a system security and integrity POV, but hardware threads are nice. The only things that should reside in kernel-space are drivers and system functionality. I can't even see justifying IP stacks in the kernel since that could run on top of the ethernet or whatever is underneath because a layered approach can install/remove/restart parts of the system w/o killing the whole thing. Btw, MINIX [cs.vu.nl] starts in less than 1 sec from a HD on a 486. And pretty darn stable too, except for the lack of preemptable kernel processes. I suggest running it on Bochs [sourceforge.net].

    Multimedia codecs should remain where they should be, in libraries.
  • - Disclaimer: This is a nostalgic trip ;-)

    The problem is that the Linux kernel is currently missing a retrace interrupt counter...

    The Amiga had a vertical refresh interrupt - actually you could even synchronize to an arbitrary (x,y) coordinate on the screen via the "copper" co-processor.

    Some of the SGI graphics boards actually had an input jack called "genlock" into which you plugged a video signal ... This meant that the graphics signal was perfectly in sync with other equipment in the facility and so could be "mixed" with other signals in the facility.

    A "genlock" could be bought for the Amiga as an accessory for very little money (~100$ or so).

    The Amiga was very well adapted to television display. So much that the CPU clock frequency was some multiple of the horizontal scan frequency of either PAL or NTSC! This made for incredibly smooth graphics on a standard television, and so with an Amiga you actually had a very cheap system for generating title screens etc. for your homevideo. And this was more than 10 years ago! Actually i believe that some of the cable companies here in Denmark are still using an Amiga for generating their "info channels".
    • Actually i believe that some of the cable companies here in Denmark are still using an Amiga for generating their "info channels"

      The college I work for has their own cable TV network set up for the dorm residents. One of the channels that they get is an endlessly looping "info" channel with campus events and the like on it.

      We _just_ retired the Amiga 1200 that's been doing this for years. And it was only because the poor thing had started to overheat and crash - the job it did was still more than adequate.

      --saint
  • Lots of small tools that fit well together.

    1. Input grabbers-(tv cards,dvd,VCD,files,etc)
    2. Input decoders-(avi,mpg,divx,quicktime,etc)
    3. players (to screen, to video out, to net)
    4. output encoders (see input decoders in reverse)
    5. tools (convert from avi to mpg, split/join files, add sub titles/CC/OC/alternate audio & video tracks. Convert size/fps/color depth)

    Make all the tools pipable (that's the *nix way) That way I can implement the parts based on my hardware, time and disk space constraints.

    Don't tell me it can't be done - There are amny pieces already out there. They just need to be put together.

    BTW, I have a system that can capture TV, DVD, and VCD, play most formats, and record to VCD now. It took multiple tools to do the job, and some are not even the most 'up-to-date' ones that _should_ work. I will soon try recording to digital tape (aka a DAT drive) and see how that works as well.
  • Looks like video on Linux is only going to be the domain of professionals, motion picture houses with renderfarms and electronics manufacturers with the wherewithall to embed it into appliances. The age of the bedroom hacker hacking his PC to use Linux for video editing is almost over.

Slashdot Top Deals

Living on Earth may be expensive, but it includes an annual free trip around the Sun.

Close