Thursday, 17 January 2013

How to edit a DVB transport stream (TS) and retain the DVB subtitles (subpicture) stream using Linux or Windows tools and burn the result to DVD without transcoding

Many digital TV broadcasts in Europe include subtitles in the form of DVB subtitles. Unlike old-fashioned teletext subtitles, which are transmitted as text and rendered in the receiver, DVB subtitles are actually broadcast as bitmap images, which allows any font or character set to be used and is generally much easier on the eye. It also means it should be possible to retain these subtitles when archiving TV recordings to DVD, because DVDs also render subtitles as graphics.

In fact, you can make video DVDs from DVB television recordings without having to convert or re-encode the video, the audio or the subtitles, because the MPEG video and audio streams from the DVB broadcast are also broadly DVD-compliant. This is great because it saves a lot of time and processing power and won't degrade the quality of the original recording. (Technically, the audio should be AC3 for guaranteed compliance; but not all TV channels broadcast an AC3 soundtrack, and I've never met a DVD player that can't handle MP2 audio.) You can also top-and-tail recordings, and strip out commercial breaks, at the same time. You can't, however, put HD recordings on to DVD without re-encoding them into SD; and if the recording is too long to fit on one disc, you'll either have to split it over multiple discs or re-encode it at lower quality. Neither of those topics is covered here.

I'm running Debian Linux (wheezy) and all the tools I've used are in the standard repository or Christian Marillat's wonderful http://deb-multimedia.org/ repository, except IfoEdit, which is Windows-only but runs perfectly under Wine. ProjectX is Java and will run anywhere if you have the Java Runtime Environment installed. All of the steps should be fairly easily achieved under Windows.

I'm assuming you have some means (e.g. a PC TV card, or a PVR-capable satellite receiver box with a USB connector) of recording the transport stream of the broadcast. My Comag SL40HD satellite box, for example, records programmes as a series of TS files, so the first step for me is to join these together into one big TS file using the 'cat' command. Open a terminal and navigate to the directory containing the stream files, and use something like this:

$ cat data0001.ts data0002.ts data0003.ts > ~/Videos/raw.ts

(I believe Windows also has a CAT command that will achieve the same result; you'll need to check the syntax.) This will create a new file called raw.ts, which is the three input files stuck together, and put it in the Videos folder in your home directory. The input files aren't modified. Obviously you can skip this step if the recording was so short it only ran to a single TS file; but even so I would recommend copying the file to a separate working folder. The transport stream file includes literally everything that comprised the original broadcast, including the MPEG video, all the multiple audio streams, all the teletext pages, and, crucially, the so-called subpictures stream holding the subtitles.

The second thing to do is to demultiplex (demux) the streams from the TS file, i.e. separate out the video, audio and subtitles into individual files. You can do this using a complicated but powerful program called ProjectX. Start by going to File > Add... to load your TS file. Next, to tidy the beginning and end, and to remove commercial breaks, go into the 'CutControl' tab and use the slider to move to the beginning of the footage you want to keep. The same button (the film strip with the green plus sign) is used to mark both in and out cut marks; the program works out which are which. The navigation buttons automatically jump to keyframes, which means you can't achieve single-frame accuracy, but this is because it's impossible to manipulate MPEG video between keyframes without re-encoding the video. Besides, keyframes usually come every dozen frames or so, so it's accurate enough.

Click the button marked 'prepare >>' to open the Process window, and select PreSettings > PreSettings. Go  to the subtitle tab and select SUP as the first subtitle export format. You need to select something appropriate in the 'simulate DVB Subpictures IRD color mode' dropdown; I found 'RTLdeu' worked for a recording made from the German Vox channel (which is owned by RTL) while the default four colours option produced illegible output. Now close the PreSettings window, and in the Process window press the big start button to demux the streams.

All being well, ProjectX will create an m2v file from the video stream, an mp2 file from the audio stream, and  a sup file from the subpictures stream (i.e. the subtitles). Now we need to remultiplex (remux) the files, i.e. stick them all back together again in a format your DVD player can read. Note that we're not at any point re-encoding the data, but merely changing the container it sits in. To do this, open up IfoEdit. Select DVD Author > Author new DVD. Use the file chooser buttons in the Video, Audio and Subpicture sections to select the m2v, mp2 and sup files output by ProjectX. Select a suitable, empty folder for the output files in the Output Stream section, then press OK. IfoEdit will now generate BUP, IFO and VOB files which any DVD player should be able to read.

Unless you intend to keep the finished product on a hard drive for playing back on a computer or through a media player, you'll probably want to author a DVD menu. For this step I would recommend Bombono DVD, because I know it's available for Linux and Windows; I know it will retain our hard-won subtitles; and I know it won't reintroduce pauses between multiple VOB files, but will automatically chain them into a single title. Open it up and press the Add Media Files button. Select all of the VOB files generated by IfoEdit, by holding down Ctrl to make multiple file selections, and press Open. When Bombono asks you if you want to run import, say OK. The Select Videos to Import wizard that pops up will offer you one video, but it should be the length of your entire recording, so select it and press Forward. Select an empty folder when it asks you to; Bombono will use it to create a single, temporary VOB file that we won't actually need in the end. Then press Apply and wait for it to import your whole recording.

When it's done, go into the Menu tab and press the Add Menu button. Select the new menu in the list, and press the green button to edit it. At this point it's up to you how much imagination and effort you want to invest, but personally I would just stick some text up saying the name of the film or TV show you've recorded, and some more saying "Play movie". Then right-click on the "Play movie" caption, and add a link to Video01-01. Now right-click on the video in the media list and change the End Action to Menu 1. If you don't do this, your disc will freeze at the end of the show. In the Output tab, select yet another output folder (the default one, in Linux at least, points rather unhelpfully to a hidden folder in .cache in the user's home directory) and choose author mode of Write DVD folder. Press Build DVD-Video and Bombono will assemble for you, in the specified output directory, a folder called 'dvd' which contains the AUDIO_TS and VIDEO_TS folders found on every Video DVD. AUDIO_TS will be empty (it's not used for video DVDs) but the DVD specification says it must exist.

At this point, open the newly-generated VIDEO_TS.IFO file (either the one from Bombono's VIDEO_TS output folder, or the one generated by IfoEdit, if you skipped the menu-generation step) in the VLC media player. You should see your edited recording, with sound and pictures in sync. Let it play for a while through a section where people are talking, and then try to turn on the subtitles by selecting Video > Subtitles Track > Track 1. If your experience matches mine, the subtitles should be displayed, but the colours will be all wrong, or all black, or all green, or otherwise not right. This is because the subpictures colour map is stored in the DVD IFO files and transport streams don't have IFO files. To fix this, open IfoEdit again and press the 'Open' button. Open VTS_01_0.IFO (again, use the one output by Bombono, or from the previous run of IfoEdit if you didn't use Bombono), then in the expanded tree in the top panel under VTS_PGCITI single-click on VTS_PGC_1. Scroll down a bit in the bottom panel and you'll find a block describing a colour map for the 16 colours available, not as red green blue (RGB) but rather in the unhelpful Y Cr Cb format. You can change the three values for each colour by double-clicking on the row to open a hex editor.

I'm afraid this is the one part where you might have to fiddle about a bit. My recording from RTL Germany used only four colours from the palette. Colour 0 was transparent (for superimposing the subpicture on top of the video) so its value could be ignored. Colour 1 was black, for the text shadow. Colour 3 was white, for the text, and colour 2 was grey, for anti-aliasing (smoothing off) the edges. These colours in hexadecimal Y Cr Cb format are as follows: black is 00 80 80; grey is 78 80 80; and white is F0 80 80. I would suggest you try setting these three colours first.

While you're at it, highlight the tree root VTS_01_0.IFO file in the top panel, and double-click in the lower panel on the row for Audio 1. Select the correct language in the Language dropdown, and change the Language type to 'present'. Next, double-click on the row for SubPicture 1 and again choose the correct language and change the Language type to 'present'. This will allow the DVD player to label the audio and subtitle languages correctly, and will mean you won't have to wait till someone actually speaks to be able to turn the subtitles on. Finally, press the Save button (and click Yes to save the BUP file too) and try the video again in VLC. If you don't get the results you're after, you may need to try setting some of the other colours. You may also need to look up how to generate the Y Cr Cb values for different colours from RGB values; I used an online calculator tool. Don't forget you'll need to enter the final values in hexadecimal. Google can convert decimal to hex for you (just google "240 in hex" or whatever).

If you want to burn an actual physical disc, you need to create an ISO9660/UDF filesystem disc image containing your AUDIO_TS and VIDEO_TS folders. I'm sure Windows has lots of nice programs for doing this; some of them are probably free. Under Linux I use K3b. It's designed for KDE but it runs fine under Xfce. Tell it you want to create a new Video DVD project, and it will automatically choose the right filesystem and generate empty AUDIO_TS and VIDEO_TS folders for you. Drag all the contents of the VIDEO_TS folder that Bombono generated into the VIDEO_TS folder on the disc, and press Burn. In the Writing tab, tell K3b to create an image, and only create an image (i.e. don't burn an actual disc). In the Image tab, you can tell it where to put it (it will be a single ISO file), and in the Filesystem tab you can give the disc a name, which by convention should be all in capitals. Press Start.

The easiest and most reliable way to burn an ISO image on to a physical DVD is from the command line. Put a recordable or rewritable DVD in the drive and open a terminal. Navigate to the directory holding the new image file and run the command:

$ growisofs -dvd-compat -Z /dev/sr0=JUNO.iso

but replace /dev/sr0, if you need to, with the pointer to your DVD burner drive, and JUNO.iso with whatever your image is called. (Actually I doubt I need to tell you this if you've made it this far.) When the disc is finished, eject it and put it back in. Find out the size of the ISO file in bytes by running

$ ls -l JUNO.iso

and verify that the image is identical to the freshly burned disc by running

$ cmp -n 2642817024 /dev/sr0 JUNO.iso

but replace that long number with the size of the ISO file in bytes. The cmp (compare) command will take a while to run, because it has to compare every byte on the DVD against every byte of the image on your hard drive, but it should finish without printing any message at all. It will only output anything if it finds a discrepancy between the disc and the image.