HTTP PUT with multipart/form-data using pycurl

Let’s suppose you have a REST interface to talk to, and there’s a PUT request you want to make, sending data over using the multipart/form-data encoding (as opposed to application/x-www-form-urlencoded). If you’re using Python and pycurl, you’ll find out that if you try to combine setopt(pycurl.PUT, 1) with setopt(pycurl.HTTPPOST, [ (key1, val1), … ]), it doesn’t work. You could try to use setopt(pycurl.POSTFIELDS, “…”), but you’d have to handle encoding to multipart/form-data by hand, or use a third party library such as poster. But in any case it looks like more hassle than it should. The pycurl.HTTPPOST option can already do what’s needed, it’s just that it implies the POST method, while you want to use PUT.

A solution came to me when reading a thread on the curl-with-python mailing list. I knew I could already do what I needed using the command line utility, like this:

curl -X PUT -F 'fieldname=@filename.json' http://localhost:8000/

If you add an option like --libcurl foo.c to such call, you’ll get a C program which does what your command line invocation would do. This revealed, that “-X PUT” did not translate into setopt(pycurl.PUT, 1), but into setopt(pycurl.CUSTOMREQUEST, “PUT”). It might look like a subtle difference, but the latter does what I wanted, while the former doesn’t. A minimal working example would look like this:

import pycurl

c = pycurl.Curl()
c.setopt(pycurl.URL, "http://localhost:8000")
c.setopt(pycurl.HTTPPOST, [('foo', 'bar')])
c.setopt(pycurl.CUSTOMREQUEST, "PUT")

If you run “nc -l 8000” and run the above code, you’ll see:

PUT / HTTP/1.1
User-Agent: PycURL/7.26.0
Host: localhost:8000
Accept: */*
Content-Length: 141
Expect: 100-continue
Content-Type: multipart/form-data; boundary=----------------------------2def70e0b37a

Content-Disposition: form-data; name="foo"


…which is exactly what I wanted.

Canon XM2 (DV) to DVD, on Linux

I wanted to transfer some material from DV cassettes to DVD. My main workstation is running Ubuntu 12.04, and I decided to use the tools that are available with the distribution. I tried multiple ways of doing each of the tasks, and git many dead ends, mainly due to crashing programs, bugs, or incompatible tools. For instance, tovid looked very promising until it turned out that it is not compatible with the new version of the ffmpeg utility. My source material was DV, recorded by Canon XM2, the video format was 768×576, interlaced (576i), with audio at 48kHz, PCM, stereo. Interlacing was giving me some headache, because the first attempts lead to unsightly stripey output. The camera outputs double-scan interlace, which should be interpreted as 50 frames per second with reduced resolution. Interlacing might be tricky

The first step is to capture the video from the camera. Connect the camera to the laptop, switch the camera to the playback mode, rewind the tape and:

dvgrab birthday-

The “birthday-” bit is a prefix that will be added to the saved .dv files. dvgrab will save multiple 1GB files, each file about 4 minutes long. Once the material is captured, you can merge the multiple files into one, by simply concatenating them:

cat birthday-001.dv birthday-002.dv birthday-003.dv > birthday.dv

Once you have one file with the complete material, fire off a player and note down (I used paper and pencil) the times of segments you want to extract. You won’t be able to do a lot of cutting that way, but if it’s a couple of segments, it shouldn’t be too labor intensive. Once you know what are the segments you want to extract, you can extract them and encode as .vob files. Suppose one fragment starts at 02:13 and is 135 seconds long:

avconv -i birthday.dv -target pal-dvd -flags +ilme+ildct -b:v 6000k -ss 02:13 -t 135 birthday-01.vob

The “+ilme+ildct” bit is responsible for correct handling of interlacing, because DV uses different field order than DVD. Repeat the above command for each segment, and you’ll get a list of VOB files. These VOB files are DVD compliant, and they are implementing the interlace correctly. They must not be re-encoded when transferred to DVD, otherwise the interlacing settings will be most likely lost. You can try if your interlacing settings are correct by watching the VOB file using VLC with automatic deinterlace detection:

vlc --deinterlace -1 --deinterlace-mode bob --play-and-exit birthday-01.vob

You should see no stripes during movement in the video, and the displayed frame rate should be 50fps (although the video frame rate is set to 25fps).

The next step is to create a DVD menu. There is a number of DVD authoring software. I had most success with DVD Styler. I also tried tovid, and Bombono.

In DVD Styler, I managed to create a DVD directory structure, but not an ISO image, and I was not able to burn a DVD directly from DVD Styler. Instead, I only generated the DVD structure on disk, and used k3b, using its DVD template. I created a new project, found the generated VIDEO_TS directory from DVD Styler, and added it to the project in k3b. This was enough to arrive at a working DVD.

DVD Styler would recognize that the files are already DVD compatible and did not attempt to re-encode them.

The above method is rather basic and crude, but gets the job done. There isn’t a video editor used at any stage; instead we just note down the times and then extract time regions using the -ss and -t options of avconv. I tried to use pitivi for video editing, but there were issues with rendered video, and since I didn’t really need any editing, I dropped pitivi from the workflow. The main problem to solve in pitivi would be to encode a DVD compliant VOB video file. You can select a DVD VOB as the output format, but there’s still a lot of things you can mess up, for instance accidentally encode audio in 44.1kHz instead of 48kHz, which results in a DVD disc with no audio.

I suspect that tovid will be reasonably soon adapted for use with the new ffmpeg tools (using /usr/bin/avconv instead of /usr/bin/ffmpeg), which will make it easier to script out the process if I had more of such (e.g. archival) DVDs to make.

Driver Theory Test under Linux

Linux is not one of the platforms supported by the Irish Driver Theory Test software, but I decided to give it a go under Wine. It did not work, but not because of a problem in Wine (at least not directly), but because of some sort of a copy protection in DTT. For some reason the DTT software thinks that the CD is a virtual CD, even though it’s a physical disc inserted into the physical drive.

The dialog window says:

This CD is probably a “virtual CD” or a counterfeit copy (which may have been sold or given to you illegally). Pirated software damages the industry, may not contain the most recent questions and will be not supported by us. If you have inadvertently purchased a pirated copy then you should return it to the seller demanding a full refund. We would appreciate notification at: todtsu…

Perhaps the book store sold me an illegal copy, but it looks genuine. I’m wondering why is DTT thinking that the CD is not a physical disk.

PS3 and Matroska with subtitles

Suppose you have a video file with subtitles, in the Matroska format.  How to play it on a PS3?  Probably set up mediatomb or ps3mediaserver and… I’ve made a number of attempts, and each had certain problems.  Here’s the list of ideas I had:

  • Try to just play .mkv with PS3
    …won’t work because PS3 doesn’t support Matroska.
  • Use ps3mediaserver to recode .mkv on the fly
    …won’t work because ps3mediaserver doesn’t hardsub (doesn’t embed the subtitles in the image).
  • Use mencoder to create an .avi file with embedded subtitles (using mpeg4 for video and mp3 for audio)
    …in theory should work, but my PS3 said it’s an “unsupported format”, no idea why.
  • Use mencoder to create an .mp4 file with embedded subtitles
    …doesn’t work, because mencoder’s mp4 support is broken, causing audio and video to go out of sync.
  • Use mencoder to create an .avi file with embedded subtitles (using mpeg4 for video and mp3 for audio), and use ffmpeg to convert it (without recoding) into MP4
    …doesn’t work, because PS3 won’t play an .mp4 with mp3 audio
  • Use mencoder to create an .avi file with AAC audio (using faac), and use ffmpeg to convert it to MP4
    …does work if you pass -faacopts mpeg=4:raw=1 options to mencoder, but there’s still a problem with a/v going out of sync
  • Add the -mc 0 option
    …fixes the a/v sync problem in most cases, but there’s still the problem with the video being crap quality
  • Use x264 to encode video
    …gives the best results:

I’ve also searched the web to look for encoder options and I came up with the following recipe.  It requires mencoder with x64 and faac support, and ffmpeg.  The following example uses Japanese audio track and English subtitles.  It leaves the intermediate .avi file. The first steps are to set the file names and recode the video and audio content, saving it in an .avi file.

mencoder -o "${outfile}.avi" \
   -oac faac \
   -faacopts raw=1:mpeg=4 \
   -ovc x264 \
   -x264encopts bitrate=3000:pass=1:threads=auto \
   -mc 0 \
   -vf pullup,softskip,harddup \
   -alang jpn \
   -slang eng \

Repeat that with pass=2. After the second pass has completed, it’s time to move the encoded content from the AVI container to the MP4 container.

ffmpeg \
   -acodec copy \
   -vcodec copy \
   -i "${outfile}.avi" \

The resulting MP4 file should be playable on PS3. This example doesn’t contain any further automation. I’m leaving it to you. Other possible improvement include skipping the audio recoding, if audio is in AAC already.  There’s also a lot of potential for tweaking the x264 options for better video quality. I’ve decided to leave these out for now to keep the example as simple as possible.

Please leave a comment whether it worked for you or not; suggestions for updates and improvements are also appreciated.


File changed as we read it

I’ve tracked it down, so you don’t have to.

When creating an archive, GNU tar monitors the file it’s reading. If the file’s mtime has changed, it’s reporting that “file changed as we read it”, as defined in create.c in the source code.

Tar’s man page doesn’t say anything about tar system exit codes, but a relevant piece of documentation can be found on-line.

Depending on tar version, it returns different exit codes.

  • tar-1.13 and earlier: 1 or 2
  • tar-1.14: == 0
  • tar-1.15: == 0
  • tar-1.17: == 1 (change)
  • tar-1.19: == 1

The change was recent enough that as of November 2008, you can be still encountering both behaviors.

Bridge and ipv6 autoconfiguration

“There’s no sound”, said my missus.

Few minutes before, as usual, I picked up my 17-inch laptop and placed it on the second desk, where it can face the room. I don’t really like dragging the armchair every time I want to watch a video together, but the flat isn’t big enough to arrange a place with a permanent setup. (I also don’t own a TV and I’m proud of it.)

This time there was indeed no sound.

“Wait a minute”, I said and opened Gnome Terminal. At my place, there is only one set of speakers connected to one computer, and shared over the local area network. After a while I figured out that mplayer had trouble contacting PulseAudio server. The server wouldn’t respond.

I tried playing sound from a different machine – it worked. So it’s not the server. Next test:

maciej@quince ~ $ ping -c 1 leon
PING ( 56(84) bytes of data.
64 bytes from ( icmp_seq=1 ttl=64 time=0.165 ms

— ping statistics —
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.165/0.165/0.165/0.000 ms

Next test…

maciej@quince ~ $ telnet leon 4713
Trying 2001:

Timeout. And… ah. I forgot I was running native IPv6 in my LAN. I tried connecting over IPv4, it worked. I’ve set ${PULSE_SERVER} to the IPv4 address of the PulseAudio server and happily watched the movie together with my missus.

But… what’s going on? Running ping6 gives good results – there are answers. But when I ran ping6 leon, I got no answer. How can ping work for an external host but not for a host in my LAN?

After more poking at the machine, I’ve found out that the response packets (ACK) over IPv6 are in fact coming to the slave eth0 interface, but never to the bridge interface, br0.

Then I’ve noticed that IPv6 routing is wrong! There are two routes:

2001:660:952::/64 dev eth0
2001:660:952::/64 dev br0

Packets should leave through br0, not eth0. First idea, restart the bridge. However, the wrong route stayed there even after interface restart. It made me think what happens when I stop the bridge.

Well, eth0 goes back from slave state into a normal state, and then… it’s autoconfigured. That means, it gets a route assigned to it. When the bridge is being started, eth0 becomes a slave of br0, but there probably is no mechanism to remove the route which was assigned during eth0’s autoconfiguration.

A quick ad-hoc fix is:

ip -6 route del 2001:660:952::/64 dev eth0

A permanent solution would be better, but I happen to be lazy.

UPDATE 2009-02-01: Jeroen Massar from SixXS offered a better workaround:
put in /etc/sysctl.conf:
net.ipv6.conf.eth0.accept_ra = 0

When talking about this problem, the conversation usually goes like this:
– I have this problem with IPv6 autoconfiguration… (yada yada)
– What distribution are you using?
– Gentoo.
– (laughter)

Well, turns out, they were right. I tried this config on Ubuntu:

maciej@ubuntu-vbox:~$ cat /etc/network/interfaces
auto lo
iface lo inet loopback
iface br0 inet dhcp
bridge_ports eth0
auto eth0
auto br0

After “/etc/init.d/network restart”, the surplus route was there. But after a reboot, it wasn’t there any more. Ubuntu/Debian somehow handles this case.

Training notes from Atlantic Linux

Atlantic linux writes:

We started with an Introduction to Linux course after finding some interested customers around the Galway area. (…) we decided to release our training materials under a Creative Commons license (…)

– Introduction to Linux
– Linux System Administration and Support
Linux Performance Tuning

Full post and training materials