OpenCSW provides a software stack which allows to build catalogs with Solaris packages, and use a utility, such as pkg-get or pkgutil to download and install packages together with all the dependencies.  It works very nice for OpenCSW packages, but what if one wants to install a SUNW, that is, a Sun-provided package?  Solaris DVDs are not in the OpenCSW format and pkgutil can’t use them directly.

I wrote a tool which converts a collection of Sun packages in the directory format (they’re distributed this way) into a set of gzipped srv4 format files (.pkg.gz) which can be served over http or ftp and downloaded via pkgutil.

To get started:

1. Install pkgutil and pkgutilplus from OpenCSW
2. Check out the OpenCSW utilities:
   svn co https://opencsw.svn.sourceforge.net/svnroot/opencsw/utilities/
3. Have a Solaris DVD available in your filesystem
4. Patch /opt/csw/bin/bldcat if necessary (the patch is there at the end of this post)
5. Run the package converter:
   ./build_sun_catalog.py -p “/somewhere/Solaris_10/Product” -c “/target/directory/sparc/5.10″
6. Run bldcat and create the catalog:
   (cd /target/directory/sparc/5.10; bldcat .)
7. Server the target directory via http under, say http://packages.example.com/solaris-10u5
8. From your Solaris machine, use the following command to install a package together with all the dependencies:
   pkgutil -t http://packages.example.com/solaris-10u5 -i SUNWfoo

A word of warning though: upgrading SUNW packages can break your system! If you attempt to upgrade a core Solaris package such as SUNWcakr or SUNWkvm.  If pkgutil tries to do that, use the -x option to filter out those packages. For example:

pkgutil -t http://packages.example.com/solaris-10u5 -i SUNWfoo -x SUNWcakr -x SUNWcar -x SUNWkvm

--- bldcat.orig 2009-11-23 13:45:58.436220417 +0000
+++ bldcat      2009-11-23 13:42:27.098746854 +0000
@@ -100,3 +100,3 @@
foreach (glob "$tmpdir/*") {
-    ($pkgname) = (/([\w\+-]+)$/) if (-d);
+    ($pkgname) = (/([\w\+-\.]+)$/) if (-d);
}

Executive summary: OpenCSW rocks. http://ftp.heanet.ie/pub/opencsw/ (and the like) is the mirror setting you want. (See the README files for more mirrors.)

Solaris 10, as installed from a DVD, has a very small number of software packages. It has packages that are relevant to the OS itself, but it does not have packages with things such as Apache or PostgreSQL vim, or subversion client, or GNU utilities (awk, grep, etc). These ones you need to either compile yourself or use a precompiled binary package. A project called blastwave used to provide those for quite a number of years. However, sometime in August 2008, “things happened”. What those things are, I’ve decided I don’t want to know. The current state of affairs is that there are 2 (two) projects now, one is called “blastwave” and one is “opencsw”. I did a bit of research on the web to find out about their source repositories and binary package statuses.

One thing that struck me pretty strong is that none of those projects advertises its code repository. With open source projects, the code repository is usually the first thing published. “We have a new project! Here’s our code: http://svn.example.org!” That’s what the phrase “open source” means, according to a common sense understanding. If you’re interested in the project, you can download the source, tinker with it and if you come up with something useful, you offer your change in a form of a patch. Without the source code, you can’t do that. So, where’s the source?

Finding the source code of OpenCSW was a bit harder than the source code of Blastwave. Here it is:

http://gar.svn.sourceforge.net/viewvc/gar/csw/mgar/pkg/

To check out the source (takes a long time to complete):

svn co --ignore-externals https://gar.svn.sourceforge.net/svnroot/gar/csw/mgar opencsw

The code repository, looking at the statistics page, looks pretty active. There’s also a wiki page with instructions.

If you want to update a package or change something in a package from CSW, or you’re wondering how does package building work there, check out this source, then change into a directory with a package, like cups/trunk and type ‘gmake’. It works in a similar fashion to BSD ports.

I didn’t do full exhaustive stats, but it seems that packages in OpenCSW are generally much more recent than the ones from Blastwave. So, check your pkg-get.conf (yes, it’s a newer version of or pkgutil.conf if you use this alternative to pkg-get) and replace the old blastwave mirror with OpenCSW one, and see the goodness flowing in.

(updated on 2009-03-11 according to Phil’s comments)

Few months ago, I’ve decided to try out a Xen virtual server service in a German provider, xencon. Then I decided I had a better solution (also Xen-based), and I don’t need xencon any more. However, it turns out that it’s not so easy to cancel the service. I had similar problems with two telecom providers in Poland. I said I want to cancel the service and they kept on billing me.

Dreamhost however, is awesome: if you want to, or need to, you can actually close your Dreamhost account on-line! They kindly ask you “Please, stay!”, but you have a button to press, stating that you really want to close your account.

This is by itself an excellent reason to open an account there.

I’ve recently had to move a few databases from MySQL 4.x to MySQL 5.x. One of the most important differences is that the 5.x family understands character encodings. Not exactly fresh news, version 5.0 was issued in 2003, but there is still a lot of 4.x installations around.

MySQL 5.0 no longer happily accepts any byte string into a VARCHAR or TEXT field. It stores encoding names as part of the table structure, and converts between encodings when necessary. MediaWiki or WordPress, when run on MySQL 4.x, store data in UTF-8, but the database itself doesn’t “know” about it. Everything seems fine, until you dump your database to a file and load it into MySQL 5.0 (or above). What happens, is that your text is considered to be latin1 (a.k.a. ISO-8859-1). If you happen to have any non-English characters as, say, article names in MediaWiki, you’re going to end up with an error message such as:

A database query syntax error has occurred. This may indicate a bug in the software. The last attempted database query was:

(SQL query hidden)

from within function “Article::pageData”. MySQL returned error “1267: Illegal mix of collations (latin1_bin,IMPLICIT) and (utf8_general_ci,COERCIBLE) for operation ‘=’ (localhost)”.

To fix the problem, you need to tell MySQL that your text data is really UTF-8, not latin1. You need to find all the columns of type VARCHAR or TEXT, and modify them to have UTF-8 character set. For example, if your column is VARCHAR(255), you can execute this statement:

ALTER TABLE your_table
MODIFY COLUMN your_column VARCHAR(255)
CHARACTER SET utf8
COLLATE utf8_bin;

(The utf8_bin collation is needed to keep your sorting case-sensitive.)

However, MySQL will convert your text from latin1 to UTF-8, and your text will still appear “wrong”. You can fix it in one more step. The problem is that you had UTF-8 taken to be latin1 and then stored as UTF-8. To fix this, you need to “convert” your text from UTF-8 back to latin1, and then make MySQL take it as UTF-8, but, importantly, without converting it. This can be achieved by temporarily casting your data to binary — this operation doesn’t trigger encoding changes. You can then cast your data into any encoding you want. In a nutshell, you need to go: UTF-8 →(conversion)→ latin1 → binary → UTF-8.

UPDATE your_table
SET your_column = CONVERT(
CONVERT(
CONVERT(
your_column
USING latin1
)
USING binary
)
USING utf8
);

It may take you a while to understand it. If you want to get a better feel of what’s going on, consider the following, equivalent example in shell. Let’s assume you’re using Polish diacritics: ąćęłńóśżź. Your Polish page title “Café” might in a garbled form look something like this:

Café

Assuming your system is natively UTF-8 (most of modern Linux distributions are), an easy way to simulate text garbling is the following shell expression.

echo Café \
| iconv -f utf-8 -t utf-8 \
| iconv -f latin1 -t utf-8
Café

Converting from UTF-8 to UTF-8 seems stupid, but I wanted it to be very clear: what we have here, is an UTF-8 string, as output by the second line, taken to be latin1 in the third line. This is how your text can become garbled. An obvious way to fix it, is to run the process backwards:

echo Café \
| iconv -f utf-8 -t latin1 \
| iconv -f utf-8 -t utf-8
Café

Again, the UTF-8 to UTF-8 conversion is preserved to make this crucial point explicit. Your string was converted to latin1, and then taken to be UTF-8.

Back to our problem. We know how to convert the data, but we need to find all the tables and columns taht need converting. Conveniently, MySQL offers an “information_schema” database, which allows us to read information about MySQL tables. It’s enough to run this query to find all the tables of interest:

SELECT
table_name,
column_name,
column_type,
character_set_name
FROM
columns
WHERE
table_schema = ‘your_table’
AND
(
data_type = ‘varchar’
OR
data_type = ‘text’
)
AND
character_set_name != ‘utf8′
;

If you’re really lazy, as I am, your data has exactly this problem (UTF-8 taken to be latni1 and looking garbled), you can use a Python script I wrote. But please be careful! You’re using it at your own risk! Backup your database first! If this script damages your database and you lose all your data, it’s your problem, not mine. You have been warned. Here’s the script.

Note: the script will only print SQL statements to screen. You have to execute them youself.

I’ve tracked it down, so you don’t have to.

When creating an archive, GNU tar monitors the file it’s reading. If the file’s mtime has changed, it’s reporting that “file changed as we read it”, as defined in create.c in the source code.

Tar’s man page doesn’t say anything about tar system exit codes, but a relevant piece of documentation can be found on-line.

Depending on tar version, it returns different exit codes.

• tar-1.13 and earlier: 1 or 2
• tar-1.14: == 0
• tar-1.15: == 0
• tar-1.17: == 1 (change)
• tar-1.19: == 1

The change was recent enough that as of November 2008, you can be still encountering both behaviors.

I’ve recently checked in changes to django-phpbb, my Django-phpBB integration project. It’s now possible to authenticate users against unmodified phpBB database. I’ve also added installation instructions.

Current focus will be on removing the parts that are specific to my project and making django-phpbb more generic.

“There’s no sound”, said my missus.

Few minutes before, as usual, I picked up my 17-inch laptop and placed it on the second desk, where it can face the room. I don’t really like dragging the armchair every time I want to watch a video together, but the flat isn’t big enough to arrange a place with a permanent setup. (I also don’t own a TV and I’m proud of it.)

This time there was indeed no sound.

“Wait a minute”, I said and opened Gnome Terminal. At my place, there is only one set of speakers connected to one computer, and shared over the local area network. After a while I figured out that mplayer had trouble contacting PulseAudio server. The server wouldn’t respond.

I tried playing sound from a different machine – it worked. So it’s not the server. Next test:

maciej@quince ~ $ ping -c 1 leon
PING leon.home.blizinski.pl (192.168.1.3) 56(84) bytes of data.
64 bytes from leon.home.blizinski.pl (192.168.1.3): icmp_seq=1 ttl=64 time=0.165 ms

— leon.home.blizinski.pl ping statistics —
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.165/0.165/0.165/0.000 ms

Next test…

maciej@quince ~ $ telnet leon 4713
Trying 2001:660:952::20a:e4ff:fe28:a3cf...


Timeout. And… ah. I forgot I was running native IPv6 in my LAN. I tried connecting over IPv4, it worked. I’ve set ${PULSE_SERVER} to the IPv4 address of the PulseAudio server and happily watched the movie together with my missus.

But… what’s going on? Running ping6 ipv6.google.com gives good results – there are answers. But when I ran ping6 leon, I got no answer. How can ping work for an external host but not for a host in my LAN?

After more poking at the machine, I’ve found out that the response packets (ACK) over IPv6 are in fact coming to the slave eth0 interface, but never to the bridge interface, br0.

Then I’ve noticed that IPv6 routing is wrong! There are two routes:

2001:660:952::/64 dev eth0
2001:660:952::/64 dev br0

Packets should leave through br0, not eth0. First idea, restart the bridge. However, the wrong route stayed there even after interface restart. It made me think what happens when I stop the bridge.

Well, eth0 goes back from slave state into a normal state, and then… it’s autoconfigured. That means, it gets a route assigned to it. When the bridge is being started, eth0 becomes a slave of br0, but there probably is no mechanism to remove the route which was assigned during eth0’s autoconfiguration.

A quick ad-hoc fix is:

ip -6 route del 2001:660:952::/64 dev eth0

A permanent solution would be better, but I happen to be lazy.

UPDATE 2009-02-01: Jeroen Massar from SixXS offered a better workaround:
put in /etc/sysctl.conf:
net.ipv6.conf.eth0.accept_ra = 0

When talking about this problem, the conversation usually goes like this:
- I have this problem with IPv6 autoconfiguration… (yada yada)
- What distribution are you using?
- Gentoo.
- (laughter)

Well, turns out, they were right. I tried this config on Ubuntu:

maciej@ubuntu-vbox:~$ cat /etc/network/interfaces
auto lo
iface lo inet loopback
 
iface br0 inet dhcp
bridge_ports eth0
 
auto eth0
auto br0

After “/etc/init.d/network restart”, the surplus route was there. But after a reboot, it wasn’t there any more. Ubuntu/Debian somehow handles this case.

Atlantic linux writes:

We started with an Introduction to Linux course after finding some interested customers around the Galway area. (…) we decided to release our training materials under a Creative Commons license (…)

- Introduction to Linux
- Linux System Administration and Support
- Linux Performance Tuning

Full post and training materials

Suppose you use ZFS and have pool/foo mounted at /foo.

zfs create -o mountpoint=/foo pool/foo

Suppose you’ve created a snapshot:

zfs snapshot pool/foo@bar

Now they tell you that you can access it inside .zfs/snapshot directory. However, if you go into /foo and type ls -la (option ‘a’ means you want to see files and directories beginning with a dot), you don’t see the .zfs directory. You’ll see ‘.’ and ‘..’ and all others, but not ‘.zfs’. However, an innocent ‘cd .zfs’ or ‘cd /foo/.zfs’ will get you there. A truly hidden directory indeed.

What was it called at esr’s The Art of Unix Programming? The Principle of Least Surprise?

My home network has native IPv6 with radvd running on the router. Any host connected to it automatically gets an IPv6 address. (Unless it’s a Solaris zone, but that’s another story.) For a reason I haven’t worked out yet, all IPv4 traffic on my downlink seems to be throttled to 60kB/s. However, when downloading over IPv6, I’m easily getting 150kB/s or 200kB/s.

When setting up a Solaris 10 system (this time using kvm), I’m getting an IPv6 address straight away:

bash-3.00# ifconfig -a6
lo0: flags=2002000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv6,VIRTUAL> mtu 8252 index 1
inet6 ::1/128
rtls0: flags=2000841<UP,RUNNING,MULTICAST,IPv6> mtu 1500 index 2
inet6 fe80::5053:ff:fe13:9457/10
ether 52:53:0:13:94:57
rtls0:1: flags=2080841<UP,RUNNING,MULTICAST,ADDRCONF,IPv6> mtu 1500 index 2
inet6 2001:770:18d:0:5053:ff:fe13:9457/64

(the ‘rtls0′ interface is a name for a rtl8139 NIC)

I want to use an IPv6-enabled OpenCSW mirror! Such as ftp.heanet.ie.

You may want that too. Edit /etc/wgetrc (or /opt/csw/etc/wgetrc if you’re using OpenCSW wget) in your Solaris system and add a line:

prefer-family = IPv6

…and…

bash-3.00# pkg-get -U
WARNING: gpg not found
Getting catalog...
--20:58:52--  http://ftp.heanet.ie/pub/blastwave/unstable/i386/5.10/catalog
=> `catalog'
Resolving ftp.heanet.ie... 2001:770:18:aa40::c101:c140, 193.1.193.64
Connecting to ftp.heanet.ie|2001:770:18:aa40::c101:c140|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 357,696 (349K) [text/plain]

100%[==============>] 357,696      452.07K/s

20:58:53 (451.18 KB/s) - `catalog' saved [357696/357696]

wget will now try to use IPv6 where available. When IPv6-enabled site is not there (no AAAA record, for instance), it will fall back to IPv4.

UPDATE: Putting network interface in a (VirtualBox 1.6.4) virtual machine into promiscuous mode does in fact make the interface accept icmp6 traffic.

UPDATE (2009-03-11): Added information about OpenCSW wget configuration path.