How Can I Explain This?

A warm and powerful time in unity

2011-07-25T21:46:00.007+02:00

Occasionally, I have faith in my fellow Norwegians.

This afternoon was one of those times.

As international media will tell you, this afternoon was very special for those of us living in Oslo.

It started with a Facebook event, that rapidly spread via national media, friends talking and whatnot. If it hadn't been a planned event, it would have been a very large flash mob.

It was with some trepidation that my colleague and I walked towards Rådhusplassen (the city hall square), as we well knew that a gathering of tens of thousands of people would be a very decent terrorist target.

The general idea was that people would gather there, carrying roses, and then walk peacefully to somewhere else (I've forgotten where). The plan did not survive. One of the major suppliers of plants and flowers said they would sponsor free roses. Every florist was out of flowers in no time, except for potted ones, it looked almost like locusts had passed by.

As we walked on, the crowd density increased, until it exceeded anything I have ever experienced. We Norwegians are very fond of going downtown for our constitution day May 17, or supporting events like the Nordic World Ski Championships or the Winter Olympics in 1994, and the crowds are enormous for a small country such as ours. Those days and events cannot compare.

The sea of flowers, the human warmth and compassion, the sense of unity of purpose – it was almost overpowering, and when Maria Mena started singing Ole Paus's Mitt lille land, I let the tears flow. At the end, we were asked to leave slowly, take care of eachother, and take our flowers – which the arrangers had planned to be placed in a single, potently meaningful place – and leave them all over Oslo.

If you weren't there, I wish you were.

Christmas Star

2010-12-24T14:04:00.005+01:00

I grew up with my parents, the only child.

I also grew up with friends, and had very tight and warm family ties with my father's sister's children - my cousins, my brothers in spirit.

About twenty years ago, my father's health had declined so much that he could hardly work. He had recently been laid off, as a part of cutbacks just before his employer went bankrupt. Physical labour was hard on him, and we, his family, did not understand why, how, or how much. He received unemployment benefits as long as that was permissible, and then he got a job with an old friend, who owned a technical clothing store. That did not work out very well, because it was really tough on father to walk among aisles and restocking clothes. He even fainted once.

The late eighties and nineties were wondrous times in many ways, but the medical profession had hardly heard about ME or COLD/COPD as we know these classifications today. Getting medical recognition that my father was effectively unable to work was almost impossible, particularly considering that he – as so many from his generation – was a smoker. If you have any kind of problems with general shortness of breath, and a doctor asks you, "do you smoke", answering "yes" was then (and apparently still is) a very, very bad idea. The smoking then becomes the only focus, and the rest of the patient's problems seem pretty irrelevant.

But my father did get his disability pension.

In 1994, his father - my grandfather - fell in the snow and froze to death. He had been a widower for five years. Father and his sister inherited a small farm house with a little bit of land, in Vang in Hedmark. My parents sold our house in Oslo, bought my aunt's share, and moved. I moved to an apartment in a more central location in Oslo.

It was strange to not see my parents every day anymore, but it was also a sort of relief - a young grown-up probably benefits from moving, and I think I did. I certainly felt it was over due. :) But I kept regularly in touch with my parents, visited them every month or so, and we continued spending Christmas together every year, with my surviving grandmother, as we had for every Christmas since I was born in 1972.

Through the years, my father's health kept declining, and he became less and less active. Traveling became a chore.

In 2001, a very good friend of mine was diagnosed with particularly aggressive melanoma, but there was hope.

Around the same time, my father's health was so bad that he hardly walked around, and he didn't drive a car anymore because he thought it was irresponsible and unsafe. My parents sold the farm house and the land in 2002, and moved to an apartment in central Hamar. The apartment was fairly well suited for the elderly and disabled, and it saved my mother lots of house and garden work - which of course my father could take no part in.

My friend's cancer spread.

I still celebrated Christmas with my parents, every after grandmother died in 2003. The same year, my friend collapsed in my arms in a seemingly epileptic fit. That was a brain tumour. The same spring, they discovered that the cancer had spread to his colon. A few friends and I were visiting him a fine May day, when he suddenly panicked, started hyperventilating, pressed his pain pump's button repeatedly and pulled the alarm string. We were ushered out by the nurses. Five minutes later, he died, shortly before his family came to see him.

A couple of years later, my father was completely dependent on an electrical wheelchair, a chair or a bed - standing unaided was a risky affair. My mother and father rarely left Hamar, socially or otherwise. This was a pretty heavy burden for us to bear, my parents were both socially active before, and had many good friends. My mother was diagnosed with operable breast cancer in 2005, and the tumour and lymph cysts were surgically removed within a couple of weeks.

I had started sleeping with my phone next to my pillow.

We still celebrated Christmas together, and in 2007, my father took a chance, and visited his sister in Oslo for her 70th birthday. It was a wonderful day.

I never went anywhere without my cellphone, and made sure that I had the wireless house phone with me even in the bathroom.

In the summer of 2008, my father's brother-in-law, my uncle, was admitted to hospital because of what he thought was severe constipation. It wasn't, it was cancer. I visited him about half-way into his first week in hospital, and he was already so weak that he couldn't sit up and had to wear an oxygen mask, but he was fairly lucid. But he looked no healthier than my friend did just before he died, and right enough, a week later, my uncle was dead. My aunt and my cousins still came for my father's 70th birthday shortly afterwards, and it was a very fine celebration, despite that I had to drive my father home early because he was too tired.

My father went through a lot in those later years. He broke his hip from a fall indoors, and he needed extensive home care. He was frequently in the hospital for emergency care. I started having trouble sleeping, and was constantly worried for The Phone Call.

Late in 2008, my father was admitted to palliative care at Prestrud in Hamar, it was now too hard and difficult to care for him at home. My father had probably already suffered brain damage from lack of oxygen - which was untreated, because, hey, he was a smoker, and he just couldn't stop smoking - his withdrawal symptoms were too bad. And, hey, if you don't stop smoking, you don't get proper medical care.

My mother and I still hoped that we could celebrate Christmas in their apartment, but it was soon obvious that it would be impossible. So we prepared to celebrate part of Christmas with him at Prestrud.

I recall that Christmas Eve more vividly than most. My mother and I followed most of the old routine. The TV was on, and the traditional movie "The Journey to the Christmas Star" was showing as it always is. A character, Sonja, sings the song "Christmas Star, come to me", and I start crying.

We visited father at Prestrud, had Christmas cookies, biscuits, hot chocolate with cream and a good time, all considering.

A day in February, my mother called me at work. Father had pneumonia. I told my boss that I had to go, and went home to pack. Shortly afterwards, my mother called again; father was dead.

It was a strange funeral. Warm and melancholic, a wonderful priest - Helge Hognestad - who had an exceptional grasp of the humanity of it all. The first part of the funeral was tough going for me, but then I held my speech for father, and talked about many of the good things and times that I associate with him. Carrying his casket was tough, too, but when the hearse drove off, I was so relieved that I couldn't stop smiling.

Father's suffering was over. Mother's suffering was over. My suffering was over, too.

But then came the double whammy: in late spring, mother was diagnosed with pancreatic cancer, the kind which has seen an amazing increase in surviving patients, then at an estimated whopping 5%. It was a very hard blow, but then my mother and I started talking about it, and she decided to live her life in full, for however long that was.

In spite of awful side effects from chemo therapy, in spite of bloody vomit and other unsavory details, she kept on going. We went traveling together twice, my mother went alone or with others several times. She sometimes had to cancel before going, sometimes had to abort the vacation half way, but she never really gave up. We kept more in touch than before, talking at least every week, and meeting once or twice a month. We talked about good things and bad, and had a stronger relationship than in many years.

She started dancing swing again. She took up bridge, which she thought she was awfully bad at after so many years. She did almost all the things she couldn't do while father was too ill to go anywhere, or too ill to be left alone.

Her health kept declining, in some ways much as my father's had, but far quicker and therefore more visibly. Nobody blamed smoking, though.

She got help from Hospice Sangen, a day facility for people with severe illnesses, particularly cancer patients. She had an "open return" at Hamar hospital, in case she needed medical care, there was always room there. She got home care from nurses, whenever necessary, and at regular intervals with increasing frequency.

This fall, she was admitted to Hospice Lovisenberg, Sangen's "mother" facility in Oslo, for help with her significant pain medication problems; the medication just wouldn't take. She was fairly active while she stayed at Lovisenberg. Two Saturdays, she went shopping with me, the first time also with her elder sister. Lovisenberg got the pain sorted out, and on Monday a month ago - my birthday - I drove her home to Hamar again, thinking that she seemed a bit tired and reduced, but all understandable, since she had been stressing herself out a bit with packing and whatnot. Besides, she had been getting enough morphine to kill grown men several times over, on a regular basis, for weeks.

The following Wednesday, we talked on the phone again. She was more tired, and felt a bit unsteady on her legs, and had therefore gotten a four-wheel walker. I decided that I wanted to check up on her after the coming weekend, which was full of plans for myself - birthday party, roleplaying, etc. I told her I was coming the following Monday, and she was happy that I wanted to.

I called again on Friday during lunch hour, and my mother said: "I don't have much time left now, Jan. Can you talk to my sister?" and handed the phone over to her sister (not the same who went shopping with us just two weeks earlier). She explained to me that mother was lying on the couch (as she often did when she felt tired), but that she had hardly eaten that day, and my aunt didn't dare leave her alone. I said I'd try to get there this weekend.

I talked to my boss again, tidied up a few pressing tasks at work, went home and packed. I cancelled the party and notified others that I was not coming for roleplaying or other social events. I then told my mother and aunt that I was coming that evening, got into the car, and did as I said.

When I got there, we got help from a nurse so that mother could get to bed and sleep there. She was very tired, and barely coherent, though she heard and understood us.

The following morning, mother couldn't even sit up without aid, and she could not drink and swallow her pills without help. My aunt came back during the day, and we convinced a supervising nurse to have a look at my mother's now fairly swollen legs. We then convinced the nurse to call for an ambulance, and had mother admitted to the hospital that Saturday night.

At hospital, I called my youngest cousin, the closest person I've had to a brother, and told him briefly how it was. He said he'd come and keep me company.

During the night, mother had occasional pain in spite of the better pain medication and well-adjusted morphine pump, and occasionally lucid moments. Her breathing was laboured, but when the nurses managed to adjust the morphine dosage up by a third and the bolus frequency limit down by a third, she was well relieved. I had to push the button, though, because my mother could not.

In the early morning, her breathing slowed and became weaker. She was sleeping or dozing all the while, and I was holding her hand, and my cousin was holding mine.

Her face was tilted slightly towards me, and one eyelid was slightly ajar, the pupil aligned with my face. She died so peacefully that I can't say exactly when it was. It was beautiful, and I wouldn't have missed it for the world.

But I miss you, mother, and most of all today, and now I think of Sonja's song and cry.

Community boost

2010-08-01T15:05:00.002+02:00

It is now three days since the release of Rakudo Star.

Everything has met my personal expectations, except for one major thing:

There is a bunch of new people on #perl6, who are actively engaged in trying out Rakudo Star/Perl 6, in testing, submitting bug reports, patch suggestions, asking for help, providing help, helping themselves…

I really do not have anything profound to say, but I can say this:

Now we are really getting somewhere!

Rakudo Star is here

2010-07-29T14:29:00.004+02:00

The wait is over, and a usable Rakudo-based Perl 6 distribution is ready for early adopters. Yay!

More and more of Perl 6 is implemented in Perl 6. That is a very good tendency.

Coding styles that make me twitch, part 8

2010-03-04T17:00:00.000+01:00

One of the worst programming habits I know of, is assuming that everything will work out; you just perform the command/function call, and it has to work. In other words: do not bother checking for error conditions or propagating them, because they are not going to happen anyway. right?

Wrong.

But the code works. Mostly. Except when an error condition occurs. And it will, eventually.

In some instances, such code is a glorified shell script. I have ranted about that before. But catching error conditions properly can be tricker than with in-Perl functions and modules. Especially if you have no control over what the external program does, but the original programmer did.

Even with in-Perl functions and modules, you might run into, ehrm, interesting usage.

Usually, the root of the problem is that the original programmer did not foresee that some his code could grow into something else.

While it may have been a good idea to create an SQL query wrapper to hide parts of what DBI does for auto-committing statements:

sub do_sql {
    my $qry = shift;
    my $ignore = shift;
    if (!$dh) {
        &slogin();
        return if (!$dh);
    }
    my $sh = $dh->prepare($qry);
    if ($sh && !$sh->execute) {
        if ($sh->errstr =~ /(MySQL server has\
 gone away)|(Lost connection to MySQL server)/) {
            &log("Lost MySQL, reconnecting.");
            &slogin();
            return if (!$dh);
            $sh = $dh->prepare($sqlstmt);
            undef $sh if ($sh && !$sh->execute);
        } else {
            undef $sh;
        }
    }
    &log("SQL failed: '$qry'.") if (!$sh && !$ignore);
    return $sh;
}

that is no guarantee that the implementation will be future-proof, when the subroutine is used like this:

my $sh=&do_sql("SELECT * FROM tab1 WHERE\
 x<>'$cgi->{data}' AND y<>42");
if (my $res=$sh->fetchrow_hashref) {
    $cgi->{ref}=$res->{ref};
}
&do_sql("UPDATE tab1 SET x='$cgi->{newdata}',\
 y=23, z='$cgi->{ref}'");
&do_sql("UPDATE tab2 SET tab1_changed='yes'");
&log("Updated tab1, ready for externals");
# Process the changes made above:
system("/usr/local/bin/changestuff.pl");
&log("Finished processing");

Imagine now that changestuff.pl also performs changes in the database tables mentioned above, and that the code above is called in a cron job every minute or so.

Here is an attempt at listing the worst parts:

do_sql only pretends to do proper error checking, it mostly does not do anything useful about the error situations.
The query result seems of little consequence.
There is an obvious need for bound variables in the prepared statement, but do_sql does not support that. So the code pretends the problem does not exist.
When do_sql is used for updates, there is no check whether the returned statement handle ($sh) is empty or not, there is no way of knowing whether we are clobbering the database.
Why do we not care whether the external commands succeeds or not?
SQL transactions, anyone?
Yeah, other parts of the style sucks, too.

Now imagine the example above multiplied to thousands of lines of inter-dependant code.

I am happy to say that I do not see things like this too often.

However, cleaning up code like this is a PITA, and it is often easier just to close your eyes, add your own code, and leave well enough alone.

Coding styles that make me twitch, part 7

2010-02-25T16:42:00.001+01:00

Parentheses and precedence

This week, I will strive to be better at explaining the problem scope, while heroically and miraculously maintaining brevity.

I used to be annoyed at seeing extraneous parentheses with the ternary conditional operator pair ? :, but then I discovered that the PHP language designers decided to break precedence (see example #3). So I have stopped twitching as much as I used to when seeing parentheses galore in that context.

In Perl Best Practices, Damian Conway notes that even though the low-precedence logic operators and, not and or may seem nicer, they can confuse the reader regarding the intended meaning of the code. But he does not — as far as I can recall or find — address how these relates to parentheses and precedence (I suppose that should be "parentheses && precedence").

I start twitching when someone insists on using parentheses like this (imagine that the example was more convoluted):

if (expr1 ||
    (expr2 && expr3) ||
    expr4
    ...)

That does not help understanding. It confuses me, as the read, regarding your intentions. Is there a piece of code missing, perhaps a little bit of || expr2b or || expr3b?

Sometimes, I even see misguidedly mixed-in letter-literal operators:

if (expr1 or
    (expr2 && expr3) or
    (!expr4)
    ...)

I much prefer seeing

if (expr1 ||
    expr2 && expr3 ||
    expr4
    ...)

if (expr1
    || expr2 && expr3
    || expr4
    ...)

depending on what floats your boat in the most stylish way imaginable.

I would have thought that the logical and/or/not part of operator precedence would be easy to understand. It is essentially the same in most programming languages: Ruby, Python, Perl, Java, C, ... and not even PHP managed to mess this one up.

The chance that anyone is going to be confused by your code because of these parentheses not being there is far, far smaller than the chance that they are going to be confused by their presence.

I have heard the defense "but the code is so non-obvious that I had to add them!" Well, make your code obvious instead.

Coding styles that make me twitch, part 6

2010-02-18T21:23:00.003+01:00

Edit: There are, apparently, strong feelings that I should not post my personal preferences. Reader discretion advised: this post expresses my personal opinions regarding a limited use case for file handling and filehandles, and must not be read as general advise on how to deal with filehandles in Perl, and so on.

Today's theme is: unnecessary variable clutter.

my $fh = new Filehandle("/usr/bin/somecmd someargs |");
while (my $l=<$fh>) {
    if ($l=~m/foo bar zot/) {
        # lots of code that does not depend on
        # a variable file handle, nor creative
        # usage of $l where $_ could not be used
        # implicitly just as easily
    }
}

Sure, there are situations where it makes sense to assign filehandles to variables, or using $l instead of $_, but the above example is not one of them. I found no particular reason why the original programmer had used new Filehandle, either.

Perl 5 has, on purpose, made this easier for us than in the above example:

open my $FH, '-|', '/usr/bin/somecmd someargs'
    or croak "OMG\n";
while (<$FH>) {
    if (/foo bar zot/) {
        # lots of code
    }
}

Missing feature

2010-02-11T01:33:00.003+01:00

A few hours ago, I suddenly had a bright(?) idea, or desire if you will:

Proper (Unicode) exponents in Perl.

That is, I want to be able to write 2², 4¹³⁷, 3^-9, etc. and have Perl understand them.

For Perl 5, I suspect someone would use a source filter to implement it.

For Perl 6, PerlJam++ suggested introducing each of the exponents as postfix operators, using this example for squaring:

our &postfix:<²> := &infix:<**>.assuming(b => 2);

But then a negative exponent would complicate things a bit.

It's a thought, anyway, and not one that I'd want to distract more pressing implementation concerns.

And ifwhen someone decides that this is a good idea to have in the language core, I'll start nagging about Knuth's up-arrow notation. Not that I'd want anyone attempt calculating 4↑↑↑↑4.

Spooky

2010-02-04T00:21:00.004+01:00

This is just a small anecdote.

Tonight, I finished a small Perl 5 script that I've been wanting to complete for a while, but where I was a bit nervous that I'd fsck it up right and good.

It was a script designed to handle two tab-separated text sources; one a list of tournament IDs and tournament names, the other a list of player results, one line per result with the player ID, tournament name, position and score achieved.

I achieved this by creating a hash of hashes for each file, referencing the first while parsing the other, and bravely inserting the data into a single database table.

I tested my code piece for piece while building it, which is sensible in itself, but what spooked me was this:

There was not a single bug. The script did what it was supposed to do, all along.

That's not supposed to happen.

I need a drink.

Rakudo ng - what will it mean for us?

2010-01-29T18:24:00.000+01:00

If you've been hanging around the right blogs and the #perl6 IRC channel on Freenode, then you've probably seen references to a slightly mysterious "ng", or "Rakudo ng".

That's the upcoming (next) version (generation) of Rakudo, which will form the basis for Rakudo *.

In essence, this is a refactoring/rewrite of Rakudo for the purpose of better compliance with the specification and performance improvements (yay). The old Rakudo master made it difficult — if not impossible — to implement several essential parts of the Perl 6 spec and top priorities on the Rakudo roadmap.

In January, this has led to less focus on the current Rakudo version's bugs and gotchas, and instead on working to prepare ng as the new master branch — that is, the Rakudo that you will be downloading the next time.

For those of us who do some Perl 6 coding in Rakudo, this means that we can expect a nice little bunch of incompatibilities as compared to the current master. And yes, it's very close, so it's time to prepare.

Here's a list of the blindingly obvious things I think we need to watch out for:

Older Rakudo was not in line with parts of the spec that ng will be.
The spec has changed. (ng development has uncovered several necessary changes.)
Older Rakudo is in line with parts of the spec that ng perhaps isn't.
Rakudo ng is, of course, not feature complete when it replaces older Rakudo as the master.

In other words: let's not fool ourselves into thinking that we all of a sudden have a new Rakudo that's both compatible with the older as well as being spec compliant.

The good news about Rakudo ng

If you judge by the above paragraphs, you'd think that Rakudo ng was bad for Perl 6 developers. But that's far off the mark. I prodded #perl6 and Patrick Michaud before publishing this post, and here's a brief summary of (most of) the improvements we can see coming with Rakudo ng as opposed to the current implementation.

Most of the top priorities of the Rakudo roadmap will be implemented!
Laziness will mostly work (the spec is undergoing change)
Performance improvements, many due to laziness
Array/List/Parcel/etc. will be compliant with the updated spec
Protoregexes
Better longest token matching
Meta-operators are really meta, and generated on demand
The base object metamodel is far closer to the spec than before
Major portions of the metamodel are implemented in Perl 6
Array and hash vivification will work properly
Lexical subs and variables work properly
Operators have the correct names (with angles)
Subs have the correct sigils (with ampersands)
Phasers work, and the phaser model is much improved

Our programs will need a bit of attention. I recommend subscribing to perl6-language for up-to-date information about changes to the specification and language discussions.

There's still a lot of work to be done, and I'm sure the Perl 6 developers are happy for any help they can get.

"Your Unix Is Leaking Perl"

2010-01-21T22:40:00.003+01:00

That has to be one of the weirdest statements I've read in, oh, at least fifteen minutes.^[1]

(The image is a link to the original strip; if you've got javascript enabled, you can see this bonus strip by hovering over the burgundy red button below to the left.)

A huge thanks to Saturday Morning Breakfast Cereal for this piece of wisdom, but I'm afraid that Zach Weiner got it backwards when he thought that would be expensive.

May I suggest that when a computer "leaks Perl", it does so because the (hopefully brilliant) Perl programmer is contributing a lot to CPAN?

;)

Yes, I've been communicating with customers lately, how did you guess?

feather.perl6.nl - a Sysadminish Tale

2010-01-14T16:45:00.007+01:00

<@Juerd> frettled: Blog about the mess you found when
         you first logged in on feather yesterday :)

Sure.

This will, incidentally, also explain why Trac is kindof unavailable now.

feather.perl6.nl is a Xen guest (a virtual machine, hereafter "VM") that's hosting several important services for the Perl 6 community. There's SVN web access, a Trac installation, and a bunch of other stuff I honestly don't know the half of.

Recently, the VM started running out of memory too often for comfort. What was going on? Juerd asked for help in tracking down the problem, as he didn't have the time to do so himself. And needing some distraction from work -- something to help me procrastinate -- I volunteered.

By now, you're probably banging your head on your keyboard in sympathy with me for saying something that may have been slightly less than brilliant. You know the feeling; Matt Trout reaches his right hand towards you, the world is suddenly in slow-motion, you see his hand closing in on you, his grin widening, and a voice saying "thhhhhaaaaannnnnkssss ffffooooorrr vooooluuunnnteeeerrriiiinnnnng", and you're basically up that creek with all the mud and dirt in it.

After handing over an SSH public key and getting sudo access (yeah, yeah, I know), I had a look anyway.

First, I went on a brief but wild goose chase, finding some error messages regarding ConsoleKit which appeared to be more frequent just before the server went out of memory, checked the Debian version (an unholy mix of Debian unstable and Debian experimental with lots of package updates pending someone's attention), and generally tried to get a feel of how the system was configured.

We already knew that Apache somehow might be responsible for gobbling up available memory, so my first action was to have a look at the last 100,000 lines of the Apache access log, using a simplistic log analysis script.

But which log? There were three Apache log directories to choose from. I (correctly) guessed that the one called simply /var/log/apache2 might be the interesting one, the others seemed to be legacy directories which should have been removed ages ago.

According to the script, there were 0 accesses in the last 100,000 lines.

Knowing the script, that was not so strange, because it makes a few assumptions regarding the log format, using a regexp belonging to the days before named captures and whatnot:

while (<>) {
  if (/^(\S+) (\S+) - - \[[^\]]*\] \"(GET|POST) \S*
       HTTP(|\/1\.[01])\" \d{3} (\d+) \"/) {

The regexp line has been split for the sake of the line width of this blog. There's nothing to be proud of here.

Anyway, I first had to remove the first capture; feather's logs weren't showing the virtualhost as the first column, and access types were most certainly not limited to only GET and POST:

frettled@feather:~$ sudo awk '{print $6}' /var/log/apache2/access.log|
sort -u
"CHECKOUT
"CONNECT
"DELETE
"GET
"HEAD
"MERGE
"MKACTIVITY
"OPTIONS
"POST
"PROPFIND
"PROPPATCH
"PUT
"REPORT

Right.

After straightening that up (and adding %v to the LogFormat specifications in the Apache config for future use), I got the following result:

Use of uninitialized value $size in addition (+)
at /usr/local/sbin/bandwidthips line 39, <> line 1002.
Use of uninitialized value $size in addition (+)
at /usr/local/sbin/bandwidthips line 40, <> line 1002.
Use of uninitialized value $size in addition (+)
at /usr/local/sbin/bandwidthips line 44, <> line 1002.

AAAARRGH! Idiot! Imbecile! Inept half-wit! Yep, I'd forgotten to renumber my captures. See, this is why Perl should be in version 5.10.1 or 6 when fiddling with those bloody annoying regexps.

frettled@feather:~$ sudo tail -100000 /var/log/apache2/access.log|
/usr/local/sbin/hitips|head
193.200.132.146: Bytes = 14329487 (3.44%), Hits = 51503 (51.82%)
66.249.71.2: Bytes = 132116084 (31.73%), Hits = 18111 (18.22%)
66.249.71.37: Bytes = 50846948 (12.21%), Hits = 6236 (6.27%)
93.158.149.31: Bytes = 54880221 (13.18%), Hits = 1894 (1.9%)
71.194.15.106: Bytes = 460200 (0.11%), Hits = 1894 (1.9%)
209.9.237.232: Bytes = 433388 (0.1%), Hits = 1686 (1.69%)
193.200.132.135: Bytes = 1726871 (0.41%), Hits = 1635 (1.64%)
193.200.132.142: Bytes = 429358 (0.1%), Hits = 1609 (1.61%)
208.115.111.246: Bytes = 8461415 (2.03%), Hits = 1238 (1.24%)
67.218.116.133: Bytes = 18945415 (4.55%), Hits = 1126 (1.13%)

So, uhm, around 52% of the hits come from feather3.perl6.nl, and nearly 25% from Google's indexer. Lovely.

Looking at the accesses from feather3, I quickly saw that they mostly had to do with svnweb.

Juerd had already stopped Apache, but someone -- I don't know who -- started it again at 12:00, probably anxious that SVN and such didn't work.

I then followed the running processes using the top command, updating each second (top d1), sorting by memory usage (typing M while top was running), hoping to catch some quickly growing processes.

Nopes. None, zilch, nada. Nothing that appeared horribly wrong. Sure, the apache2 processes used some memory (30-60 MB resident set, 50-100 virtual), but nothing appeared to be out of the ordinary. I changed the update frequency to each third second -- top sometimes uses an inordinate amount of CPU, depending on magic -- and waited. After a while, a couple of apache2 processes were using more CPU and memory than the others, around 60-90 MB resident. And they were growing. And according to lsof, they were active in the svnweb directory (and used a metric shitload of libraries). And after growing, they didn't release memory, they just kept on using it. But it wasn't enough to use up memory, there was still a bunch of free RAM.

So that was perhaps svnweb's fault, then?

Maybe.

But then my time ran out, and I had to drop the ball, leaving the top process running.

Five minutes later, the memory ran out again. It's just as if someone was waiting for me to go idle in order to produce the problem that I was looking for.

Sigh.

svnweb kindof remained the main suspect, until Juerd caught whatever was happening at the right time.

And catching what happens at the right time is bloody important.

Here's what he found, using Apache's server status:

Well, that's not svnweb. That's Trac. And the IP addresses belong to Google.

And that's spam, effectively creating a DoS or DDoS attack on our services as a side effect when search engines try to index the Trac webpages. It probably isn't intentional, but spammers just don't care.

So, what can we do to protect feather from suffering from such attacks in the future?

There's a lot that can be done. It takes effort. It takes time. It takes someone.

Here are a few suggestions on how to improve the robustness of the kind of services feather provides:

Add a captcha to the web form. The disadvantage is that this does not really save processing resources, but it probably should be done anyway.
Add an unnecessary and bogus input field to the web form, e.g. "Phone number". This input field should be hidden with CSS so that web browsers don't display it, and if someone submits anything with data with that field's name, then you can be nearly 100% certain it's spam from someone who's used a web scraper before automatically filling the form. Filter it out.
Change the webserver delegation architecture, so that each Apache process isn't loading tens of megabytes of libraries and keeping them in memory. Off-loading to shorter-living FastCGI daemons or similar solutions, or even sacrificing program startup speed by using CGI+suexec, etc., may be decent starting points.
Consider using a front-end proxy like Varnish to gloss over underlying nastiness.
Start with a new VM and migrate services to that one, gradually.
Document configuration choices and what each web service does/is there for, so that the next sysadmin coming along can make educated guesses quicker. :)

These tasks can rather easily be split into manageable one-person projects.

Does this sound interesting to you, or did I lose you at the third line of this blog entry?

Pop in on #perl6 on the freenode IRC network and say so.

Typing More Or Less

2010-01-05T22:10:00.005+01:00

Not too long ago, there was a bit of minor cleanup in the Perl 6 specification regarding the use of whatever (*); there were some inconsistencies in how it behaved, depending on context.

The net result is that you now must use @arr[*-1] to get the last element, you cannot get away with simply using @arr[*].

Some may feel that this extra typing is bothersome, especially if you have a Unicode-friendly keyboard setup.

However, we can sneak our way past this problem by using a constant.

This also works with the current release of Rakudo, so it's not quite science fiction:

constant Ω = *-1;

Or, if you're feeling Cyrillic rather than Greek:

constant Ѡ = *-1;

Now we can substitute our nice constant for *-1 anywhere in the following code:

my @letters = 'a'..'z';
say '→'~@letters[*-1];
→z

Like so:

constant Ω = *-1;
my @letters = 'a'..'z';
say '→'~@letters[Ω];
→z

And, of course, you can do this with other things that are so tedious to type when you're dealing with maths:

constant π = pi;
say '→'~π
→3.14159265358979

2009 In Perl

2009-12-31T00:31:00.004+01:00

Repeat after me: I will not pretend to be an analyst or doomsayer, even though the end (of 2009) is nigh.

In 2009, Perl grew up a bit more, both as a language and as a community.

Language Development

Perl 5.10.1 came with a pony to those of us who fear the .0 releases.

The Perl 5.11 development tree got started, and it looks like it is rolling on rails. At this rate, we will see 5.12.0 quicker than you can say antidisestablishmentarianism.

... Perl 6 has made progress both on the specification side and in implementations -- yep, that is plural. It is sometimes confusing when naming changes under your feet, but it is acceptable while the spec is still settling.

Community

In 2009, I think I saw more openness regarding the internal conflicts in the Perl community as a whole; there were abundant admissions that we were not communicating nearly as well as we should, that there was at least a small amount of internal bickering over the present and future state of the onion -- onions, I must inject, tend to come in many shapes and flavours, and are not always the same inside -- really, which way we are going, are we having a conflict or not (yes we are -- no we are not -- huh, are we talking? -- pass the chips), and get off my lawn before I shoot or hug you.

In brief, it looks to me like 2009 was the year when the community showed renews signs of self-awareness.

But much more happened. We got a closer focus on Perl visibility, from my POV mainly owing to Matt S. Trout's lightning talk challenge from NPW 2009, plus a whole range of people working on other PR aspects for ourselves. And mst still keeps his hair colour. Wow.

Other Stuff

I made new friends, I learned a lot, I even got to help out a bit, and I hope that this will continue in 2010.

I hope you will too.

Happy new year!

Dice Roller Deconstructed

2009-12-21T21:56:00.003+01:00

As promised, here are the elements of last week's dice rolling code:

use v6;

This is a nice way to say that we are in Perl 6 land.

subset D10 of Int where 1..10;

A "D10" is a 10-sided die, and it can only have integer values in the range 1..10. Subtyping Int is an acceptable way of taking care of that.

sub is_success (D10 $roll, D10 $target) {

Here, I am already using the subtype D10 of Int. This subroutine compares the rolled die $roll with the target number $target, and is called from the subroutine roll() for each die in the dice pool. I chose to create an explicit subroutine because it seems a bit clearer what happens in the special case of a rolled 10, which means that you get to re-roll that die for a potential new success.

    my $n = 0;
    if ($roll == 10) {
        say "10 again";
        $n += roll 1,$target;
    }

If we roll a 10, then the roll() subroutine is called with a dice pool of 1 and the same target number as we got originally for determining success.

    $roll >= $target ?? $n + 1 !! $n;
}

We always return the number of successes from the roll for the "10 again" rule (if it happened), and in case this roll was a success, we return an additional success.

sub roll (Int $poolsize where { $_ > 0 }, D10 $target? = 8) {

The dice pool size can of course not be negative, but it also cannot be zero; you always get to roll a die, so I have added a type constraint for that. The target number is optional, defaults to 8, and has to be possible with a D10.

    my D10 @rolls = (1..10).pick($poolsize, :replace);

From left to right:

@rolls is an array that will contain the results of the normal die rolls
(1..10).pick($poolsize is a way of picking $poolsize dice having possible values in the range 1..10 and "rolling" (randomizing) each of them.
pick($poolsize, :replace) means that we not only pick a result, but we also make it possible to achieve the same result again. Specifically, it is important for us that each die can have ANY value, not just values that have not been picked before. The semantics of pick() are explained in .pick your game (the 15th gift in the Perl 6 Advent Calendar).

    say "Roll: " ~ @rolls.sort.join(",");

@rolls.sort.join(",") sorts the elements of the @rolls array and stringifies them joined with a comma, e.g. "1,2,3,3,4" for @rolls = 4,1,3,2,3

    [+] @rolls.map: { is_success $_,$target };
}

This piece of code maps is_success $_,$target on every value in the @rolls array and creates a sum of those results. In other words, it sums up the number of successfull die rolls.

given @*ARGS.elems {

The @*ARGS array contains the command line arguments to the program, and .elems therefore is the number of arguments used.

    when 2   {
               say "Target number: " ~ @*ARGS[1];
               continue;
             }

This block only runs in case we have two arguments, but it explicitly says that we may not be done yet: the continue statement counters the default implisit break to ensure that we can match the input value against other tests.

    when 1|2 {
               my $n = roll |@*ARGS>>.Int;
               say "Successes rolled: " ~ $n;
               $n >= 5 and say "Exceptional success!";
             }

We start off with a junction to say that either 1 or 2 is fine by us, we want both to match. Then we call roll() with the same arguments we got in, but each converted to Int. White magic. We store the value, and exclaim that the result is an exceptional success if it is.

    when *   {
               $*ERR.say("roll.p6 poolsize [target]");
               exit(64);
             }
}

This is the equivalent of C's default, the catch-all that handles remaining uncaught cases. We print a helpful usage string to STDERR ($*ERR in Perl 6) and exit with the correct Unix exit code, praying that nobody uses a different kind of system.

Dice Rolls for Role-Players

2009-12-16T15:50:00.002+01:00

I realize that the title of this post is a bit of an oxymoron, because a Real Role-Player of course doesn't roll dice often. ;)

But in the cases where the Real Role-Player does roll dice, wouldn't it be nice to have a computer program to forget at home rather than some even more easily mislaid dice?

The Perl 6 Advent Calendar provided some inspiration for this post.

A problem with many minor programming examples you see on the net, is that they do not take into account the needs of a role-player. Role-players play many different systems, with different criteria for success in dice rolls. D6 (the regular six-sided cubic dice used for playing Monopoly, Yahtzee, etc.) are not used much in the majority of systems.

Therefore, I'll look at the Storyteller System, which is used in the World of Darkness series of games.

The general principle is that you have a pool of dice to roll, and you count your successes, which in this system is the number of dice that have a value greater than or equal to a given target number for the roll. The standard target number is 8 in most implementations. Five successes in the same roll is an exceptional success. Obviously, it's nice to have many dice to roll!

Here's a real Perl 6 program that works with Rakudo today: it accepts two command line parameters, the first being the size of the dice pool, the optional second parameter defines the target number for success:

use v6;

subset D10 of Int where 1..10;

sub is_success (D10 $roll, D10 $target) {
    my $n = 0;
    if ($roll == 10) {
        say "10 again";
        $n += roll 1,$target;
    }
    $roll >= $target ?? $n + 1 !! $n;
}

sub roll (Int $poolsize where { $_ > 0 }, D10 $target? = 8) {
    my D10 @rolls = (1..10).pick($poolsize, :replace);
    say "Roll: " ~ @rolls.sort.join(",");
    [+] @rolls.map: { is_success $_,$target };
}

given @*ARGS.elems {
    when 2   {
               say "Target number: " ~ @*ARGS[1];
               continue;
             }
    when 1|2 {
               my $n = roll |@*ARGS>>.Int;
               say "Successes rolled: " ~ $n;
               $n >= 5 and say "Exceptional success!";
             }
    when *   {
               $*ERR.say("roll.p6 poolsize [target]");
               exit(64);
             }
}

Thanks to moritz++ for ironing out two annoying mistakes!

Here are a few usage examples:

$ perl6 roll.p6
roll.pl poolsize [target]

$ perl6 roll.p6 5
Roll: 1,2,7,8,9
Successes rolled: 2

$ perl6 roll.p6 5 2
Target number: 2
Roll: 1,2,2,4,9
Successes rolled: 4

$ perl6 roll.p6 5 4
Target number: 4
Roll: 6,8,9,10,10
10 again
Roll: 8
10 again
Roll: 2
Successes rolled: 6 - Exceptional success!

There are no comments in this piece of code, I want people to try to understand it as-is, based on the Perl 6 Advent Calendar. If you have any questions, comments, corrections, etc., don't hesitate, just write!

In my next blog entry, I'll pick the program apart and comment on what I've done and why, and who knows, maybe someone has come up with an elegant solution to the same problem.

GCD - A Small Language Enthuser

2009-12-09T18:43:00.003+01:00

fun gcd (x:int,y:int) : int =
    case x of 0 => y
  | _ => if x < 0 then gcd(y,0-x) else
         if y < 0 then gcd(0-y,x) else
         if y > x then gcd(y-x,x) else gcd(x-y,y);

"But that's not Perl!"

Yeah, yeah, I hear you.

I'll rectify that minor detail in a bit.

But first, an anecdote.

Back in the late nineteennineties, I was studying computer science, and one of the classes was about program specification and verification.

Several of the students already had a background with several programming languages, some were functional, some were imperative, and other languages were a bit confused about what they really were.

When studying program specification and verification, you either become rather obsessed with program correctness -- and hopefully elegance -- or you fail spectactularly.

There are several ways to muster enthusiasm when dealing with such studies; they can be rather, ehrm, theoretical.

I therefore flitted about, flirting with various programming languages, comparing them with the eagerness that young idealists do.

For some reason, I found Euclid's GCD algorithm to be particularly fascinating, for reasons unknown to men to this day.

The Perl version I saw was rather awful, and technically incorrect:

sub gcd {
    if (!$_[0]) {
        return $_[1];
    }
    if ($_[1] > $_[0]) {
        return gcd ($_[1]-$_[0],$_[0]);
    }
    return gcd ($_[0]-$_[1],$_[1]);
}

Yikes. I mean, eep. And Perl does have a modulo operator.

sub gcd {
    my ($x, $y) = @_;
    $y ? gcd ($y, $x % $y) : abs $x;
}

I won't claim that the above code is the epitome of elegance, but it solves the problem in a general and easily read way (I admit a prejudice against $_[N]), while retaining correctness.

This is, BTW, one place where some golfers miss the boat; the GCD cannot be a negative integer. That's why the ML code at the top is so verbose.

Small challenges like these kept me going, and it can be an inspiring way to learn details in a new language. So, what would it look like in Perl 6?

sub gcd (Int $x, Int $y) {
    $y ?? gcd($y, $x % $y) !! $x.abs;
}

What's your favourite algorithm for playtesting languages?

Oslo.pm Past and Future

2009-12-01T18:22:00.004+01:00

In case the title wasn't a give-away: this is a non-technical blog entry.

I became an Oslo.pm member by signing up for the mailing list shortly after the Nordic Perl Workshop 2009. That's cheap (well, free!), easy, and therefore newbie-friendly.

Last week, I dropped in at the general assembly and exercised my speaking and voting rights, and got an inside scoop on how this Perl organization works. The board members were, after all, the guys who did a terrific job arranging not only this spring's workshop, but also mostly the same people who held the workshop of 2006, which also went quite well.

From my point of view, Oslo.pm has come from being an anonymous group to a rather solid little volunteer organization. Before 2006, I'd have said "huh?" if someone asked me who might have anything to do with Perl in Norway, afterwards, I knew there was something called Oslo.pm, and so did a few people in Europe and the USA. After NPW 2009, I think it's safe to say that the organization is now known as a stable and capable group of Perl mongers. That's a decent achievement, especially in this age, when it seems like almost nobody (in Norway) is willing to do anything free of charge.

So what did they think about themselves, and what's going to happen in the near future?

True to the Norwegian spirit, they were modest and self-disparaging, but they were very happy that the attendees were apparently happy, even months later.

Salve J. Nilsen, the Great Leader of 2009, bowed and said farewell to the post of chairman, and now Marcus Ramberg is at the helm.

The new Oslo.pm board will attempt to increase local activity, and there will probably be some kind of technical talk on the first Wednesday of almost every month in 2010. They aim to increase cooperation with local Perl-using companies, as well as aiming for some cross-language and language agnostic sessions.

First out is tomorrow's Perl 5.10 session at Redpill Linpro, which I'm sure will be technically rewarding for those who show up. I plan to!

The morality of helping

2009-11-25T22:28:00.003+01:00

Yesterday, a friend asked me, "are you a Perl expert?"

I answered in the only way possible: "eeeeh..."

It turned out that my friend did not ask for help for himself, but for someone else, who had posted a programming class question on a non-programming bulletin board.

The poor fellow was struggling with a question of parsing a two-column input to generate a certain output format, essentially also two-column, except with a slightly different layout.

On Usenet, there was - and maybe still is - a long-standing tradition of not solving people's homework for them. The reasoning behind this is that we do not learn quite as well when people solve our problems for us, as when we struggle with them ourselves.

In some cases, school questions would be met with derision, in other cases with genuinely unhelpful and false answers, and sometimes with helpful clues about how to solve the problem; where to look, tips for using stepping debuggers, which book chapter or manual page would clarify the problem, etc.

Okay, that is fair enough.

The guy had gotten only one answer, from another guy who regretted that he had not touched Perl in ages, and therefore could not help. And I thought that Perl was like learning a particularly catchy, but annoying song: you might think that you have forgotten, but then someone hums or whistles the tune, and WHAM - there it is, stuck in your head again.

So I had a look. Maybe I could provide a hint or two, you never know. I know my way around some of the less scary parts of Perl 5 City, anyway.

This guy had essentially nailed the problem semantically, but he was evidently struggling with his code, it just did not work.

I immediately saw a few major concerns:

Some parts were copy-pasted from bad textbook Perl.

Some parts must have come from a poor programming education.

The code was overly complex and verbose.

There was no error checking or debug print-outs.

And it would take me more time to helpfully point out these things than write something that might be a solution myself.

The moral dilemma then was:

Should I help the guy out by tearing his code apart and pointing out all the flaws that made it thoroughly lousy code, thereby provoking a true emo-American melodrama?

Or should I just write an alternate solution, with sound error-checking, simplicity, and debug print-outs?

In this case, I thought the latter was the way to go. I put the code up anonymously somewhere, gave the link to my friend, and perhaps the fellow with the problem now has a better understanding of how simple and beautiful Perl can be.

Yeah, right. :D

What stops me from using Perl 6, today?

2009-11-15T23:56:00.006+01:00

Since I got hooked on the Perl community, and got a taste of Perl 6, I've been wondering about:

what, exactly, is it that I could use Perl 6 for, right now?
why am I not actively using Perl 6 now?

Those are easy questions, but answering is hard, so this may be a long post.

Sure, the points listed below are not exactly Perl 6 specific; I could probably have picked some other programming language, but I somehow feel more comfortable in the way that Perl 6 still is Perl.

What I could use Perl 6 for right now

I think it's fair to say that using Perl 6 today mostly means using Rakudo, and that I wouldn't use it in what we popularly call a "production setting". But many of us programmers, sysadmins, geeks and nerds have perfectly suitable hobby projects, where we won't have clients wringing our necks if there is three minutes of downtime in a month, or if we don't deliver the Speedy Gonzalez of services; we have projects that are neither computing performance constrained or stability constrained.

So that's where I could have started using Perl 6 half a year ago, and of course still can.

I know I can use Perl 6 for e.g. a fairly complex web site using Web.pm and Squerl for a SQLite backend. It will probably work just fine, for a lot of projects.

I know I can use it for lots of one-liner scripts.

I know that in some regards, Perl 6 will outperform classic Perl 5 in terms of programmer time spent. An example is the given-when control structure, which (to me) is semantically superior to if-elsif-elsif-elsif. Programmer time is important to me, I hate coding too much for menial tasks. And I'm sorry to say that Perl 5.10 doesn't do it yet for me, as I cannot rely on its presence, even for hobby projects.

And I know I can use Perl 6 to refresh some of the knowledge about programming language specifics (terminology, technique, methodology, etc.) that I've allowed to rust since I left university in 2001.

Concrete projects, in no particular order

web page for registration of pool billiards tournament results; it's not performance critical, and the users could check and verify the dataset themselves after input

conversion of historical results data in CSV format to a database; one-time job, needs manual verification no matter what programming language I use to do it

contributing to the Temporal.pm specification and implementation in Perl 6

personal web gallery generation; I positively loathe most of the online galleries, because they sooner rather than later are discovered to have HUGE, GAPING security vulnerabilities

blogging tool; I'm not very comfortable with blog software running on servers, either, and whatever blogging I do, it's not actual work

That's quite a lot, isn't it? It ought to have been enough to get me going in a jiffy!

Why I'm not actively using Perl 6 now

This may be a surprise to some: it's not because of a lack of matureness in the tools, a lack of confidence in the language or tools, stability issues, etc. As I tangentially mentioned above, I believe there is no technical hindrance for me to start coding on a hobby project.

I have plenty of hobby projects to choose from. They are also quite manageable in terms of eventual lines of code.

However, there is something holding me back, and that's a certain degree of perfectionism mixed with procrastination fever.

mst mentioned during the NPW hackathon this spring that perfectionism was a barrier against getting started. If you're too obsessed with getting things right at first, at wanting to avoid failure, procrastinating is too easy. Getting slightly intoxicated (yup, drinking alcohol, which of course is only a recourse for adults) is a way of reducing your own perfection anxiety. This is almost exactly what Randall Munroe's xkcd calls the Ballmer Peak:

But I don't sleep too well after drinking alcohol, and I also tend to do hobby projects in my "running breaks" during work hours, in which case alcohol intake may be a very bad idea.

In addition, my time at work is a series of interruptions, which really isn't conductive to sitting down and learning something new and complex.

When I get home from work, I'm usually so fed up with computers that I don't want to have anything to do with them.

So my spare time, whatever is left of it, usually isn't spent on programming. Note that I don't even do these projects in a programming language I already know well; they are on hold regardless of that.

All in all, there's nothing much wrong with Perl 6.

Blaming the immaturity of Rakudo would just be a silly excuse. There's something wrong with my capacity for finding the time to get down and dirty with it, that's what; I'm apparently not currently capable of saying honestly:

This is my Perl 6 hour. This hour, I'm going to do Perl 6 stuff, and this time is sacred.

What the #perl6 IRC bots do

2009-11-09T00:49:00.005+01:00

Do you feel like a n00b on #perl6, like I do, and wonder what the different bots do?

I keep forgetting what they are, so here's a list for you and me both:

dalek

Announces commits (mainly to rakudo, nqp-rx and the Perl 6 book)

hugme

Used for hugging another user without "direct" contact:

hugme: hug masak

ilbot2

Near-realtime IRC logs with automatic link generation to irclog.perlgeek.de. The original ilbot sucked, according to moritz.

ilogger2

Another logging bot

lambdabot

Keeps track of karma ("moritz++" adds one to moritz's karma score, "frettled--" subtracts one from mine)

lisppaste3

Announces entries pasted to http://paste.lisp.org/new/perl6 (which is where we paste code and other stuff, so that we avoid spamming the channel too much, and also don't have to worry about creating our own temporary web pages)

masak

Submits rakudo bugs. Aw, okay, then, he's not a bot, just a really nice guy!

mubot

Also tracks karma, attempting to be slightly less annoying than lambdabot. mubot is clever enough to recognize that your nick may vary slightly from time to time and channel to channel. mubot is written in Perl 6!

p6eval

Perl 6 code evaluation bot. We use this for live testing of code that may be of interest to others; it chats back to the channel. perl6: my $a; will result in a test against several Perl 6 interpreters (elf, mildew, mildew-js, pugs, rakudo, sprixel), nqp: say('foo') tests nqp-rx, std: my $a will parse the expression with STD.pm.

phenny

Our secretary. Sample usage:

phenny, tell frettled I'll get back to you on that

phenny will then let me know when I become active on the channel again.

pointme

Provides links to projects tracked by proto. Example usage:

< carlin> pointme: rssbot
< pointme> carlins's rssbot is at http://github.com/carlins/rssbot

pointme is written in Perl 6!

pugs_svn

Tracks commits to the pugs repository, most of which are changes to the test suite and spec.

zaslon

Tracks blog posts from a certain group of bloggers. Zaslon is written in Perl 6!

Thanks to carlin, Juerd, jnthn and moritz for late night clarifications!

Checking and fixing Unix file modes

2009-11-03T19:33:00.000+01:00

Among other tasks in my sysadmin role at a web hosting provider, I work with fallout from poorly designed PHP code - which is ubiquitous - and I use Perl 5 to perform a bunch of semi-automated tasks.

If you just want to see how I utilize the Fcntl module, look further down.

One of the many things that tend to go wrong is the assumption that PHP always runs as mod_php (blatantly disregarding php-cgi, suphp, and other per-user PHP frameworks), and therefore directories (folders) and files used by PHP "must be" prepared with chmod 777 chmod 666. The latter number is a BIG FRIGGIN' HINT, it's the number of the beast.

Whenever documentation tells you to use the number of the beast for your chmod command, that documentation is also telling you to lube up and bend over.

Unfortunately, customers and users don't necessarily see this gotcha; no matter how good the hosting provider's documentation is, they'll naturally trust the software monger's instructions.

That means that the hosting provider ought to have tools available for identifying and fixing such writable directories and files. There are many wrong ways of doing it, one is:chmod -R, since that touches ALL files, recursively, overwriting the ctime stamp. An okay way is to use find, which (in most versions) allow you to fix things up quite neatly (bash/sh compatible syntax, GNU find compatible options, $dir represents the directory to fix recursively, $user is the username whose files you want to change):

find $dir -xdev -user $uname \
\( \( -perm /og=w -exec chmod og-w {} \; \) -o \
   \( -perm /g=w -exec chmod g-w {} \; \) -o \
   \( -perm /o=w -exec chmod o-w {} \; \) \)

Phew, that was quite a mouthful, but it's rather nice in resource usage, and it doesn't cross filesystem boundaries (-xdev).

So why would I want to do this in Perl, you ask?

"Eeerrr. Good question, let me tell you why!"

There are many other problems to look for, which it is sensible to look for while you're at it, just to mention a few:

Backdoors
Hidden IFRAMEs
Malicious JavaScript
Malware URLs
Malware redirects (e.g. in .htaccess)
Outdated software versions
Root exploits
Spamming scripts
Viruses

These things belong in a program, not a teeny weeny Unix one-liner, or even a set of them.

While you're at it, you might want to create a log of what you found, and perhaps which line numbers are relevant for which files, both for pointing out where to consider fixing things, as well as having something to use for debugging your false positives.

The code

Here's how I identify the files with too liberal write permissions, utilizing the Fnctl module. The filename is stored in $_, the user's real UID in $r_uid, and I also store various file information and what kind of file we're looking at.

use Fcntl ':mode';

my %badperms;
my ($dev,$ino,$mode,$nlink,
    $uid,$gid,$rdev,$size) = lstat($_);
my $g_write = $mode & S_IWGRP;
my $o_write = $mode & S_IWOTH;
my ($isfile,$isdir,$islink) = (-f _, -d _, -l _);

my $filetype = $isfile ? "File" : \
   $isdir ? "Directory" : \
   $islink ? "Symlink" : "Other";
my $fn = $_;

# Writable for others                                                    
if (!$islink && ($g_write || $o_write)) {
    $badperms{$fn} =
        sprintf ("%s: [%04o] %s\n", $filetype,
                 S_IMODE($mode), $fn);
}

In a future post, I'll try to get back with how this fits in my bigger picture of vulnerability detection.

As always, suggestions for improvements and questions are very welcome.

Small and cute shell and Perl scriptlets

2009-10-26T23:56:00.006+01:00

Once upon a time, the world was full of small and cute shell and Perl scripts that served a small but useful purpose, and which were shared freely - just because hackers were nice guys. So we bring some of these around with us, in slightly modified or improved versions.

I'd like to carry on the tradition of sharing some of these before they are lost, even though they are about as trivial as you can get.

Eval

Here's one that I picked up at the University of Oslo 10-15 years ago, which I still use, just out of habit. The file is traditionally named Eval, and I believe this particular version was concocted by Kjetil T. Homme (who has some other nice hacks, BTW). The idea is to have an easy-to-use command line calculator:

#! /bin/sh -

exec perl -e '$a = ('"$*"');
 if ($a == int($a)) {
   print $a, "\n";
 } else {
   printf ("%.3f\n", $a);
 }'

So what's the point?

For me, it's the ease of use, and not least the ease of installability. All I need is a Perl version that understands a very simple set of commands. I don't have to worry whether bc or dc is installed, I don't have to open a GUI, and I can work with fairly advanced expressions. They just have end up as valid Perl.

7

Did you ever forget exactly which octal code to use for the ASCII letter 'h', or the hex number for the backtick (`) character? 7 to the rescue!

#! /usr/bin/perl -C

for (32..126) {
    printf "%3d 0x%02X 0%03o 0b%08b %c\n", ($_)x5;
}

And if you still haven't made the transition to Unicode/UTF-8, printing the printables in e.g. a Latin character set may still be interesting; create a copy of the file called 8, which works on the range 160..255 instead. Or be fancy with basename checking and all that. :)

Answer

When I write scripts that do potentially Dangerous Stuff on production servers (that happens way too often), I usually like to include some code that tells the user to think a bit before continuing. That is, I don't want all scripts to be fire-and-forget. This is another classical piece of code, which I've massaged for my own needs (and perhaps your needs as well?):

print "About to run $code.\n\n";
print "Are you sure? (y/N): ";
my $answer = <STDIN>;
chop $answer;
if ($answer !~ /^y(es)?$/i) {
    die "Okay, aborting.\n";
} else {
    print "Okay, continuing!\n";
}

183 days of Iron Man blogging

2009-10-19T17:18:00.002+02:00

Today it's 183 days - more than half a year - since I started blogging, spurred on by mst's lightning talk at NPW 2009.

That ain't half bad.

But he's still not losing his bet yet, argh! :D

Coding styles that make me twitch, part 5

2009-10-19T16:45:00.002+02:00

Let us say that we have some code using DBI - old-fashioned, but it still works, kindof.

How would you like to see the following in a 6000 line CGI script you're supposed to debug?


    my $sth=&query("SELECT id FROM invoices WHERE invoiced='N'");
    my $invoiced=$sth->rows;

    my $sth2=&query("SELECT count(*) nusers FROM users \
LEFT JOIN people ... # long SQL statement
    my $row2 = $sth2->fetchrow_hashref;
    my $nusers = $row2->{nusers};

    my $sth3=&query("SELECT count(*) npeople FROM users \
LEFT JOIN people ... # long SQL statement
    my $row3 = $sth3->fetchrow_hashref;
    my $npeople = $row3->{npeople};

    my $sth4=...

And then, after carefully checking scope, you discover that the variables $sth2, $row2, $sth3, $row3, $sth4, $row4 etc. are not used anywhere else within the same scope.

Would you develop a tick?

I did, and the tick didn't lessen in strength when I discovered unsafe variable interpolation as well as a sub query that used $dbh->prepare($qtext) but didn't allow passing of usable parameters, such as, you know, bind variables.

I've started on "refactoring" the real life code this example is based on, but I get depressed from the sheer amount of work in fixing many problems at once. Maybe I should pick up drunken coding to become less perfectionist.

Use Digest::MD5 - it's easy

2009-10-12T00:54:00.003+02:00

In my previous entry, I presented and purposefully ignored a rather non-portable way of getting an MD5 sum from a file:

$md5 = `md5sum $filename.new | awk '{ print $1 }'`;
$md5 =~ s/\n$//;

There are stupider and better ways of doing this in the system call, but it completely ignores the problem that the command is not always called md5sum.

Digest::MD5 comes to the rescue!


use autodie; # Hee-hee
use Digest::MD5;
my $digester = Digest::MD5->new;
open(FH,"<$filename.new");
$digester->addfile(*FH);
my $md5 = $digester->hexdigest;

Okay, that looks slightly over-complicated, one might argue that Digest::MD5 should hide the file handle fiddling from the user. The value added comes when you have a chunk of data already in a variable, then you just do use Digest::MD5 qw(md5_hex); and call md5_hex($data).

Oh, and I snuck in something else again, didn't I.

Coding styles that make me twitch, part 4

2009-10-06T00:17:00.006+02:00

All too often, I see code like this:


$md5 = `md5sum $filename.new | awk '{ print $1 }'`;
$md5 =~ s/\n$//;
if ($md5 ne $origmd5) {
  system("mv $filename $filename.old");
  system("mv $filename.new $filename");
  …
}

Well, you get the idea.

Sure, Perl can pretend to be a glorified shell script, but there are perfectly well-functioning internal functions for these things.

Let's ignore the non-portability of the md5sum command for the time being, and also avoid shaving that awk call down with cut and an enclosing echo -n $(…) - we're here for the Perl, not the shell, right?

The above system() calls can easily be replaced with the following to save yourself a few unnecessary forks:


rename($filename,"$filename.old") and
rename("$filename.new",$filename);

Note the slight refinement of using and to avoid clobbering the file. But the documentation for rename() suggests that we use move() from File::Copy instead:


use File::Copy qw(mv);
mv($filename,"$filename.old") and
mv("$filename.new",$filename);

Similarly, there are nice built-in functions for many other frequent victims of system(), e.g.:


chmod
chgrp + chown
link
mkdir
rmdir
symlink
unlink

And of course there is a bunch of more or less helpful modules on CPAN (in addition to the already mentioned File::Copy), e.g. File::Path.

Contributing to Perl

2009-09-30T23:21:00.000+02:00

Inspiration is a big bother, at least when you don't have it.

I never felt that I had anything to contribute to the Perl community until the Nordic Perl Workshop this year, when I suddenly was involved in a small way.

I'm still involved in a small way, and only occasionally, but that works rather well.

In retrospect, my expectation that contributing would be a Big Deal turned out be wrong.

I don't have to solve all the problems in the world, and I don't have to solve the Big Ones, either.

It doesn't even take much of my time, and this level of contribution is one that suits many people.

Inspiration comes from the impression I have that there are many more people like me, and that all those little trickles of water we add to the pool are enough to keep the more active contributors going.

What is difficult, however, is to keep up with the Iron Man blogging challenge. I knew it would be when I got started, but I figured that there would always be something to write about without seeming too inane.

Today's post shows that I was wrong on that count. :D

I blame society

2009-09-23T22:30:00.003+02:00

This is only vaguely Perl-related.

This weekend, I visited Abigail and her spouse in their home in the Netherlands.

I know Abigail from several settings, and Perl is one of them; Abigail has taught me many cool finesses in Perl 5, which has saved me considerable time both personally and professionally.

The friendships and connections we make "virtually" can also be very strong IRL, and I feel that has been the case with us.

If there is a point to this post, then that is: visit your Perl friends!

Some ways that Perl 6 is grand, part 3 of ?

2009-09-13T21:55:00.006+02:00

I'll be really brief this time, I promise!

Several things about Perl 6 are there to save programmer time. Some of them even will do so. ;) This week's favourite is pure laziness. Let's imagine we have one of those lists with month names in them again, and for some odd reason only want to do something to each third month.

for ^12 :by(3) {
  say @months[$_];
}

I think that's just neat.

The caret notation is the "upto" operator, and shorthand for a range from 0 and up to its argument. So in the example above, we're asking for a range from 0 and up to 12.

The by adverbial is new in the specification. It denotes the increments for the upto operator, and allows e.g. real numbers, so that you can increment by 0.25 if it makes sense for your code to do so.

The above example then essentially iterates over 0, 3, 6, 9.

The by adverbial isn't yet implemented in Rakudo (well, duh, the spec just saw it added), where you have to settle for this for now:

for ^4 {
  say @months[$_*3];
}

… and something different if you want to increment by 0.25. I hope you don't want to increment through the list of months by 0.25.

Print-and-log in Perl 6

2009-09-06T21:28:00.008+02:00

In one of my early blog entries, "Simple print-and-log subroutine", I shared a small piece of code that has been a nice, every-day tool - in Perl 5.

Today, I set about converting that to a naïve Perl 6 version, and being quite the Perl 6 n00b still, I ran into a few hurdles along the way.

The hurdles were easy enough to avoid, once masak++ and moritz++ had bonked my head sufficiently.

I'll walk through the code, step by step, to illustrate what I learned today, and what others might need to know.

use v6;

Ah, first, remember this statement. It's a nice hint for other software if you want to continue using the .pl file suffix instead of .p6l etc.

my $verbose = 1;
my $logfile = "/tmp/test.log";

my $*PREFIX = "plog";

Variables starting with the twigil $* is what we call contextual variables, and they just started working yesterday in Rakudo. Contextual variables follow a dynamic call path. When we reuse this variable later, it will depend on which call path we followed.

sub plogwarn ($msg)
{ 
  my $*PREFIX = "plogwarn";

$*PREFIX now has a different value only for the cases where we've called the plogwarn subroutine, and the subsequent call to plog inherits this value.

  plog $msg,1;

Oh, BTW, it's really, REALLY important to keep track of where you add whitespace or not. I'd been sleeping and class, and forgotten completely that plog($msg,1) would call plog with the parameters $msg and $1, while plog ($msg,1) would call plog with the single parameter ($msg,$1) (yep, a list). It may be safer to avoid parentheses altogether when you're not dealing with lists - except when you have to.

}

sub plogdie ($msg)
{ 
  my $*PREFIX = "plogdie";
  plog $msg,2;
  exit 1;
}

sub plog ($msg is copy, $level?)

The trait is copy means that I want to make a copy of the incoming parameter $msg, so that I can modify it inside of plog. Normally, parameters in Perl 6 are read-only and pass-by-reference (though not "reference" as you know it from Perl); the parameter as named is merely an alias for the actual variable. I can change the parameter in the caller if I use is rw, but that's not what I want to do here. The question mark in the second parameter - $level? - means that the parameter is optional, and I've used that in both plogwarn and plogdie above.

{ 
  my $dt = Time.gmtime.date.iso8601
         ~ " "
         ~ Time.gmtime.time.iso8601;

Normally, I'd use localtime, but this is NYI (not yet implemented) in Temporal.pm.


  $msg = ($dt,"[$*PREFIX]",$msg).join(" ");

Here, I abuse that is copy to save myself one precious variable, just as an example.


  if $verbose {
    given $level {
      when 1 { warn $msg; }
      when 2 { die $msg; }
      default { say $msg; }
    }

The given…when construct is similar to the switch…case construct in other languages, but subtly different, as it allows more possible values and value types than most. Coming from a world of Perl 5.8 and older, this is simply lovely.

  }
  # Append message to $logfile
  my $*OUT = open $logfile, :a;

Here's another use of a contextual variable, but this time it's the one normally associated with STDOUT. print FILEHANDLE $msg is no longer necessary, because you can always assign $*OUT in its own scope. The open statement has :a to indicate that I'm opening for append (so the syntax is less unixy than Perl 5's >>).

  say $msg;
  $*OUT.close;
}

This is how we can use the three subroutines, and the output should illustrate the different contextuals nicely:


plog("Oh, hello, there, sweetie.");
plogwarn("Consider yourself warned");
plogdie("Die, frackin' monster, die!");

…outputs…

2009-09-06 19:59:32 [plog] Oh, hello, there, sweetie.
2009-09-06 19:59:32 [plogwarn] Consider yourself warned
2009-09-06 19:59:32 [plogdie] Die, frackin' monster, die!

And here's the complete example code, which works with the current Rakudo:

use v6;

my $verbose = 1;
my $logfile = "/tmp/test.log";
my $*PREFIX = "plog";

sub plogwarn ($msg)
{ 
  my $*PREFIX = "plogwarn";
  plog $msg,1;
}

sub plogdie ($msg)
{ 
  my $*PREFIX = "plogdie";
  plog $msg,2;
  exit 1;
}

sub plog ($msg is copy, $level?)
{ 
  my $dt = Time.gmtime.date.iso8601
         ~ " "
         ~ Time.gmtime.time.iso8601;

  $msg = ($dt,"[$*PREFIX]",$msg).join(" ");

  if $verbose {
    given $level {
      when 1 { warn $msg; }
      when 2 { die $msg; }
      default { say $msg; }
    }
  }
  # Append message to $logfile
  my $*OUT = open $logfile, :a;
  say $msg;
  $*OUT.close;
}

Some ways that Perl 6 is grand, part 2 of ?

2009-08-31T23:17:00.009+02:00

Okay, this is really part 1b of ?, but…

In my earlier post, I used the zip operator to join two lists into a hash.

There was one obvious use of the operator that escaped me at the time, and that was how I sometimes need to create a new hash from the keys of two hashes, or keys and values. And now I think it's starting to look neat:

my %A = { a => 1, b => 2 };
my %B = { z => 9, y => 8 };

my %AB = %A.keys Z %B.keys;
# { "a" => "z", "b" => "y" }

%AB = %A.keys Z %B.values;
# { "a" => 9, "b" => 8 }

However, this is a bit unpredictable, since the hash key order is undefined. So if you expect sorted keys, do that at the same time:

%AB = %A.keys.sort Z %B.keys.sort;
# { "a" => y, "b" => z }

# Sort by B's values - two variants
%AB = %A.keys.sort Z map { %B{$_} }, %B.keys.sort;
%AB = %A.keys.sort Z %B.sort.map( { .value } );
# { "a" => 8, "b" => 9 }

The equivalent Perl 5.10 version would be:

use List::MoreUtils qw/zip/;
my @k = sort(keys(%A));
my @v = map { $B{$_} }, sort(keys(%B));
%AB = zip @k, @v;

I now have a nice-ish argument for upgrading to Perl 5.10.1 on $workplace's servers. :D

masak++ for helping a tired me with the map expression.
Chas. Owens++ for spotting the missing use statement for Perl 5.
Pm++ for another way of sorting by value, just what I was hoping for!
isec++ for spotting a missing sort() for Perl 5.

Autovivification - a reminder

2009-08-23T22:52:00.007+02:00

Most of you already know this by heart, but the odd reader may have forgotten.

Autovivification is what we call the process of automatically creating entries in built-in data structures (Perl 5: array/list and hash), usually at the time we check whether an inner element exists or not.

This can be a royal PITA, if you don't pay attention to the problem. That's why it keeps being mentioned.

Here's a simple example:


my %hash;
my $n;
while (!exists ($hash{x}) && $n < 5) {
    $n++;
    if (!exists ($hash{x}{y}) {
        print "hash{x}{y} does not exist: $n\n";
    }
}

Q: How many times does the above while loop run in Perl 5?
A: Once.

The simple matter of checking the existence of the inner hash resulted in an entry being created for $hash{x}.

That means that tests like these should be written more carefully:


…
    if (defined ($hash{x})) {
        if (!exists ($hash{x}{y}) {
            print "$n: hash{x}{y} does not exist.\n";
        }
    } else {
        print "$n: hash{x} is undefined.\n";
    }
…

This prints $n: hash{x} is undefined. (with an incrementing $n) five times.

Edit 2009-08-24 18:49 UTC: MST commented that there is an autovivification module on CPAN that lets us say no autovivification; - and it's even lexically scoped! That's just $notreallyanexpletive brilliant! Thanks, Matt, and thanks, Vincent!

Oh, BTW, Perl 6 has a useful specification for autovivification, which illuminates the problem further.

Hash key sort order - a Perl 6 community sunshine story

2009-08-16T11:36:00.004+02:00

The order of hash keys is implementation dependent and arbitrary. Unless %hash is altered in any way, successive calls to .keys, .kv, .pairs, .values, or .iterator will iterate over the elements in the same order.

S09 - Hashes

This is new. It may not actually say much, but it does say what was implicit before, so that there is little room for doubt.

So here's my little sunshine story about how easy it is to clarify a part of the spec.

In my previous post, I used an imaginary case for showing off some features of Perl 6 - some of which also are available in Perl 5.10, as mentioned by Robert 'phaylon' Sedlacek in a comment.

This wasn't the only useful comment, I think there's a bit to be learned by reading those, so please do.

But I digress from the point of this post, which is a question that was raised in another comment to last Sunday's post:

By the way, do you know where the behavior of ~%h is spec'ed? I keep getting the keys back in the same order I put them in and don't know if that is an implementation quirk or a feature.

- Chas. Owens

The spec wasn't very clear about this; S32/Containers - Hash said that certain iterator methods iterate "… the elements of %hash in no apparent order, but the order will be the same between successive calls to these functions, as long as %hash doesn't change."

S09 - Hashes didn't say anything about it at all.

I said I would ask around. Thanks to the excellent community channel #perl6 on Freenode, I got an answer similar to this: no, this is unspecified/undefined behaviour, but feel free to come up with a better way of saying it, and update the synopses.

And how hard is it to update the synopses? Not at all! If we want to contribute, we get access. It's as easy and simple as that.

First, you need to check out the (part of) the svn repository that you want to contribute to:

svn co http://svn.pugscode.org/pugs/docs/Perl6

Then you change whatever you want to change, preferably discuss it with some of the experienced souls on #perl6 or e.g. the Perl 6 language mailing list, show diffs on e.g. gist.github.org or paste.lisp.org, and if you think you're doing the right thing - commit the change.

"But I can't commit, I only have read access" you might say. Just ask in any of the mentioned fora for a "commit bit" and state your e-mail address and preferred username, and someone will help you out with that part.

As we say on #perl6: community++

I got started, will you join me?

Some ways that Perl 6 is grand, part 1 of ?

2009-08-09T23:58:00.013+02:00

(After YAPC::Europe in Lisbon, we no longer say "awesome", we say "grand". ;))

In Lisbon, there were several talks that aided us to a better understanding of how Perl 6 may be more pleasant and useful than Perl 5.8, or even Perl 5.10.

I thought I'd try to illuminate some of these as I progress in my own knowledge of Perl 6.

First choice: the zip operator and the new quoting syntax for generating lists and selecting items from hashes:


my @months = <Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec>;
my %days = @months Z 31,28,31,30,31,30,31,31,30,31,30,31;
say "Days in March: " ~ %days<Mar>;
say "Days in June: {%days<Jun>}";
say ~%days;

That was quite a bit in one go. :) First, I create a new list of abbreviated month names. Then I combine the list of months with a list of days in that month (for 2009, obviously) using the zip (Z) operator, and print the number of days in March and June. Finally, I pretty-print each month with the corresponding number of days.

The result looks like this:


Days in March: 31
Days in June: 30
Jan     31
Feb     28
Mar     31
Apr     30
May     31
Jun     30
Jul     31
Aug     31
Sep     30
Oct     31
Nov     30
Dec     31

Now imagine what that would have to look like in plain Perl 5.

There are several new things here that makes your programming days easier:

The well-known sigils $, @ and % are now used in a consistent manner. @ signifies a list, % signifies a hash, also when you access items by index.
Interpolation of non-scalars is now handled with curly brackets {}.
Building a list can be done using commas, no parentheses are necessary.
Building a quoted list can be done using angle brackets, which are auto-quoting.
Accessing a hash item is done using angle brackets, which should be welcome to those of use who don't use US/UK keyboards (bye-bye to {'argh'}, but it still works).
say is a new pretty-printing friend, which automatically adds newlines.
The concatenation operator is now tilde (~) instead of a period.
As a unary prefix operator, tilde by default stringifies a hash pair-wise with newline separators (while a list is stringified with space separators). It is possible to change this behaviour in derived subclasses.

Edit 2009-08-10 08:34-09:18 UTC: Clarified and added interpolation.

Mini report from YAPC::Europe 2009

2009-08-04T19:44:00.002+02:00

The second day of the conference is now finished.

Yet again, the lightning talks have been a great source of entertainment, but the best regular entertainment were probably Paul Fenwick's and Damian Conway's sparring ... in Klingon.

If you can lay your hands on the video of their respective talks, see them in order, and enjoy the show.

You might also learn some really interesting Perl as well, or fall deep into the pits of insanity.

YAPC::Europe 2009

2009-07-31T00:43:00.003+02:00

Here we go again!

Thankfully, my employer thought that this year's YAPC::Europe (Corporate Perl) could be relevant for me, and so I'm going, hoping to learn yet more from all the excellent minds present there (modulo swine flu, I suppose ...).

The schedule contains way too many interesting subjects at the same time, but I'm happy to say that there's plenty of breathing room and time for socializing.

And this time, I'm bringing a computer, so that I can pretend to be doing something useful. Haha.

See you there!

Perl, open source and gender bias

2009-07-27T23:30:00.004+02:00

Recently, I have been reading a discussion regarding gender bias in the tech industry.

I realize it is a problem, but for the past years, it has been more of an intellectual realization than something that hit me in the gut. Perhaps it has something to do with the work place culture where I have been working, combined with the informal technical fora I have seen, as well as the general state of things here in Norway.

However, these things are perhaps not talked about quite as much as they should, and today I re-learned that although things may seem mostly rosy, they are not necessarily so.

Kirrily "Skud" Robert has been active in various open source contexts for years, and should be no stranger to the Perl community. She recently held a presentation at OSCON -- Standing Out in the Crowd -- where she discussed some of the challenges when women are in minority, and experiences with open source projects where they are in majority.

What kicked me in the gut this time was the rather low percentage of women involved in Perl. While better than open source in general, it seems excessively low.

But Skud does not sling mud at the Perl community. Instead -- if I understand her correctly -- she aims to help open source communities -- and I suppose particularly new projects -- to utilize the resourcefulness of women in development, to reduce the gender bias by increasing the overall activity.

I can only recommend that you read the presentation, and contemplate the issue for yourself.

It certainly made me think. Again.

A comment on sandboxing and web security

2009-07-21T00:36:00.003+02:00

In "What I Want In Firefox (Parrot)", Ovid expresses his desire for Perl 6 scripting functionality, with the caveat "if the sandbox is secure enough".

Aha, there's your problem.

The Firefox developers clearly didn't have sandboxing in mind when they designed the browser, scripts are essentially free to do what they wish at least with the DOM, as well as access many central browser functions, plus a bunch of things that we really don't want security vulnerabilities for.

Another problem is that JavaScript is fundamentally entrenched in Firefox; too much of Firefox internals are based on JavaScript, and I suspect that a Perl 6 addition would only work so-so at best, even if that hypothetical secure sandbox existed.

Oh, and BTW: the example commits one of the great no-nos of web programming: pushing server side security into the browser, which essentially is no security at all, as seen form the server's point of view.

Why am I being so negative all of a sudden? Well, it might be sudden for this blog, but I've been working with security related cleanup and detection in the context of Linux system administration for a few years now. It's not enjoyable, I can tell you, and it definitely colours my perception of these things. Also, I've had a fair bit to do with thinking about security earlier.

Recommended reading (that should be required) for web programmers:

Innocent Code by Sverre H. Huseby (a buddy of mine), ISBN: 978-0470857441

Iron Man going strong

2009-07-17T00:04:00.002+02:00

90 days on ...

English, German, Hebrew, Japanese, Magyar, Perl 5, Perl 6, Russian, ...

We're positively flourishing here!

When mst announced the Iron Man concept in his lightning talk at NPW 2009, I felt inspired, but I didn't expect it to work this well.

There are many extremely interesting blogs out there, sharing knowledge and experience at a level I definitely didn't expect.

It feels unique, it's wonderful, and I hope this will continue snowballing!

Thanks, guys!

Coding styles that make me twitch, part 3

2009-07-08T00:49:00.004+02:00

Today's twitchiness is sponsored by ... no, wait, I don't have sponsors. Ah, well.

I have an issue with people who insist on using <"> as a string delimiter when the (static) string itself contains that very character. It gets fugly all too soon:

my $html_output = "<a href=\"http://www.example.com/foobar/$pagename.html\" title=\"Oh, a link to $pagename\"> ...\n";

It's so easy to avoid having to quote the <"> while still allowing variable interpolation:

my $html_output = qq(<a href="http://www.example.com/foobar/$pagename.html" title="Oh, a link to $pagename"> ...\n);

Since the example is HTML (and could be e.g. SQL), and it might be multi-line, why not ...

my $html_output = <<EOL;
<a href="http://www.example.com/foobar/$pagename.html"
   title="Oh, a link to $pagename>
EOL

That wasn't so hard? Or fugly?

Ruby users anonymous

2009-07-02T00:09:00.007+02:00

Last weekend, I visited a couple of friends -- Ina and Stig -- and being nerdy geeks, we (well, Stig and I, anyway) ended up chatting about programming and stuff.

Stig has just started programming Perl again, after a long period where he'd probably claim to be apostate.

Everybody, meet Stig. Stig used to program Ruby, but now he's recovering.

Edit: Stig's blog address injected.

I hope this shows that there's hope for the ones who got lost.

Virus scanning with File::Scan::ClamAV

2009-06-24T09:58:00.006+02:00

This is almost ridiculously easy.

Problem: a bunch of user directories need virus scanning and per-user reports

Solution: A Perl script using File::Scan::ClamAV

Prerequisites:

A unixy OS
Perl 5.8 or 5.10
A functional and running clamd, preferably listening to a socket
The module File::Scan::ClamAV (and its dependencies)

The following code could (after some sensible adjustments) run in a loop through all usernames on your system.


use File::Scan::ClamAV;
# (...)
# $dir contains the full path of the user's directory
my $av = new File::Scan::ClamAV (find_all => 1,
                                 port => '/tmp/clamd.socket');
# find_all means that we wish to recurse directories.
# /tmp/clamd.socket is where my clamd has its socket.
# Other clamd configurations may differ.

unless ($av->ping) {
    plogdie "clamd isn't running, aborting virus scan";
} else {
    plog "Performing virus scan for $uname";

    # Save virus information per username ($uname).
    # Note! scan() returns a hash.
    $a_viruses{$uname} = $av->scan($dir);

    if ($a_viruses{$uname}) {
        my @vfiles = sort keys %{$a_viruses{$uname}};
        plog "$uname has ".@vfiles." viruses.";

        # Home assignment: print contents of $a_viruses{$uname}
        # to a file, using the sorted list @vfiles.
    }
}

Frequently freaky freakin' one-liners

2009-06-15T14:01:00.006+02:00

So, hey, I'm sitting here without anything good to blog about, probably like most people on the net.

I'm wondering what daily Perl usage that's even vaguely useful that I do, which could be improved upon.

Ah, of course, triple-f one-liners!

As a tool, the perl command often seems to replace a jungle of echo + egrep + cut + tr + sed + awk and whatnot. perl -nawe and ctrl+r (reverse i-search) in bash are good friends of mine, but after using the same one-liners a few times in a row, I usually end up converting them to tidy files with Getopt::Long, comments and other insanities.

And at some stage later, I say to myself: damnit, I should've coded this more generally, I start a recode, get distracted, solve a new problem with one-liners, and the circle of life goes on.

Do I need professional help?

DateTime performance hit

2009-06-08T12:31:00.011+02:00

This is mainly in answer to the comment to my previous post on revisiting plog.

plog is a piece of code ready for copy+paste into whatever codebase you're in at the moment, and almost completely agnostic. Yes, in code where I'm already using DateTime, then I use those routines.

You could also create a small sub to hide the "nasty idiom".

Or you could use Date::Format instead (ca. 10 kB codebase instead of 110 kB, and between one and two orders of magnitude quicker for this particular purpose):

require Date::Format;
@lt = localtime();
$dt = strftime("%Y-%m-%d %T",\@lt);

Of course, that leaves you with the problem of whether the TimeDate packages are installed or not, and another test for that.

Besides, the bloat is not exactly insignificant if you have code that's running repeatedely.

The following example with 100,000 repetitions may seem ludicrous, but I actually have production code where timestamped log entries run into that order of magnitude. And yes, I would very much like to save that extra time.

Edit 2009-06-09 00:32 UTC: thanks to Dave Rolsky for the simplified testing code, and to Ilmari for reminding me of POSIX::strftime, which I'd previously rejected based on other people's claims that it was dog slow. I've removed the home-grown tests in favour of the results from a slightly modified version of Dave's sample test script.

Here are the numbers for 100k iterations, times derived from the rates:

Solution	Time	Rate
DateTime	490.2 s	204/s
DateTime (cached tz)	76.7 s	1 304/s
Date::Format	6.4 s	15 528/s
POSIX	3.3 s	30 030/s
localtime	0.6 s	163 934/s

Print-and-log revisited

2009-06-07T23:24:00.005+02:00

A month ago, I made a post with a simple print-and-log subroutine called plog.

I was recently asked two questions about this piece of code, and I'll answer them briefly now:

"What's up with the curly braces on a separate line after the sub declaration?"
"Why don't you use DateTime?"

Okay, those should be easy to answer while retaining the illusion of clue:

That's merely a personal preference, it quickly aids me in noticing whether something is a subroutine or a control statement block. I am aware that many Perl programmers disagree, and prefer all blocks to start in the same way regardless of what kind of block it is.
DateTime is an external module which, believe it or not, is not installed on all systems. Some people would also argue that it's bloated and slow. But in case you want to use DateTime instead, and/or check whether it's available run-time, then ... well, see below.

sub plog
{
  my $msg = shift;
  my $level = shift;
  my $dt;
  eval {
    require DateTime;
    $dt = DateTime->now(time_zone=>'local')->strftime("%F %T");
  };
  if ($@) {
    my @lt = localtime;
    # Format current datetime manually:
    $dt = sprintf("%d-%02d-%02d %02d:%02d:%02d",
                  $lt[5]+1900,$lt[4]+1,
                  $lt[3],$lt[2],$lt[1],$lt[0]);
  }

(...)

As you can see, there isn't much code saved by using DateTime, even if we know it's already installed and don't need to add paranoia. The method of extracting data from localtime() is well-known, proven, and fairly nice on resources. Use DateTime if you want to, but perhaps it's best to save it for when you need to do more complicated stuff.

Coding styles that make me twitch, part 2

2009-05-30T01:02:00.003+02:00

I'll make today's post short, because it's all about line length.

Limiting the length of code lines is something I try to be good at, not because I think the next guy will have a VT100 terminal and needs a friendly piece of code, but because of basic readability.

When we read an ordinary text, e.g. in a book (you remember books, right?), there is usually quite few characters printed on each line. Here are a few from Douglas Coupland's JPod:

the door to see that all my new furniture was gone, and my
original furniture hadn't come back. Fuck. I phoned Greg,
but realized he was on Cathay Pacific 889, headed to Hong
Kong. I phoned Mom.

Notice how the text area is even narrower than that in this friendly blog?

This comes from a long tradition in printing, it isn't as if they couldn't have squeezed twice as many words in there, if the print was only smaller. But if the print was smaller, they'd probably make the book narrower as well.

It's far easier to read something when you don't have to move your eyes around too much from line to line. This is important to both slow and quick readers.

So, back to coders who make lines of 100+ characters:

What in friggin' Ruritania are you thinking? Not much, that's what.

*GROWL*

Querying quotas with Quota.pm

2009-05-22T22:55:00.002+02:00

I never claimed that this blog would be an exercise in expertise. ;)

Prerequisites: Unixy OS with quotas enabled, Perl 5.8.8, Quota.pm 1.6.3.

I always like to provide some sort of progress display for my programs. As a sysadmin, there are times when it might be prudent to check users' files without inspecting them by hand, e.g. when checking for parasitical exploits in websites. The number of files and/or amount of disk space used seem like reasonable measurements for keeping track of that progress.

So you use Quota; and get coding, right?

Except that you may not know beforehand whether you're scanning a local file system, or a remote file system, and Quota.pm requires that you have a magical device identifier before asking what the quota is.

The manual says that you should do something like this:


my $r_uid; # User's real UID
my $dev = Quota::getqcarg($directory);

my @quotadata = Quota::query($dev, $r_uid);

Now we've got some nice quota data, in the following order:

Current blocks used, block soft limit, block hard limit, soft block time limit, current inodes used, inode soft limit, inode hard limit, soft inode time limit.

But no, there's a catch! If you run this on the local file server, rather than via NFS/RPC, then Quota::query() will barf, because $dev is erroneous. How did that happen?

Well, Quota::query() doesn't work if the device is local!

So we have to do this after calling Quota::getqcarg():


if ($dev =~ m{^/dev/}) {
    $dev = "127.0.0.1:$directory";
}

The irony is then that there appears to be a need for an RPC listener on the loopback device, at least.

Anyway, I hope this is useful; it helped me to make tings Just Work.

Coding styles that make me twitch, part 1

2009-05-15T00:51:00.008+02:00

We'll see how long this particular series gets.

I'll try to come up with some example of coding styles that annoy me, and post about it.

First off is the appended conditional at the end of long one-line Swiss knife code snippets:


my @var = sort { length($b) <=> length($a) } split /[-.,_+ ]/ , $input{longvariablename} if defined $input{longvariablename} && length $input{longvariablename} > 4;

(Yes, that's supposed to be one line, though it doesn't look like it.)

Please, pretty please, don't append conditionals at the end of long one-liners.

Really, just don't, m'kaaay.

Code should, unless it's a one-off one-liner in your shell prompt, be maintainable for others. "Others" includes yourself some time in the future, when you've forgotten what the (insert mst-inspired expletives here) you were thinking at the time you coded the stuff.

The above example isn't particularly complex, or difficult to understand, but it's all on one line, and hardly is easy to parse even if you've got that 170 char wide window to code in.

A few parentheses, a helper variable and a few more lines -- preferably keeping well within 80 columns -- surely won't hurt that badly.


my $lvn = $input{longvariablename};
if (defined $lvn && length($lvn) > 4) {
    my @var = sort { length($b) <=> length($a) }
              split (/[-.,_+ ]/, $lvn);
}

Of course, these are just my personal opinions, and I won't be knocking on your door with a baseball bat in hand if you don't do as I suggest.

Simple print-and-log subroutine

2009-05-07T03:41:00.006+02:00

I find that I have a use for this almost all the time. It's a silly little set of subs, but in my role as sysadm, I often need to go back and see what all those printed messages were.

So here is my not-so-elegant workhorse for when I need to stuff things into logs, and shuffling modules isn't an option. I hope it's useful for someone else, too.

Dependencies, assumptions and prerequisites:

Perl 5-ish
Pre-defined global variables:
- $level - log level (undef = normal, 1 = warn, 2 = die)
- $logfile - a logfile that we can append to
- $msgprefix - a program or subroutine specific prefix
- $verbose - whether to print to STDOUT
Preferably disabled output buffering

Usage:

&plog("Log this");
&plogwarn("Warn about and log this");
&plogdie("Log this and die");

The code:


sub plogwarn
{
   my $msg = shift;
   &plog ($msg,1);
}

sub plogdie
{
   my $msg = shift;
   &plog ($msg,2);
   exit 1;
}

sub plog
{
   my $msg = shift;
   my $level = shift;
   my @lt = localtime;
   # Format current datetime sensibly:
   my $dt = sprintf("%d-%02d-%02d %02d:%02d:%02d",
                    $lt[5]+1900,$lt[4]+1,
                    $lt[3],$lt[2],$lt[1],$lt[0]);
   warn "$dt $0: sub plog: No message!\n" unless defined $msg;
   unless (open(F,">>$logfile")) {
       warn "$dt $0: sub plog: Failed to open logfile ($logfile) for write.\n";
   } else {
       print F "$dt $msgprefix$msg\n";
       close F;
   }
   if ($verbose) {
       unless (defined($level)) {
           print "$dt $msgprefix$msg\n";
       } elsif ($level == 1) {
           warn "$dt $msgprefix$msg\n";
       } elsif ($level == 2) {
           die "$dt $msgprefix$msg\n";
       }
   }
}

Schwartzian Transform - Perl 5 vs. Perl 6

2009-04-30T12:02:00.002+02:00

This isn't quite news, but it's a cool little bit of code anyway.

Perl 5:


@sorted = map  { $_->[0] }
          sort { $a->[1] cmp $b->[1] }
          map  { [$_, foo($_)] }
               @unsorted;

Perl 6:


@sorted = @unsorted.sort: { .uc };

I'm willing to claim that Perl 6 makes this a bit more readable, in spite of the smoke, mirrors and curtains.

Read more about the Schwartzian Transform in Wikipedia.

Perl 6 - how to get started

2009-04-27T01:02:00.008+02:00

Are you curious about Perl 6, and wonder how to get started?

Use proto, Carl Mäsak's Perl 6 installer (which will download parrot and rakudo for you).

Just make sure you have Perl 5 (5.8.8 or 5.10.0), git and svn installed first!

jani@knuth ~/prog/proto >./proto

*** CONFIG FILE CREATED ***

Greetings. I have created a file 'config.proto' that you may want to review.
Next time you run './proto', these settings will be used to bootstrap your
system into a working Perl 6 installation.

If you're new to this, or if configure settings scare you, you probably want
the default settings anyway. The most important ones are:
Rakudo   -> /ping/knuth/home0/jani/prog/rakudo
Projects -> /ping/knuth/home0/jani/prog

jani@knuth ~/prog/proto >./proto
Downloading Perl 6...downloaded
Building Perl 6...

This part may take a while; parrot is now building stuff for you, and when it's finished, you can run the perl6 binary:

Building Perl 6...built
jani@knuth ~/prog/proto >cd ../rakudo
jani@knuth ~/prog/rakudo >ls -F
CREDITS       MANIFEST  Test.pm  parrot/  perl6.o    perl6_s1.pbc  tools/
Configure.pl  Makefile  build/   perl6*   perl6.pbc  src/
LICENSE       README    docs/    perl6.c  perl6.pir  t/
jani@knuth ~/prog/rakudo >./perl6
> sub sum (*@numbers) { return [+] @numbers; }; say +(sum <1 2 3>)
6

See? Easy! Now get testing!

Enlightened Perl Iron Man Competition

2009-04-21T23:54:00.003+02:00

Well, if this isn't inspiring ...

This week, I hope to dedicate some time to what I said I was going to do at the NPW 2009 Hackathon, which was to spec complex number representation and presentation.

Thanks to pmichaud, I've at least confirmed my suspicions on which parts of the spec to do my changes in.

Maybe I fail, maybe I don't, but I think Perl 6 is definitely worth the effort.

Group Photos

2009-04-19T15:40:00.003+02:00

LR:
ilmari, nothingmuch, claes, szabgab, krunen, ingy, baest, masak, TimToady, Ellen\
trafl, jrockway, batman, pnu, jnthn, pmichaud, abigail, Gloria\
mst, sjn, mberends, marcus, sadrak
Photographer: frettled

And, finally, as prompted by Ingy:

NPW 2009 Hackathon

2009-04-19T12:17:00.000+02:00

The NPW 2009 Hackathon is well into its second day, and I've learned quite a bit about Perl 6 and more about Perl 5 than I expected.

I didn't originally intend to participate in the Hackathon, and I'm not doing much, but it's definitely worth it.

Yesterday, I stated a goal of adding to the spec a description of how complex numbers should be represented and presented, since that was apparently at least partially unclear.

This lead me into a quagmire of other things I needed to do first, and the "ooh, shiny!" phenomenon lead me astray a few times, ganged up with my general desire to have a functional working environment (Unicode strings in PODs didn't go down well with my current Latin 1-based working environment and the then-installed Perl version), as well as problems getting irssi on my side
But I did get to participate and act as distraction in an entertaining and useful discussion regarding types in Perl 6, as well as host a social dinner for those who wanted a break from the hackathon.

Today, I'm picking up where I left off, trying to form a mental picture of the spec that's good enough to add and/or change relevants bits of it.

The rest of the guys are adding code and doing other useful things. :)