Recently, I wanted to create a local testing environment for websites that are normally deployed on the Apache HTTP Server. While there are already some nice plack-based local web servers, I had a laundry list of my own requirements, and nothing quite fit the bill.

In order to solve two of my issues, I created and released two new Plack middleware modules to CPAN.

Plack::Middleware::DirIndex::Htaccess

In a standard Apache setup, developers often define default pages other than index.html using the DirectoryIndex directive inside a .htaccess file:

$ less gallery/.htaccess
DirectoryIndex gallery.html

This module is an extension of Plack::Middleware::DirIndex. But first it checks for an .htaccess file and, if it has a DirectoryIndex directive, serve that file as the default. Otherwise, it will serve the global default page if present, which by default is index.html.

Plack::Middleware::DirListing

When a directory doesn’t have a default index page, the web server usually generates a file listing. There are Plack implementations for this, including Plack::App::Directory and Plack::App::DirectoryIndex. But these act as standalone apps, and I wanted a middleware solution that would only generate a directory listing and not serve any other files.

If the request is a directory, Plack::Middleware::DirListing generates a rich, HTML-formatted listing of the directory contents, complete with last modified dates, file sizes, and file types. It also includes sorting functionality (by name, date, etc.).

Example Static Site Web Server Implementation

By combining these two modules, you can build a app.psgi that effectively emulates an Apache server for static sites. This example also includes Plack::Middleware::SSI as I (still) occasionally use server-side includes. For directories, it checks for a directory index file first first and falls back to a directory listing if none are found. Otherwise, it serves static files if requested.

#!/usr/bin/env perl
use strict;
use warnings;
use utf8;

use Plack::Builder;
use Plack::Request;

my $app = sub {
  my $env = shift;

  my $req = Plack::Request->new( $env );

  my $root = '.';
  my $status = '200';
  my $response = 'OK';

  if (! -e "$root$env->{ PATH_INFO }") {
    $status = '404';
    $response = 'Not found';
   }

  return [
        $status,
        [ 'Content-Type' => 'text/html; charset=utf-8' ],
        [ $response ]
        ];
};

builder {
        enable 'SSI';
		enable 'TrailingSlash', ignore => [ '.htaccess' ];
		enable 'DirIndex::Htaccess';
		enable 'DirListing';
        enable 'Static', path => qr/(css|eot|gif|heic|html|ico|jpe?g|js(?:on)?|pdf|png|svg|swf|txt|ttf|webp|woff2?|xml)/, root => '.';
        $app;
}

Available on CPAN

These modules are available now. If you are looking to mirror your Apache environment locally using the power of Perl, give them a try:

• Plack::Middleware::DirIndex::Htaccess

• Plack::Middleware::DirListing

My philosophy for programming tools is to try to keep things as simple and transparent as possible. I often find that modern package and version managers, while powerful, add layers of complexity I don’t need. I rarely require multiple versions of a programming language on the same machine, and I don’t want to hunt for where my binaries and libraries are located. Instead, I prefer a stable, predictable architecture where I know exactly where to find everything.

Prerequisites

This guide assumes you have a basic command-line development environment set up on your Mac. Specifically, you’ll need the Xcode command line tools and Homebrew, as we’ll use Homebrew to install a couple of dependencies.

If you’re starting from a fresh macOS installation, I recommend following my other guide first: Prerequisite Programming Tools for macOS: Xcode, Homebrew, and Bash.

Once you have Homebrew installed, you’re ready to proceed.

The Golden Rule: Don’t Touch System Perl

Before we start, here is the most important rule for using Perl on macOS, which I have learned the hard way:

********************************************
** Do *not* use the system Perl on MacOS. **
**     It is for Apple, not for you.      **
********************************************

Apple includes a version of Perl with macOS for its own system scripts and internal use. You should treat it as part of the operating system, not as a development tool. If you try to install modules to it or modify it, future macOS updates could overwrite your changes or, worse, break system functionality that depends on it. Leaving it alone is the safest and smartest path.

Why Not Just Use Homebrew?

The most common way to install packages on macOS is with Homebrew. So, why not just run brew install perl?

In my experience, installing Perl via Homebrew creates a complex web of symbolic links and version-specific directories (e.g., /opt/homebrew/Cellar/perl/5.38.2/...). This makes it difficult to have a stable, version-agnostic path for my scripts and libraries. Upgrading the Perl version via Brew can sometimes feel like a major migration.

Because Perl is famously backward-compatible, I don’t feel this level of compartmentalization is necessary for my workflow. I prefer to build it from source and install it directly into /usr/local/, just like in the old days. This gives me one predictable location for the Perl binary (/usr/local/bin/perl) and its libraries.

The Installation Process

Here are the step-by-step instructions for downloading and compiling Perl from source.

Step 1: Install Prerequisites

We’ll need wget to download the source code. If you don’t have it, you can install it with Homebrew.

brew install wget

Step 2: Download and Unpack the Perl Source

Navigate to your Downloads folder, grab the latest stable Perl source code from CPAN, and unpack it.

# Go to your downloads directory
cd ~/Downloads

# Download the source (check cpan.org for the latest version)
$ wget https://www.cpan.org/src/5.0/perl-5.40.0.tar.gz

# Unpack the archive
tar -xvzf perl-5.40.0.tar.gz 

# Change into the new directory
$ cd perl-5.40.0/

Step 3: Configure the Build

This is the most important step. We’ll run the Configure script with specific flags to tell it exactly where to install everything.

$ Configure -des \
    -Dprefix=/usr/local \
    -Dsitelib='/usr/local/lib/site_perl' \
    -Dsitelib_stem='/usr/local/lib/site_perl' \
    -Dsitelibexp='/usr/local/lib/perl5/site_perl' \
    -Dsitearch='/usr/local/lib/site_perl/'
    -Dsitearchexp='/usr/local/lib/site_perl/' \
    -Dinstallsitearch='/usr/local/lib/site_perl/' \
    -Darchlib='/usr/local/lib/perl5/' \
    -Dprivlib='/usr/local/lib/perl5/'

What do these flags mean?

• -des: A common set of defaults. d uses default answers for configuration questions, e ensures the build continues on its own, and s silences most of the verbose output.

• -Dprefix=/usr/local: This is the master key. It sets the main installation directory to /usr/local. The Perl binary will be installed at /usr/local/bin/perl.

• -Dprivlib & -Darchlib: These flags define where Perl’s standard library files will go. We’re pointing them to a clean /usr/local/lib/perl5 directory.

• -Dsitelib & -Dsitearch: These define where add-on CPAN modules will be installed. We’re pointing them to a subdirectory, /usr/local/lib/perl5/site_perl, to keep them separate from the core library.

Step 4: Build, Test, and Install

Now we compile the source code, run the built-in tests to ensure everything works correctly, and finally, install it.

# Compile the source code. This can take a few minutes.
$ make

# Run the test suite. This is crucial to ensure the build is stable.
$ make test

# Install the compiled files into /usr/local/.
# We use `sudo` because we are writing to a system-protected directory.
$ sudo make install

After this, you should close and reopen your terminal. You can verify the new Perl is your default by running which perl, which should now output /usr/local/bin/perl.

Essential Tools: cpanm and ack

With Perl installed, the next step is to install cpanm, a fast and friendly client for installing modules from CPAN. We’ll also install ack, a fantastic code-searching tool written in Perl that acts as a better grep.

# Install cpanm (cpanminus)
curl -L https://cpanmin.us | perl - --sudo App::cpanminus

# Use our new cpanm to install ack
sudo cpanm App::Ack

My Commonly Used Perl Modules

Here are a few of the modules I install on every new machine.

# For classic web scripting and handling parameters
$ cpanm --sudo CGI

# A modern, simple API for file and directory manipulation
$ cpanm --sudo Path::Tiny

# A beautiful data dumper for debugging
$ cpanm --sudo Data::Printer

# A powerful, real-time web framework (and associated tool set)
$ cpanm --sudo Mojolicious

Improving the Perl Debugger with Readline

The default Perl debugger is powerful but can be clunky. Installing Term::ReadLine::Gnu gives it command history and better editing capabilities, similar to the standard Bash shell. It depends on the readline library, which we can install with Homebrew.

# Install the readline library
brew install readline

# Install the necessary Perl modules for the debugger
$ cpanm --sudo DB::Pluggable
$ cpanm --sudo Term::ReadLine::Gnu

Setting Up Playwright for Browser Automation

Playwright.pm is a Perl interface to the powerful Playwright browser automation framework from Microsoft. It’s excellent for testing web applications. Setting it up requires a few extra steps because it depends on Nodejs.

Here is my post on how to install Nodejs on MacOS

Step 1: Install Node.js Dependencies

Playwright.pm is a wrapper around the JavaScript Playwright library, so we need to install it first using npm (the Nodejs package manager).

# Install the playwright library and its dependencies globally using npm.
# The -E flag preserves the environment, which can be important for sudo.
$ sudo -E npm install -g uuid express playwright

# Download the browser binaries (Chrome, Firefox, WebKit) that Playwright uses.
$ npx playwright install

Step 2: Install the Perl Modules

Now we can install the Perl modules that interface with Playwright. I’ve had issues in the past with its dependencies, so I install Test2::Harness and Sereal::Encoder separately as a precaution.

$ cpanm --sudo Test2::Harness
$ cpanm --sudo Sereal::Encoder
$ cpanm --sudo Playwright

And that’s it! You now have a clean, robust, and predictable Perl environment on your Mac, ready for development.

This guide will walk you through manually installing Node.js on macOS (and the same principles apply to Linux).

Why Install Manually?

Before we dive in, why avoid the popular tools?

• nvm (Node Version Manager): Excellent for projects that require you to switch between different Node.js versions frequently. However, it adds a small amount of overhead to your shell’s startup time and can be overkill if you just need one version.

• Homebrew: A fantastic package manager for macOS, but it comes with its own dependencies and update cycles. Installing Node.js manually keeps it isolated from your other Homebrew-managed tools.

• Official Installer: This works well, but it’s a “black box.” It runs scripts and places files on your system without you seeing exactly what’s happening.

The manual method has the advantages of simplicity and control. You know exactly where the files are, and there are no other tools to manage.

Step 1: Download the Binaries

Download the pre-built binaries from the official Node.js website. “Pre-built” means the code is already compiled for your machine, so you don’t need to build it from source.

Go to the Node.js Downloads Page

Look for the “Standalone Binary (.gz)”. You’ll need to choose the right version for your Mac’s architecture:

• ARM64: For newer Macs with Apple Silicon (M1, M2, M3, etc.).
• x64: For older Macs with Intel processors.

This will download a file like node-v20.16.0-darwin-arm64.tar.gz into your Downloads folder.

Step 2: Install Node.js

Now, we’ll unpack the archive and copy the files to /usr/local, a standard directory for user-installed software on macOS and other Unix-like systems. Placing the files here ensures that the node and npm commands are available in your terminal’s PATH.

Open your terminal and run the following commands.

Remember to replace the version number and architecture with the file you downloaded!

# Navigate to your Downloads folder where the file was saved
$ cd ~/Downloads

# Unpack the archive (your filename will vary)
$ tar xvf node-v20.16.0-darwin-arm64.tar 
$ cd node-v20.16.0-darwin-arm64/

# Copy the directories into /usr/local
# The 'sudo' command is needed because /usr/local is a system-protected directory.
# The '-a' flag on 'cp' preserves the file structure and permissions.
$ sudo cp -a bin /usr/local
$ sudo cp -a include /usr/local
$ sudo cp -a lib /usr/local
$ sudo cp -a share /usr/local

Step 3: Verify the installation

Check that node and npm are working correctly.

$ node --version
v20.16.0 

$ npm --version
10.8.1

If you see the version numbers, congratulations! Node.js is installed.

Managing Global npm Packages

To install global NPM packages, I use the -E flag with sudo to preserve the user environment, which can help prevent certain permission errors.

# Example: installing node-sass globally
$ sudo -E npm install -g --unsafe-perm node-sass

Upgrading

To upgrade node, first remove the installed version, and then repeat the steps above:

$ sudo -E rm -rf /usr/local/bin/node /usr/local/bin/npm /usr/local/bin/npx /usr/local/include/node /usr/local/lib/node_modules /usr/local/share/doc/node /usr/local/share/man/man1/node.1

GNU Typist is run from a terminal window, and is fast enough to run under an ssh session. I’ve run it remotely on RHEL and locally on Linux Mint and WSL. I did have some issues when I tried to build it from my Macbook. I forked the project and made some minor changes to the build process. Here is how I got it to build on MacOS:

Main Prerequisites: xcode, homebrew, and bash

The first step to compiling gtypist on MacOS is to make sure you have all the tools you’ll need: the Xcode command line tools, homebrew, and a modern version of bash.

You can find my guide for installing these tools here.

Install Dependencies

I use homebrew to install all the dependencies needed to build gtypist.

$ brew install ncurses
$ brew install automake
$ brew install gawk
$ brew install texinfo
$ brew install gengetopt
$ brew install xz
$ brew install help2man

A couple of notes:

1. Make sure you have the most up-to-date versions of all the tools. In particular, I found that using an outdated version of automake (that does not match the C compiler version) will cause the build to fail.

2. As an alternative, I installed the help2man directly from the GNU source. That utility is actually a Perl script, and I already have a modern version of perl installed on my system. If that doesn’t apply to you, the Homebrew installation will work fine.

Download and compile gtypist

I have forked the main GNU repository version and made a few minor edits to get the macos build working.

$ git clone -b macbuild https://github.com/kcaran/gtypist.git
$ cd gtypist
$ ./autogen.sh CFLAGS="-I$(brew --prefix)/opt/ncurses/include/" LDFLAGS="-L$(brew --prefix)/opt/ncurses/lib/"

Test and Install

You can test-run gtypist directly from the build directory before installing it for global use.

$ src/gtypist lessons/gtypist.typ
$ sudo make install

And that’s it! You now have a working version of GNU Typist installed on your Mac. To start a lesson, just run gtypist from your terminal. Happy typing!

This guide outlines how to install three prerequisites to working from the command line on MacOS: Xcode command line tools, Homebrew, and bash.

Step 1: Install Xcode Command Line tools

The absolute first step on any new Mac is to install the Xcode Command Line Tools. This package provides essential developers tools like Git, and compilers like clang and make, which are necessary for building software from source.

Running a git command will automatically prompt you to install them:

% git --version

Pro-Tip: Keeping XCode Tools Up-to-Date

Over time, these tools can become outdated, especially after a major macOS update. If you run into strange compilation errors, it’s a good idea to reinstall them to get the latest version.

# First, remove the old tools
$ sudo rm -rf /Library/Developer/CommandLineTools
# Then, trigger the installation prompt again
$ xcode-select --install

Step 2: Install Homebrew (Selectively)

I have a complicated relationship with Homebrew. On one hand, it’s fantastic for quickly installing small utilities and dependencies. On the other, I find it unnecessarily complex for core programming languages that I might want to configure or modify myself.

My rule of thumb is: if I’m ever going to want to compile a tool with non-standard flags or inspect its source, I’ll install it manually. For everything else, Homebrew is fine.

To install Homebrew, run the official command from their website.

# This command will download and run the installation script.
% /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

One fun aspect (maybe the funnest) of Homebrew is the default location of the files seems to change regularly. According to https://docs.brew.sh/Installation, the default install location is /usr/local for Intel binaries and /opt/homebrew for the newer Apple Silicon (M-series).

Step 3: Upgrade to a Modern Version of Bash

I’m (probably) too old to learn a new shell, and I don’t want to use different shells when I’m on linux, so bash it is.

Unfortunately, the version of bash that ships with macOS is ancient (version 3.2, from 2007). Apple has avoided updating it due to the more restrictive GPLv3 license of newer versions. This old version lacks critical features like modern tab-completion.

Thankfully, we can install a current version with Homebrew and set it as our default shell. The instructions on this blog post are excellent.

https://www.unindented.org/blog/change-shell-to-latest-bash-on-macos/

# 1. Install the latest bash from Homebrew
$ brew install bash

# 2. Add the new bash to the list of approved shells
#    `brew --prefix` correctly finds the Homebrew path (/opt/homebrew or /usr/local)
$ sudo sh -c 'echo $(brew --prefix)/bin/bash >> /etc/shells'

# 3. Change your user's default shell to the new bash
$ chsh -s $(brew --prefix)/bin/bash

# Add this to your .bashrc
export BASH_SILENCE_DEPRECATION_WARNING=1

Enable bash completion

Maybe this isn’t necessary?? Brew actually has multiple formulas, but bash-completion@2 is compatible with the later versions of bash.

$ brew install bash-completion@2

Then, add the following line to your ~/.bash_profile to activate it for every new terminal session:

# Add to ~/.bash_profile
[ -f $(brew --prefix)/etc/bash_completion ] && . $(brew --prefix)/etc/bash_completion

Install Better Command-Line Utilities

The default BSD-based versions of some tools on macOS are less feature-rich than their GNU counterparts. I use Homebrew to replace a few of them.

less and lesskey

The BSD version of less doesn’t support lesskey, but the Homebrew version is fine. You don’t even have to run lesskey to convert the file to binary anymore!

$ brew install less

Conclusion

And that’s it! My Mac now has the basic tools that allows me to set up a simple, stable, and predictable development environment.

Takeaway

After trying (and failing) to bolt on the IDE components to my current Vim configuration, I came away with the feeling that I can still use Vim as a text editor, but to use Neovim as an IDE I should start from scratch with one of the pre-configured flavors.

Note that I was successful in porting my current Vim configuration, so it would be possible to use Neovim as my ‘Vim’. But for now I think it is safest (and sanest) to maybe use both in tandem for different types of editing.

My Process for Installing and Converting to Neovim

Even though I may scrap my configuration, this is how I tried to install and configure Neovim on my Macbook and on our RHEL 8 web servers.

Macbook Installation

Installation

Any time I can avoid Homebrew I view it as a win. Fortunately, Neovim comes with pre-built releases for macOS:

$ curl -LO https://github.com/neovim/neovim/releases/download/nightly/nvim-macos-arm64.tar.gz
$ tar xzf nvim-macos-arm64.tar.gz
$ cd nvim-macos-arm64
$ sudo cp -a * /usr/local/

Running it the first time, Apple complains about all the shared libraries being downloaded from the internet.

Lua

This is the first big gotcha I had and I’m still kind of unclear. Neovim has Lua 5.1 embedded inside, so you don’t really need to install it, UNLESS… ?? Unless what? The lazy.nvim package manager suggests that you might need luarocks, which means you would need lua 5.1 installed globally. Oh, and maybe LuaJIT 2.1 (which will automatically build lua?)?

I downloaded lua 5.1 and built in from source and was proud of myself, but now I think I should have tried to use LuaJIT.

lazy.nvim Package Manager

My plugin manager of choice on the Vim side is pathogen with a bash script that downloads the plugins from GitHub (or wherever). So using an actual package manager was a leap, especially trying to configure it in Lua.

I went back and forth on whether to use a package manager and struggled with the first plugin I tried to import (vim-hexokinase). Fortunately, this was actually the most (only) complicated plugin. Once I had it working, it was smooth sailing.

The installation documentation was both straightforward and slightly unclear. I copy-pasted the lazy.lua file, which tripped me up later because it redefined the leader character, but otherwise worked ok. For defining (structuring?) my plugins I used the ~/.config/nvim/lua/plugins.lua file.

The lazy.nvim manager gives you a number of different hooks into the plugins: config, opts, and init. For vim-hexokinase, my woes were I was defining configuration in config, and then opts, but these were both too late in the start-up process.

 {
    "vim-hexokinase",
    init = function()
       vim.g.Hexokinase_highlighters = { 'backgroundfull' }
       vim.g.Hexokinase_palettes = { vim.fn.expand('$HOME') .. '/environ/safety_colors.json' }
    end,
 },

.vimrc to init.lua

Finally, I dumped my .vimrc file into an llm and asked it to convert the file to init.lua. It definitely saved me a lot of typing, but it really had trouble converting some strings to lua tables.

I also had to remove a number of stripped “legacy” features. This page was helpful:

https://neovim.io/doc/user/vim_diff.html#_removed-legacy-features

Attempt at adding IDE features and conclusion

Unfortunately, attempting to add all the whiz-bang IDE features that made me go on this journey in the first place was not very successful. This was mainly because I was trying to use them on RHEL 8, which is simply too old for it to be a seamless process.

While vim itself works almost everywhere and is extremely stable in that you don’t need to be on the latest version for it to work, Neovim is still in a state of flux. It has much higher aspirations to become a full-fledged IDE complete with AI code completion. This means that breaking changes are inevitable and keeping your software up to date is critical.

While I haven’t given up on using Neovim as an IDE, my next attempt will be on a single platform using a base configuration LazyVim seems to fit the bill. Then I’ll gradually add my configuration into it, instead of the other way around.

The first step I took was to try to a print-statement approach on my own fork of the project. This involved making changes on my local machine, pushing them to GitHub, then waiting to see the results of the CI Actions. It got me nowhere. What I really needed was a way to replicate how GitHub ran the actions on my local machine.

Tools required

I have access to a few different OS’s locally, but they all had their pros and cons. Ultimately, I used WSL (Ubuntu 22.04) and it worked great. In the future, I might try to replicate the environment on an M1 Mac.

The tool for running GitHub Actions locally is called act. It is written in Golang, so you’ll need to install that first. To install it on WSL I used:

$ curl https://raw.githubusercontent.com/nektos/act/master/install.sh | sudo bash

You’ll also need Docker. Of course nothing is easy and I went with the docker-ce version. To run Docker as your user (instead of root), add your user to the docker group and log back in.

$ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
$ sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
$ sudo apt-get install docker-ce
$ sudo usermod -aG docker $USER
<re-log-in>
$ sudo systemctl start docker
$ docker run hello-world

Setting up the back-end

It’s possible to run act with a default docker image, but I found it didn’t perfectly match how the GitHub Actions for Shaarli were run. I created my own Dockerfile with Shaarli’s dependencies. Mostly, I needed to install nodejs for the action to work properly. I also set the user to be ‘runner’, which is the GitHub Actions default user. Otherwise, the action would run under root, which is not what we want.

It made sense to put the Dockerfile under the .github/ directory for the project, so I created .github/act/Dockerfile:

FROM ubuntu-latest

# Install necessary dependencies
RUN apt-get update && apt-get install -y \
    sudo \
    git \
    curl \
    && rm -rf /var/lib/apt/lists/*

# Get correct version of nodejs
RUN curl -fsSL https://deb.nodesource.com/setup_18.x | sudo -E bash - \
    && sudo apt install -y nodejs

# Create a non-root user (same as GitHub Actions)
RUN useradd -m runner && \
    echo "runner ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers

# Set user to 'runner'
USER runner

# Set working directory (GitHub Actions uses this location)
WORKDIR /home/runner

CMD ["/bin/bash"]

There was definitely some experimentation required to get Shaarli’s actions running correctly, and I suspect every project has slightly different requirements.

Now, we can build and run the back-end Docker image, which in my case I named ‘shaarli’.

$ docker build -t shaarli .github/act
$ docker images

Running act

Finally, we can use act to run the GitHub Actions locally. We use our docker image (shaarli) instead of ubuntu-latest in the ci.yaml file. The --pull=false argument tells act the docker image is local. The -j option will run a specific job instead of the entire suite.

$ act -P ubuntu-latest=shaarli --pull=false -j php

If you want to ssh into the running image to further debug the action, use act --bind to keep it running. Find the container ID and run bash:

$ docker ps
$ docker exec -it 6ffdb /bin/bash

Other Docker utilities

This is definitely an iterative process. Here are some commands for cleaning up the Docker containers and images:

$ docker kill  $(docker ps -a -q)
$ docker rm  $(docker ps -a -q)
$ docker rmi $(docker images -a -q)

Debugging directly at GitHub

As a last resort (maybe?), you can even debug actions directly on GitHub using action-tmate. Add the following to your ci.yml file:

    steps:
    - name: Setup tmate session
      uses: mxschmitt/action-tmate@v3

The Actions details will show a host you can SSH into to get inside the container.

Good luck with your debugging!

It was great to finally meet so many wonderful people that I felt I already knew through using their software and watching conference talks from prior years. Here were the highlights of the conference for me.

Party Like It’s 19100+ℯ^iπ – Curtis Poe link to video

Ovid started the conference with a keynote that detailed the history of 25 years of Perl conferences. The highlights for me:

• His personal experience of 9/11 was deeply moving and a reminder that even as software developers, our main focus should be on the people we work with and the people who use our software.

• If we are to understand complicated systems, we need to understand the history of how they got to their current state. I run into this all the time at work: why does this venerable application work the way it does? Sometimes features are vestigial, obsolete remnants that should be removed, but others might have deep, important reasons for existing.

• On the controversies surrounding Perl 7 (my paraphrasing):

I don’t want to moralize but we didn’t need to have that happen. I know a number of folks who are no longer in the Perl community because of the abuse. Attacking our best and brightest is not the best strategy.

Why is it that hard to be nice? Why do we have to make fun of others—other languages, other people, other ideas? Why do we have to be abusive? It makes me very sad.

• The most important skill when working with AI is prompt engineering. It is an alchemy that is constantly changing, but it is the most important factor for getting accurate and useful information out of LLMs.

• It’s always a good reminder of how old some of the versions of Perl are that people expect to be supported as if they were new. The earliest version of modern Perl, 5.8, was released in 2004! Even 5.16 was released in 2012. What other languages are expected to carry such old versions while trying to improve the language at the same time?

Outreachy Updates – Bruce Gray for Makoto Nozaki link to video

Our company has a thriving internship program across multiple business units in an effort to recruit graduates into the insurance industry. I found a lot of parallels with the efforts of The Perl and Raku Foundation. Bruce brought up an excellent point: the hard part of internship programs isn’t finding suitable projects for the interns to work on, it’s finding mentors to guide the interns. He estimated the mentors needed five hours per week, and that tracks with my experience. Managing an intern is basically equivalent to managing a regular, full-time employee, and that effort needs to be recognized.

Structure Based Structuring of Unstructured Data – Adam Russell link to video

Adam Russell’s talk was a terrific primer on how LLMs are affecting the efforts to recognize, categorize, and ultimately use unstructured data. Given a document (or more likely, many many documents) of unknown or inconsistent formatting, how can we read in the data and organize it in a way to make it useful?

He explained the latest research into using LLMs to help automate the semantic knowledge graphs used to organize unstructured data. To me, this is exciting because working with unstructured data is hard! LLMs might be a way to unlock a lot of value from data that is currently opaque.

The Test2 Ecosystem – Chad Granum link to video

Chad actually gave three talks at the conference, and it’s always great to hear him speak about testing in Perl. The Test2 suite of modules is insanely comprehensive. There seems to be a plugin or some other way to test almost any software. He also previewed the new version of yath (Test2::Harness). Version 2.0 will significantly improve the performance and accessibility of test suites.

Some of the useful plugins and features Chad touched on in his talks include:

• Test2::Tools::Compare - Use the is() function instead of Test2::Tools::Basic. It is even being used to validate HTML form and JSON inputs!

• Test2::API - Gives you hooks to intercept events. It provides the framework for useful debugging tools such as Test2::Plugin::DieOnFail.

• Test2::Tools::Spec - Similar to Ruby’s rspec. It allows you to describe a block of tests and to define code to run before and after the suite or individual tests.

• Test2::Tools::QuickDB can spin up a basic database for your tests.

• Tests can be rerun from the log files generated by Test2 using yath. You can also use Test2::Plugin::SRand to generate “random” numbers repeatedly.

• Test2::Plugin::UTF8 is used to turn on UTF8 encoding for test output.

YAMLScript and Clojure – Ingy döt ؜Net­link to video

Ingy had three talks related to his latest project, YAMLScript, and I was blown away by its potential. It is compelling to think about both from the top down and the bottom up.

From the top down, YAMLScript can serve as a dynamic YAML parser. This will allow you to build custom YAML configuration files, depending on, say, the environment or particular server the configuration file is actually on. Ingy is striving to make the ys executable the gold standard for parsing YAML, making it unnecessary for other languages to roll their own parsers. It is also a complete and logical scripting language in its own right (with the added benefit that it is fast!).

From the bottom up, YAMLScript compiles down to Clojure, which itself runs on the JVM. This means that with YAMLScript you can reap all the benefits of the vast array of pre-existing Java libraries without actually having to write Java code! Its appeal over Clojure is that its syntax will be more familiar to developers experienced in the C-type languages and their syntax style.

I admit that up to now I’ve favored JSON over YAML when given the choice. But YAML is only growing in popularity and use for configuration of tools such as static site generators. YAMLScript is the catalyst I need to start using YAML more. Also, I would love to be able to generate PDFs using the OpenPDF library without having to resort to writing Java!

Playwright-Perl – George S. Baugh link to video

In the past I’ve used George’s Selenium::Remote::Driver to run tests against our insured account application and mobile app and he really does yeoman’s work to keep Perl support of Selenium up and running. This talk was the final straw for me and I am going to port all of them to Playwright. I believe the usefulness of his playwright-server utility could extend beyond Perl testing.

Actually Portable Perl – Gavin Hayes link to video

Gavin’s talk was about using Actually Portable Perl to compile a Perl program (including perl and any necessary modules) into a fully portable executable that can “run anywhere”. There was a palpable excitement about his work by the end of the talk. It’s a bit of a work-in-progress (it uses cosmopolitan libc as the C compiler, which is in active development), but I would love to be able to run my Perl on any Windows, Mac, or Linux machine.

Common-Sense Optimization – D Ruth Holloway link to video

Ask yourself what is the cost of not optimizing, and compare it to the estimated time and effort of trying to optimize. Also, what exactly should you be optimizing for? Is it the speed of the application? The amount of resources you need?

Remember the 80/20 principle and make sure you are focusing on the small problems that have a outsized impact.

Robert Heinlein said that proof only comes from experiment. In order to solve optimization problems, you’ll need to get your hands dirty! “You can’t talk it out”. This can apply to life in general, not just coding.

Quote from a young developer: “I don’t like old people’s code.” I thought that was hilarious, but I’m not sure who is more right. Older developers need to be cognizant of getting stuck in the way they write code that might be obsolete or just out of fashion (and therefore making it harder for others to work with it). But young developers can be just as stubborn and automatically assume that they know the One True Path for everything.

Demystifying Perl One-Liners – Walter C. Mankowski link to video

Walt’s talks about Perl one-liners are always excellent and it really should be part of any Perl programmer’s toolkit. I recently read Efficient Linux at the Command Line by Daniel J. Barrett, and more than once I thought, “why the heck would I learn sed or awk to accomplish this???”

One issue that came up near the end of the talk is how to use UTF-8 in one-liners. It turns out to be a little more involved than I expected, but I will post my findings soon!

PerlGPT, A Code Llama LLM Fine-Tuned For Perl – William N. Braswell, Jr. link to video

The State of the Onion AI – William Braswell

Will the Thrill gave two talks on Perl and AI and stressed the importance of getting the Perl community involved in AI projects so the language doesn’t get left behind. One interesting thing I learned is that Google pivoted its TensorFlow project from Python to C, which makes it much easier to create bindings to use its libraries in Perl.

PerlGPT is a specialized model based off of Code Llama trained on a curated set of modules from MetaCPAN. Code Llama Python uses other LLMs to derive the training set. This leads to “garbage-in garbage-out”-type quality issues.

Will maintains that source code is the most complex type of information that is still understandable. The human intention behind the code is necessary for true understanding, and is derived from the documentation and comments.

Code comments and names of functions and variables should capture your intentions

https://perlcommunity.org/ai/

Direct Access to PDF Internals with PDF::Data – Deven Corzine link to video

Every so often we run into issues at work with PDFs, and PDF::Data looks incredibly useful for tweaking PDFs programmatically using our language of choice. I can’t wait to try it out!

Refer to the PDF specification for a better understanding of how to manipulate the files!

The Once and Future Perl – Damian Conway link to video

Damian’s keynote did not disappoint and it’s hard to watch any presentation of his without coming away with a greater knowledge of Perl. For me and this presentation, it was that it is time for me to start learning how to use subroutine signatures.

He gave a couple of examples of how to use AI::Chat to embed normal language prompts inside your code, so that the LLM of your choice returns the output you want. This is much different than trying to use the LLM to write code (or help you to write code) that does what you want. It’s an interesting spin on how most developers think about using AI.

Paraphrasing from Brian Kernighan in The Elements of Programming Style: “Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible you are - by definition - not smart enough to debug it.” This is especially true for AI-generated code. If AI produces code more clever than you could come up with, you will have an especially difficult time tracking down the bugs that will invariably come with the code.

Wrap-up

As this post shows, I got so much out of this conference! I am so grateful to J.E. Turcotte for his tireless work in making sure that so many videos were posted. There were a few talks that I was unable to attend due to conflicts, and now I’ll be able view them! Peter and Amber Krawczyk did an amazing job finding a beautiful and appropriate venue, keeping us all fed, and giving us cool Vegas-themed swag!

I’d love to see a few more tracks (i.e. groups of talks on a similar theme). I think it might encourage others to give talks. For example, I wish there had been more talks specifically on web development.

One way that we might be able to encourage more attendees is to have a “beginners track.” Maybe there is a day or two of hands-on classes. Almost everyone I met at the conference was a seasoned pro, and the talks reflected that audience.

I look forward to seeing you all at future Perl events!

NodeJS Has Backwards-Compatibility Issues

The Node.js developers have discovered ways of making it much faster, but it would mean changing some of the internals of the project. Some programs developed in Node.js, most notably the Express web framework, link directly into those internals and would break if they changed.

The Perl community has the same kind of issues. It’s a recurring thing at conferences to hear the Perl Porters rue that they could clean or speed up the language itself but they don’t want to break working software that was written to rely on those specifics defects.

Backwards compatibility is a high goal to strive for, but it definitely comes with costs.

Link to Transcript

Programming Language Projects Are Best Written in Their Own Language

For large open-source projects, it’s best to have the majority of the code base written in the lingua franca of the project itself. Node.js is written mainly in JavaScript, not C++. Rust is written in Rust. The reason is this allows the project to expand the contributer base from the user base. The user base already knows the language of the project, but they might not be as familiar with a lower-level language.

Link to Transcript

C++ May Not Be Your Best Choice

Matteo Collina:

To be honest, even C is a significant improvement over C++, in my point of view… [laughter] So the bar is pretty low. To be honest, I prefer writing C. I love C code. C is beautiful. C++ gives me a headache half of the time because I don’t know what exactly I’m calling. So it’s really “Oh, is this a method? Is this a function? What is this happening here?” It has so much overloads… But if you look at it with the wrong eye, it just blows, it explodes on top of your face. So everything is better than C++ at this point.

JavaScript is a Trademark of the Oracle Corporation

TIL.

The trademark has no commercial value. Other than Oracle’s JavaScript Extension Toolkit, Oracle does not have any products using the trademark and presumably no planned usage. Oracle doesn’t even participate in the development of any of the JavaScript engines like V8, JavaScriptCore, or Spidermonkey. It seems very likely that JavaScript trademark infringement would be unenforceable in court due to non-use.

“The trademark is a dark cloud looming over the world’s most popular programming language,” he wrote. “Careful law abiding engineers bend over backwards to avoid its use – leading to confusing terms like ECMAScript.”

One area where everyone seems to have different requirements are reproducing blog post excerpts on the site’s home page. Here are my wants:

• By default, use the start of the content (up to a delimiter)
• Optionally, define the excerpt in the front matter configuration
• Markdown is parsed exactly the same as the original post
• Include images but allow custom styling on the home page
• Syntax highlighting

Here is how I implemented each of the requirements.

Use start of content as the default excerpt

Eleventy’s built-in excerpting handles grabbing the starting content and putting it into page.excerpt. I use <!-- more --> as the delimiter. It stands out more than --- and is easier for me to type than <!-- excerpt -->.

eleventyConfig.setFrontMatterParsingOptions({
  excerpt: true,
  // Optional, default is "---"
  excerpt_separator: '<!-- more -->'
});

Optionally define excerpt using front matter

If I want to use different content for the excerpt than the start of the post, I include it in the page’s front matter under excerpt using a YAML HereDoc.

---
excerpt |
  ### Heading
  This is an example excerpt with an ![image](/url/image.png)
---

In the post listing include template, if data.excerpt is present, use it. Otherwise use the page excerpt.

<li id="{{ post.data.title | slugify }}" class="post-list__item">
...
  <article class="post-list__excerpt">
    {% if post.data.excerpt % }
      {{ post.data.excerpt | md | safe }}
    {% else %}
      {{ post.page.excerpt | md | safe }}
    {% endif %}
  </article>
  <a class="post-list__read-more" href="{{ post.url }}">read article</a>
</li>

Markdown parsing

By default Eleventy doesn’t “do” anything with the excerpt content. I expect it to be markdown and parsed accordingly. In my eleventy.config.js file I placed the markdown-it configuration in a function that I can call in two places: one for the standard Eleventy markdown parsing, and the other as a filter that I can use for excerpts:

function configureMarkdownIt() {
  'use strict';

  // Reference: https://github.com/markdown-it/markdown-it-container/issues/23
  return require('markdown-it')({
      html: true,
      linkify: true,
      typographer: true
    })
    .use(require('markdown-it-bracketed-spans'))
    .use(require('markdown-it-container'), 'dynamic', {
      validate: function () { return true; },
      render: function (tokens, idx) {
        const token = tokens[idx];
        if (token.nesting === 1) {
          return '<div class="' + token.info.trim() + '">';
        } else {
          return '</div>';
        }
      }
    })
    .use(require('markdown-it-implicit-figures'), {
        figcaption: 'title',
        keepAlt: true,
        link: true
    })
    .use(require('markdown-it-highlightjs'), {
        code: true,
        inline: false
    })
    .use(require('markdown-it-smartarrows'))
    .use(require('markdown-it-attrs'));		// Should be last
}

  eleventyConfig.setLibrary( 'md', configureMarkdownIt() );

  // https://github.com/11ty/eleventy/issues/1380
  eleventyConfig.addFilter( 'md', function (content = '') {
    let html = configureMarkdownIt().render( content );
    return html;
  });

Excerpt images

On the home page, I might want to include an image from the post, but resize it to better fit in the list of posts. I may generalize the display of images in excerpts in the future, but for now I rely on the post title to style the images in the excerpt:

li#regex-is-a-programming-superpower article img {
    object-position: 0 0;
}
.post-list .hero img {
    max-height: 300px;
    object-fit: cover;
    width: 100%;
}

Syntax highlighting

This is where I fell down a rabbit hole. Everything worked as I wanted to this point except for the syntax highlighting. This is a known issue. I messed around with the default syntax highlighting plugin but wasn’t able to implement what I wanted.

In the end, I decided to forego the syntax highlighting plugin entirely and use the markdown-it-highlightjs plugin instead. It works great, or at least as great as hightlight.js does.

I think the only thing I really lose by going with markdown-it-highlightjs is the baked-in shortcode. But as long as I only want to render markdown, I don’t think this is an issue.

Assumptions and Restrictions

Date conversions can be confusing because there are a number of tools used in the process, each with its own quirks (or limitations). So there are the caveats to this process.

• The only date formats supported are UTC and the local timezone. The assumption here is if you live in New York, you won’t see dates from California or any other timezone.

• The dates being converted are in standard ISO format: ‘2015-03-08 06:59:00’.

Step 1: Format the Date String

Before passing the date string to Time::Piece, strip off everything after the integer seconds. If you need the milliseconds data, you can re-attach it after conversion. The same applies to any timezone indicators: add them back in after conversion.

Not following this step caused me a lot of confusion in the past. The linux strptime c-function doesn’t do a great job of supporting timezones. You are much better off stripping them off before trying to convert between UTC and local time.

Step 2: Create a new Time::Piece object using strptime

The strptime function converts a well-formed date string to a Time::Piece object. The trick here is that you use Time::Piece->strptime() if the date is already in UTC, and you use localtime->strptime() if the date is a local time.

Step 3: Convert the date using the epoch seconds

Use the epoch() function to create a new Time::Piece in the desired timezone. If you want your date in UTC, use Time::Piece->gmtime( $epoch ). Otherwise, use Time::Piece->localtime( $epoch ) for the local timezone.

Step 4: Use strftime to create the converted date string

I always use $tp->strftime( '%Y-%m-%d %H:%M:%S' ) to create a date string from a Time::Piece object, but $tp->datetime works as well (with a capital T separating the date and time).

Sample program

#!/usr/bin/env perl
#
# $Id: $
#
use feature ':5.16';

use strict;
use warnings;
use utf8;
use open ':std', OUT => ':encoding(UTF-8)';

use Time::Piece;

sub utc_to_local {
  my ($utc_ts) = @_;
  $utc_ts =~ s/^(\d{4}-\d{2}-\d{2}).(\d{2}:\d{2}:\d{2}).*$/$1 $2/
        or die "Illegal date format: $utc_ts";

  my $utc_tp = Time::Piece->strptime( $utc_ts, '%Y-%m-%d %H:%M:%S' );
  print "UTC epoch is   ", $utc_tp->epoch, "\n";
  my $local_tp = localtime( $utc_tp->epoch );
  return $local_tp->strftime( '%Y-%m-%d %H:%M:%S' );
}

sub local_to_utc {
  my ($local_ts) = @_;
  $local_ts =~ s/^(\d{4}-\d{2}-\d{2}).(\d{2}:\d{2}:\d{2}).*$/$1 $2/
        or die "Illegal date format: $local_ts";
  my $local_tp = localtime->strptime( $local_ts, '%Y-%m-%d %H:%M:%S' );
  print "Local epoch is ", $local_tp->epoch, "\n";
  my $utc_tp = gmtime( $local_tp->epoch );
  return $utc_tp->strftime( '%Y-%m-%d %H:%M:%S' );
 }

my ($utc, $local);

# DST not in effect
$utc = '2015-03-08T06:59:00.399Z';
$local = utc_to_local( $utc );
print "DST not in effect: utc($utc) = local($local)\n";

# DST is in effect
$utc = '2015-03-08 07:00:00Z';
$local = utc_to_local( $utc );
print "DST *is in effect: utc($utc) = local($local)\n";

# DST not in effect
$local = '2015-03-08T01:59:00-05:00';
$utc = local_to_utc( $local );
print "DST not in effect: utc($utc) = local($local)\n";

# DST is in effect
$local = '2015-03-08T02:00:00-04:00';
$utc = local_to_utc( $local );
print "DST *is in effect: utc($utc) = local($local)\n";

# DST is in effect
$local = '2015-03-08T03:00:00-04:00';
$utc = local_to_utc( $local );
print "DST *is in effect: utc($utc) = local($local)\n";

exit;

Here is the output of the script. Note that the dates are one minute (60 epoch seconds) apart, but Daylight Savings Time in the Eastern time zone changes the time from 1:59am to 3:00am. Both 2:00am and 3:00am for that day return the same epoch time.

$ perl timezone.pl
UTC epoch is   1425797940
DST not in effect: utc(2015-03-08T06:59:00.399Z) = local(2015-03-08 01:59:00)
UTC epoch is   1425798000
DST *is in effect: utc(2015-03-08 07:00:00Z) = local(2015-03-08 03:00:00)
Local epoch is 1425797940
DST not in effect: utc(2015-03-08 06:59:00) = local(2015-03-08T01:59:00-05:00)
Local epoch is 1425798000
DST *is in effect: utc(2015-03-08 07:00:00) = local(2015-03-08T02:00:00-04:00)
Local epoch is 1425798000
DST *is in effect: utc(2015-03-08 07:00:00) = local(2015-03-08T03:00:00-04:00)

There was only one thing that bothered me about the app. When images are exported, there is no padding or margins around the image, which can make them hard to work with. I wasn’t the only one with the issue and someone opened a github issue to discuss possible solutions. In the end, the issue was closed without a resolution, because there didn’t seem to be a generic solution that would please everyone.

The Workaround: Adding an Invisible Frame

My solution to this issue is similar to how I edit SVG icons in Illustrator. I add an invisible rectangle around the entire image with the amount of padding I want in the exported file.

The rectangle has a transparent fill and a transparent stroke to make it invisible. Even if you can’t see it, it isn’t difficult to select the rectangle by clicking around the border of the actual image. Once selected, the rectangle can be resized to increase or decrease the amount of padding around the image.

Now when the image is exported, the invisible frame adds space around the image, making it immediately usable on its own or embedded in documents.

• Example PNG image with padding.

This document is an opinionated process for creating suitable web SVGs. A “suitable web SVG” is one that can be easily and quickly tweaked by a developer by editing the code directly. It is also optimized to make it easy to embed in HTML or CSS files.

A Gentle Reminder: SVG is Code!

Unlike binary image file formats, it is possible to edit the code that generates the SVG image directly. At a minimum, you should feel comfortable with opening the SVG file in a text editor and inspecting the contents.

Editing in Illustrator

For the best results, the Illustrator output should be as clean as possible which will make any further optimizations easier.

Step 1 - Edit the Artboard

We would prefer the SVG’s viewBox to be large enough to support integer paths. Unfortunately, Illustrator ignores the viewBox and uses the width and height dimensions if you try to edit an existing SVG. So the first step is to contain the image in a suitable viewBox with padding around the actual image.

By convention, make the artboard 960px square. Create two squares: one the full height and width of the artboard (960x960) and another with 40px of padding all around the image (880x880).

Hide these outline squares before saving the image.

If the image is not square, use 960 as the smaller dimension and scale the larger dimension accordingly. Keep the same 40px padding around the entire image.

Step 2 - Export the Image as SVG

Choose File → Export → Export As

In the Export window, make sure Use Artboards is checked (and the correct one is selected).

For the SVG Options, Styling can either be Internal CSS or Presentation Attributes. If the image is a single color, Presentation Attributes is a little more straightforward to work with. Fonts should almost certainly be converted to outlines. Object IDs can be Minimal.

Unfortunately, in my experiments even with a large artboard Illustrator manages to corrupt the image if less than 2 decimal places are used in the export.

Select Minify but not Responsive.

Step 3 - Save in Illustrator format!

Finally, save (a copy) of the image in Illustrator format, either .ai or .eps is fine. Otherwise, you’ll have to go through all the above steps every time you need to edit the image!

Optimizing the SVG

Before deploying an svg on a production website, the svg should be optimized, i.e. made as small as possible.

WARNING: Optimization can lead to information loss and therefore should not be part of an automated build process.

Manual Editing Width and Height

Illustrator will output an image width and height that matches the viewBox dimensions. This is probably not what you want! The viewBox dimensions are large to (hopefully) avoid floating point numbers. The width and height attributes are the default pixel dimensions of the image. The web page will use those dimensions as a fallback if CSS is not loaded. For an icon, those numbers should probably be something like 24 or 48, not 960!

Edit the image file in the text editor of your choice and change the width and height values. This can be done either before or after any other image optimizations.

<svg id="a" xmlns="http://www.w3.org/2000/svg" width="24" height="24" viewBox="0 0 960 960">

Using svgo

The Node.js application svgo application can be used to help optimize SVGs (i.e. make them as small as possible).

Out-of-the-box, svgo removes the viewBox attribute, which destroys the ability to resize SVGs with CSS! With the following config file and command, viewBox is maintained and the numerical precision of the paths is set to 0 (integers).

svgo.config.js (Download)

$ npm install -g svgo
$ svgo --config svgo.config.js -o output.svg --input input.svg

When using the tool, take care to make sure the optimization hasn’t adversely affected the image. For example, the default precision of the tool is 3 decimal places (the recommended Illustrator output is two decimal places). Adding -p3 to the command will revert back to the default precision.

Linting with svglint

A linter is a tool that validates the correctness of a code file. The svglint application helps ensure that your SVG files adhere to standards and best practices.

$ npm install -g https://github.com/kcaran/svglint.git

The svglint application uses .svglintrc.js as its default configuration file. With this version, you can place the configuration file in your home directory and use it globally:

.svglintrc.js (Download)

$ svglint safety_icon.svg
------------------------------------ Files ------------------------------------
x safety_icon.svg
  x <?xml start tag unnecessary Starting xml tag unnecessary

----------------------------------- Summary -----------------------------------
x 1 invalid files.

Resources

https://www.aaron-gustafson.com/notebook/please-size-your-inline-svgs/

https://www.sarasoueidan.com/blog/svg-coordinate-systems/

https://stackoverflow.com/questions/18467982/are-svg-parameters-such-as-xmlns-and-version-needed

https://css-irl.info/optimising-svgs-for-the-web/

https://cloudconvert.com/svg-to-emf

A Brief Origin Story

Back in the 90’s when the World Wide Web was just getting started and you could teach yourself to publish for the web in a week, building HTML forms and processing them with CGI was my first real foray into text processing. I wrote some CGI-processing scripts in C, but it was hard! It was tedious!

About the same time a college intern I worked with introduced me to Perl. I couldn’t believe how easy it made everything. I could read in text files, process the lines, and output the results? Without allocating any memory or guessing how big the file was? I could work directly on entire strings, lines, or even files instead of processing them character by character? It was magic!

Regular expressions were (and still are) a fundamental part of Perl. The Learning Perl book, which is still an incredible resource, devotes three entire chapters to them:

Regular expressions are actually tiny programs in their own special language, built inside Perl.

Regular expressions are an integral part of Perl. Other languages seem to bolt on the functionality almost as an afterthought. My suspicion is that is why some programmers haven’t learned to use them.

My Own Regular Expression Usage

Using ack I took a quick look at my Advent of Code solutions over the years to see how many of my programs contained regular expressions:

kacmbp23:~$ ls -1 adventofcode20*/*.pl | wc -l
     266
kacmbp23:~$ ack -l \[\!\=\]\~ adventofcode*/*.pl | wc -l
     148

148/266 = 55.6% of my Advent of Code programs had at least one regular expression. I was actually a little surprised it was that low.

Running over a sample of perl scripts at work (which are mostly web applications), regular expression usage is even higher:

ramone8:/var/www/apps$ ls -1 */*.pl | wc -l
566
ramone8:/var/www/apps$ ack -l \[\!\=\]\~ */*.pl | wc -l
445

445/566 = 78.6% of our web application programs have at least one regular expression. That’s more like it!

Conclusion and Recommendation

If you aren’t using regular expressions, um, regularly, in your code, you may be doing yourself a disservice! Take it from Learning Perl, they aren’t all that terribly difficult to learn. You don’t have to use all the features immediately. Some of the more advanced ones, like /(negative\s)?look(ahead|behind)/ can come later.

Resources

Learn regex the easy way

https://github.com/ziishaned/learn-regex

A very nice, short summary of the basics of regular expressions

Learning Perl and Mastering Regular Expressions

Learning Perl book

Mastering Regular Expressions is supposed to be the absolute Tome of Enlightenment on the subject, but I wouldn’t know because I felt like I had enough background from Perl. It is probably a blind spot for me.

Mastering Lookahead and Lookbehind

https://www.regular-expressions.info/lookaround.html

I’ll admit that lookahead/lookbehind has always been troublesome for me and in the past I would go out of my way to avoid them. But this is my reference for when I do use them.

This one is also good:

https://www.rexegg.com/regex-lookarounds.html

Using regular expressions in JavaScript

https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Regular_Expressions

JavaScript seems to have a more dysfunctional relationship with regular expressions than any other language. It’s been largely corrected over the years, but I still wind up having to look up syntax and waste brain cycles over how to properly craft an expression in JS.

One of the ways I use it is to search our Apache web log files. The logs capture all requests, including ones for images, style sheets, fonts, and javascript files. But usually I’m only interested in who visited which web pages, including CGI scripts.

I’ve added a Bash function to my .bashrc file, acklog, that filters out the extraneous files automatically:

# ack (grep) for scripts in apache
function acklog() {
  ack "$@" | ack -v '\.(css|eot|gif|ico|jpe?g|js|png|svg|swf|webp|woff2?|xml)[? ]'
}

After over ten years of this process, I finally realized that we could automate this process, at least for the staging server. A cron script polling a system mailbox every two minutes kicks off the process.

CVS Command

Because the cron job runs under the Apache web server user, we need to set the correct CVSROOT before every CVS command.

my $CVS_CMD = "export CVS_RSH=ssh; cvs -d :ext:$ENV{ USER }\@cvs_server:/home/cvs";

It’s more challenging than you would think to get both the latest (head) version on the main branch as well as the currently installed version. To get the head use cvs log and to get the current version cvs status.

my ($head) = `cd $file->{ directory }
	&& $CVS_CMD log $file->{ filename } | grep head`
		=~ /head:\s+$CVS_VERSION/;

my ($current) = `cd $file->{ directory }
	&& $CVS_CMD status $file->{ filename } | grep Working`
		=~ /:\s+$CVS_VERSION/;

If the desired version was different than the current, the script would use a cvs update command to get the right version. If the desired version is the same as the head, the script adds the -A option to the command to clear any sticky tags.

As shown by the CVSROOT used, the CVS repository is on a different server, so I set up SSH keys for the Apache user on both machines.

The cron job worked well if just a little slow. I showed it to some coworkers, and one broke it right away! His installed had about 50 files, and the script would stop connecting with the CVS repository entirely after updating 10 or so files.

After playing with adding timeouts between commands, I realized that the company firewall was mistaking my script for a DOS attack! Every file needed multiple CVS commands to process, and each of these commands would open a new, individual SSH connection to the CVS repository. Eventually, the firewall would simply ignore requests SSH commands from the staging server for a few minutes, and the script would fail.

The fix was to keep the SSH connection open using the ControlMaster feature. That way, all of the CVS commands would use the same socket, and the firewall saw them as a single request.

-bash-4.1$ cat .ssh/config
Host cvs_server
        ControlPath ~/.ssh/%r@%h:%p
        ControlMaster auto
        ControlPersist 10m

I ran into a couple of peculiar bugs trying to calculate distances in SQL. Floating point math is never easy, and these are the issues I faced using our Microsoft SQL Server database.

Using Google APIs, I was able to get the latitude and longitude for each of our agency locations and added them to the database table as float. The formula for calculating the distance between two latitude/longitude pairs is the Spherical Law of Cosines.

After geolocating the search address (again using Google APIs), getting the closest locations and the distance in miles is relatively easy.

return $mssql_dbh->exec_select( <<DISTANCE_SQL );
select top 10
	rtrim(name01) as name01, rtrim(name02) as name02, rtrim(street) as street,
	rtrim(city) as city, state, zip, rtrim(phone) as phone,
	rtrim(email) as email, rtrim(web) as web,
	latitude, longitude,
	acos(sin(radians($lat)) * sin(radians(latitude)) + cos(radians($lat))
		* cos(radians(latitude)) * cos(radians(longitude) - radians($lon)))
		* 3959
		as distance
from agent_finder_info
	where (latitude > 40 and longitude < 0)
order by distance
DISTANCE_SQL

The where clause ignores any entries in the table that might not have a latitude and longitude association with it.

This appeared to work fine in production for years. One day an agent called and complained that the search for zip code 02185 (Braintree MA) wasn’t working. The agents furthest east of that location (which happened to be in Maine and on Cape Cod) were displayed - and reported as being hundreds of miles away!

The problem is for that area code, the longitude returned by Google is exactly 71 degrees North. Perl treats this as an integer and the distance formula looks like this:

acos(sin(radians(42.22)) * sin(radians(latitude)) + cos(radians(42.22))
	* cos(radians(latitude)) * cos(radians(longitude) - <b>radians(71)</b>))
	* 3959

In the SQL statement, radians(71.0) = -1.23918, but radians(71) = -1! Given an integer argument, radians returns an integer. Who the heck would want that??

My first fix was to explicitly cast the arguments to the radians() function as floats. Of course, I could have also done that in Perl.

return $mssql_dbh->exec_select( <<DISTANCE_SQL );
select top 10
	rtrim(name01) as name01, rtrim(name02) as name02, rtrim(street) as street,
	rtrim(city) as city, state, zip, rtrim(phone) as phone,
	rtrim(email) as email, rtrim(web) as web,
	latitude, longitude,
	acos(sin(radians(<b>cast($lat as float)</b>)) * sin(radians(latitude))
		+ cos(radians(<b>cast($lat as float)</b>)) * cos(radians(latitude))
		* cos(radians(longitude) - radians(<b>cast($lon as float)</b>)))
		* 3959
		as distance
from agent_finder_info
	where (latitude > 40 and longitude < 0)
order by distance
DISTANCE_SQL

This worked great.

ODBC connections require several layers of software to work. The bottom layer consists of the individual ODBC drivers for each database system. Our top layer is the DBI/DBD interface for Perl. In between these layers is the ODBC driver manager, which keeps track of the DSN’s and their corresponding ODBC drivers.

Packages and Software to Install

Ubuntu Version (tested on 9.10 Karmic Koala)

Debian Version (“unstable” distribution)

We use unixODBC as our ODBC driver manager. First, install unixODBC:

NOTE: Perl’s DBD::ODBC module requires the developer’s version of unixODBC, so install that one.

# apt-get install unixodbc-dev

Finding an ODBC driver for SQL Server was a challenge, since Microsoft refuses to directly support Linux. Fortunately, freeTDS (version 0.61 or later) seems to work fine.

# apt-get install tdsodbc freetds-bin

Debian NOTE:
Originally, I installed the libsybdb3 package to get freetds. But somewhere along the road, libsybdb3 was replaced with libsybd5. When I upgraded my Debian distribution, freetds was uninstalled in the process! Hopefully, the tdsobdc package won’t suffer from name changes.

The ODBC driver to the AS400 comes directly from IBM. The best part about the IBM website is how they constantly change their links and don’t forward the old ones. They’re worse than Microsoft in that regard:

http://www-03.ibm.com/systems/i/software/access/linux/downloads.html

# apt-get install rpm
# apt-get install alien
# alien -i iSeriesAccess-6.1.0-1.0.x86_64.rpm
# ln -s /opt/ibm/iSeriesAccess/lib64/libcwb* /usr/lib64

Red Hat Enterprise Linux Version

We are using unixODBC as our ODBC driver manager. A version of unixODBC comes with the Red Hat installation, but we need the developer’s version in order for both Perl’s DBD::ODBC module and FreeTDS to work:

yum install unixODBC-devel

Finding an ODBC driver for SQL Server was a challenge, since Microsoft refuses to directly support Linux. Fortunately, freeTDS version 0.61 or greater seems to work fine. Under Red Hat, we needed to compile the application.

1. Download the source code from http://www.freetds.org/

2. Uncompress the gzipped tar file:

   # gzip -cd freetds-0.82.tgz | tar xf -
   

3. Enter the freetds directory and configure, make, and install:

   # cd freetds-0.82
   # ln -s /usr/lib/libodbcinst.so.1.0.0 /usr/lib/libodbcinst.so
   # configure --with-tdsver=7.0 --with-unixodbc=/usr/
   # make
   # make install
   

The ODBC driver to the AS400 comes directly from IBM:

http://www-03.ibm.com/systems/i/software/access/linux/index.html

Of course, I had dependency issues. The libstdc++ dependencies are relevant, but the libXm and libXp libraries aren’t needed for ODBC connections.

# rpm -Uvh iSeriesAccess-5.4.0-1.6.i386.rpm
error: Failed dependencies:
        libstdc++.so.5 is needed by iSeriesAccess-5.4.0-1.6.i386
        libstdc++.so.5(CXXABI_1.2) is needed by iSeriesAccess-5.4.0-1.6.i386
        libstdc++.so.5(GLIBCPP_3.2) is needed by iSeriesAccess-5.4.0-1.6.i386
        libstdc++.so.5(GLIBCPP_3.2.2) is needed by iSeriesAccess-5.4.0-1.6.i386
        libXm.so.3 is needed by iSeriesAccess-5.4.0-1.6.i386
        libXp.so.6 is needed by iSeriesAccess-5.4.0-1.6.i386

# yum install /usr/lib/libstdc++.so.5

# rpm -Uvh --nodeps iSeriesAccess-5.4.0-1.6.i386.rpm

Configuration

1. Add SQL Server hosts to the FreeTDS configuration file:

   /etc/freetds/freetds.conf (Ubuntu/Debian)

   /usr/local/etc/freetds.conf (Red hat)

   [TDSproduction]
       host = 10.28.78.52
       port = 1433
       tds version = 7.0
   

2. Confirm the new drivers are in the driver config file: /etc/odbcinst.ini (Ubuntu/Debian and Red Hat)

   [FreeTDS]
   Description     = MS SQL driver
   Driver          = /usr/local/lib/libtdsodbc.so
   FileUsage       = 2
   
   [ODBC]
   Trace           = No       ;(=Yes if tracing using unixODBC)
   
   [iSeries Access ODBC Driver]
   Description     = iSeries Access for Linux ODBC Driver
   Driver          = /opt/ibm/iSeriesODBC/lib/libcwbodbc.so
   Setup           = /opt/ibm/iSeriesODBC/lib/libcwbodbc.so
   Threading       = 2
   FileUsage       = 1
   

3. Add servers to the server file: /etc/odbc.ini (Ubuntu/Debian and Red Hat) /usr/local/etc/odbc.ini (Red Hat)

   [Production]
   Driver                  = FreeTDS
   Description             = Production MS SQL Database
   Servername              = TDSproduction
   Database                = AVC
   UID                     = content1_badsg-1
   
   [AS400]
   Driver                  = iSeries Access ODBC Driver
   Description             = Production AS/400 Database
   Servername              = AS400.APPN.SNA.IBM.COM
   System                  = AS400.APPN.SNA.IBM.COM
   DefaultLibraries        = TESTMS
   UID                     = webodbc
   

Test the Connections

1. Verify unixODBC setup using odbcinst:

   # odbcinst -q -d
   	[FreeTDS]
   	[iSeries Access ODBC Driver]
   
   # odbcinst -q -s
   	[Production]
   	[AS400]
   

2. Test connection to FreeTDS servers using tsql.
   Note: make sure you use the server name, not the ODBC DSN, for the -S argument. See man tsql for more details.

   # tsql -STDSproduction -Usa
   

3. Test connections through unixODBC:

   # isql AS400 username PASSWORD
   

4. Debugging tools for connecting to AS400:

   # /opt/ibm/iSeriesODBC/bin/cwbping as400.appn.sna.ibm.com
   

Installing DBD::ODBC module for Perl

# cd
# perl -MCPAN -eshell
# get DBD::ODBC
<>quit
# tcsh
# cd .cpan/build/DBD-ODBC-1.09
# setenv DBI_DSN dbi:ODBC:LocalServer
# setenv DBI_USER sa
# setenv DBI_PASS sa
# setenv ODBCHOME /usr
# setenv LANG en_US
# perl Makefile.PL
# make
# make test
# make install

my $sql_query = qq|select PRSK01, PSTA15
                    from TESTHA.POLMAST
                    left join TESTHA.P15POLDP
                    on POLC01 = PPOL15 where POLC01 = 003950136|;

# Execute the sql query
my $sth = $dbh->prepare( $sql_query )
                    or die $sql_query, "<BR>\n", $DBI::errstr;

my $data = $dbh->selectall_arrayref( $sth, {Columns=>{}}, @sql_data )
                    or die $sql_query, "<BR>\n", $sth->errstr;

The error returned is:

DBD::ODBC::db selectall_arrayref failed: st_fetch/SQLFetch (long truncated
DBI attribute LongTruncOk not set and/or LongReadLen too small) (SQL-HY000)

The error message lead me to believe that we could fix it by setting $dbh->{ LongTruncOK } = 1 before making the query. But this actually gave us some serious side effects. Instead of a null or an empty string, DB returned two (or more) ‘\0’ characters. Here is an example from Data::Dumper. The ‘^@’ characters are zero byte (Hex 00) characters.

$VAR1 = [
           {
             'ADJNAM' =>  'A. Jonathan - Arb',
             'CNPD75' =>  '^@^@',
             'CADN75' =>  '^@^@^@^@^@^@^@^@^@^@^@^@^@^@',
             'CIDO75' =>  '^@^@',
             'CADC75' =>  '^@^@^@',
             'ADJCOD' =>  'U2'
           },

Fortunately, there is a better work-around. In the SQL, return an empty string if the column is null, otherwise return the column:

my $sql_query = qq|select PRSK01,
                    case when PSTA15 is null then '' else PSTA15 end as PSTA15
                    from TESTHA.POLMAST
                    left join TESTHA.P15POLDP
                    on POLC01 = PPOL15 where POLC01 = 003950136|;

Note that this occurs both on RHEL6 and Linux Mint 12, on both perl v5.10.1 and v 5.12.4, with DBI version 1.622 and DBD::ODBC version 1.39. It does not happen using the command line isql program to run the query.

1. If the table begins empty (with no rows in <tbody>), you can’t specify a default sort.

   If there are no rows in the <tbody> section when the page is first loaded, the sortList command will crash with a “parsers is undefined” error.

2. When the table is modified (rows are added or deleted), let tablesorter know the table has been changed with the following command:

   $( 'table.tablesorter' ).trigger( 'update' );
   

3. Sorting checkboxes - you need to add a customized parser for sorting checkboxes (and other form input fields):

   $.tablesorter.addParser({
       id: 'checkbox',
       is: function(s) {
           // return false so this parser is not auto-detected
           return false;
       },
       format: function(s, table, cell) {
           var checked = $( cell ).children( ':checkbox' ).attr( 'checked' );
           return checked ? 1 : 0;
       },
       type: 'numeric'
   });
   
   $( 'table.tablesorter' ).tablesorter({
       headers: {
           2: { sorter: 'checkbox' }
       }
   });
   

   The update trigger needs to be set when form fields are changed as well.

4. When using the zebra widget, you need to specify an ‘odd’ class for your table rows:

   body#employee-access tr {
   	background-color:#f1f0eb;
   }
   
   body#employee-access tr.odd {
   	background-color:#e3e2dd;
   }
   

   If you update the table (for example, removing rows):

   $( 'table.tablesorter' ).trigger( 'applyWidgets' );
   

5. Additional documentation:

   http://wowmotty.blogspot.com/2011/06/jquery-tablesorter-missing-docs.html

   http://mottie.github.com/tablesorter/docs/

• Time::Piece – CPAN module that provides object oriented time objects, It also provides date and time addition, subtraction and comparison and date parsing.

  use Time::Piece;
  
  my $t = localtime;
  print "Time is $t\n";
  print "Year is ", $t->year, "\n";
  

• Text::Autoformat – CPAN module that provides paragraph formatting and case transformations

• A list has a fixed set of elements, but an array is variable. You can use arrays for list functions, but you can’t use lists with array functions (push(), pop(), shift(), unshift()). <>

• @array[1] - The sigil is not the variable type. This is actually a slice with a single element. <>

• List::Util - first() - Similar to grep, but returns the first element where the result from the block is a true value.

  Also includes max(), min(), shuffle(), sum(), and reduce(). The perldoc also includes a number of example subroutines (all, any, none, notall).

• "\L$a" - Returns the contents of $a, but all lowercase.

• $| = 1 - perl filehandle select. Each filehandle has its own copy of this value, so you can’t set it globally.
  If you use IO::Handle, you can call the autoflush method to change the setting of the filehandle:

  use IO::Handle;
  open my( $io_fh ), ">", "output.txt";
  $io_fh->autoflush(1);
  

  <>

• print "@array" puts spaces between the elements, print @array does not.

• /o, which tells Perl to complie a regular expression only once, is obsolete as of 5.6.

• use re 'debug'; to debug regular expressions

• \G in regular expressions - Used with the /g flag, this anchors the last match. It uses the value of pos() as the position to start the next match. It is similar to the string anchor ^ and can be helpful in finding only consecutive matches. <>

• The special variables @- and @+ replace $```, $&, and $'` <>

• Smart match (~~) in perl 5.10 - Compare against an array of regular expressions

  my @patterns = ( qr/Fr.d/, qr/B.rn.y/, qr/W.lm./ );
  if( $string ~~ @patterns ) {
  	...
  };
  

• “A class is just a package, and its methods are just the package’s subroutines”

  See perlboot, perltoot, perlbot, and perlobj

• Although it has the same precedence as in C, Perl’s ?: operator produces an lvalue. This assigns $x to either $a or $b, depending on the trueness of $maybe:

  ($maybe ? $a : $b) = $x
  

• Don’t use the double-pipe or with unlink and other commands that are list operators, or put in extra parentheses: <>

  unlink $file || die; # This is wrong! You need to use 'or' here
  (unlink $file) || die; # This will work ok
  

• Use undef on the left side to skip return values in a list

  my ($name, $address, undef, undef, $zip) = get_address( $person );
  

• redo - restarts the loop block without evaluating the conditional again.

• Creating a module - perlmod, perlmodlib, perlmodstyle explain modules in all the gory details.

  perlnewmod gives a brief overview of the process along with a couple of suggestions about style.

  If you don’t need to use C code, other tools such as ExtUtils::ModuleMaker and Module::Starter can help you create a skeleton module distribution.

• Tom Christainsen’s article on why you almost always need a comparison function when calling sort. Perl’s cmp function (and C’s strcmp) is not an alphanumeric comparator – it is a code point comparator.

http://www.perl.com/pub/2011/08/whats-wrong-with-sort-and-how-to-fix-it.html

• Use Unicode::Collate to correctly sort unicode.

[caran@mater safetypublic]$ cvs -q update -AdP
: No such file or directoryectory /home/cvs/safetypublic/rsvp
cvs update: skipping directory rsvp

No such directoryectory? What the heck is that??

Most of my colleagues don’t work directly on the Linux-based web servers. They use TortoiseCVS on their workstations using a shared drive. For some reason, it occasionally (but not always) will convert the files in the CVS directory to DOS line endings.

The fix is simple. Use the dos2unix command to reset the CVS files to Unix line endings. It’s the repository file that causes the error.

[caran@mater safetypublic]$ cd rsvp/
[caran@mater rsvp]$ dos2unix CVS/*
dos2unix: converting file CVS/Entries to UNIX format ...
dos2unix: converting file CVS/Repository to UNIX format ...
[caran@mater rsvp]$ cvs -q update -AdP
M index.html
M rsvp-thankyou.html
[caran@mater rsvp]$ 

NOTE: Interestingly, a colleague of mine gets a slightly different error, but the cause and cure are both the same:

: No such file or directoryctory

The issue is in the way CGI.pm detects whether it is being called from a web browser or from the command line. It checks for the presence of the REQUEST_METHOD environmental variable. If it is present, CGI knows to get its arguments from the web browser.

When the inner script it called either using system() or the backticks method, it inherits the REQUEST_METHOD environmental variable from the calling program! It therefore thinks it’s being called directly from Apache instead of the command-line shell, and ignores the arguments. The trick to get it working is to temporarily clear the REQUEST_METHOD environmental variable:

my $cmd = "./gen_report.pl search_date=10/2009 pdf=1";

#
# Call gen_report.pl script to generate the report.  Since gen_report.pl is
# a CGI script that we are calling from the command line, we need to clear
# out the request method variable for CGI.pm
#
my $request_method = $ENV{ REQUEST_METHOD };
$ENV{ REQUEST_METHOD } = '';
my $display = `$cmd`;
$ENV{ REQUEST_METHOD } = $request_method;

I finally found the Holy Grail in a program I’ll call Java Terminator. The authors of this fine program just call it Terminator, but unfortunately a later Linux GNOME-based terminal emulator of the same name stole some of their thunder.

The best part of Java Terminator (and there are many excellent parts) is that, because it is written in Java, it runs on Cygwin, Linux, and MacOS. I’m able to use it at home on my Windows XP box and at work on Ubuntu, with no issues. Other terrific features are multiple tabs (liked tabbed web browsing, only with many terminals per window), unlimited scrollback, and automatic logging of everything.

As with any cross-platform program, there are a few configuration differences or issues to be aware of:

Redhat (RHEL5.2) and Centos

• Creating ~/.terminfo/t/terminator or /usr/share/terminfo/t/terminator doesn’t have any effect on the shell, vim, or less or man commands. After installing the terminfo, create a section for terminator in termcap.

  $ sudo sh -c "infocmp -C >> /etc/termcap"
  

  To do this locally (when you don’t have root access), you need to both create ~/.termcap and set the TERMCAP environmental variable.

  $ infocmp -C > ~/.termcap
  $ export TERMCAP=~/.termcap
  

• I was never able to get color working with Redhat’s version of vim. Instead, I downloaded vim and built it from source. The built version uses ncurses instead of termcap, and it colors fine.

Debian (4.0) and Ubuntu

• Create either ~/.terminfo/t/terminator or /usr/share/terminfo/t/terminator. These distributions don’t use termcap, so everything just works!

Cygwin

• I first tried creating /usr/share/terminfo/t/terminator (without a local ~/.terminfo). Vim worked fine, but the shell gave me a No entry for terminal type “terminator” error. The less and man commands both gave WARNING: terminal is not fully functional messages. Added a local ~/.terminfo didn’t help.

• Adding a record to termcap solved the shell issue, but not less or man.

  $ infocmp -C >> /etc/termcap
  

  The infocmp command is part of the ncurses package.

  It turns out, when I installed Cygwin, I set the default to Unix line feeds. But the infocmp command spits out DOS line feeds!
  So my termcap file had Unix line feeds except for the terminator section. Apparently, the shell could still read the file, but the less and man commands could not. The dos2unix command solved the problem!

  $ dos2unix /etc/termcap
  

• After that initial set-up, I’ve had no problems with emulator at all. The program has an active Google group and mailing list. The authors and enthusiasts there are more than helpful.

Everything worked fine until we put the code on a new server. Suddenly, the files downloaded by the web users were corrupted. We also began seeing the following warning in the log files:

Wide character in print at download.cgi line 124.

Here is the code…

use LWP::Simple;

# $cert_url is the URL for the internal server's pkcs12 certificate file
my $cert_p12 = get $cert_url;
if ($cert_p12) {
  # Get file name from the url
  $cert_url =~ /\/(\w+)\.p12$/;
  my $id = $1 || 'certificate';

  print "Content-type: application/x-pkcs12;\n";
  print "Content-Disposition: attachment; filename=\"$id.pfx\"\n\n";
  print $cert_p12;

It turns out that someone had installed the latest version of LWP directly from CPAN onto the new server. All of our other servers were using the version that came with RHEL5 from Red Hat. The Red Hat version is almost six years old!

The newer version of LWP::Simple sees the binary data inside of the pkcs12 file and tries (unsuccessfully) to discern its character encoding — except that it is binary data!

The solution is not to use LWP::Simple for retrieving binary data files. Use LWP::UserAgent instead.

use LWP::UserAgent;

# $cert_url is the URL for the internal server's pkcs12 certificate file
my $response = LWP::UserAgent->new->get( $cert_url );
my $cert_p12 = $response->content;

if ($cert_p12) {
  # Get file name from the url
  $cert_url =~ /\/(\w+)\.p12$/;
  my $id = $1 || 'certificate';

  print "Content-type: application/x-pkcs12;\n";
  print "Content-Disposition: attachment; filename=\"$id.pfx\"\n\n";
  print $cert_p12;

Print the string representation in hex

print join( ’ ', map { unpack “H*”, $_ } split( //, $x ) );

Text replacement on a per-file basis

perl -pi -e 's/\/business\//\/illus\/business.html/' */*.html

• -p : Iterate over each line of the file
• -i : Don’t create a backup file
• -e : Enter one or more lines of script

Muliple line pattern modifiers:

• /s allows wildcards (.) to match a newline. Use this to extend a search beyond a single line.

• /m changes the behavior of ^ and $ so they will match the start and end of any line. /^<h4>/m would match any line that began with an h4 heading tag.

• To explicitly match the start and end of the string, use \A and the EOF character, \z.

• The /s and /m modifiers are not mutually exclusive.

The break and continue keywords from C are last and next in Perl.

Use localtime to get the current year (assuming post-2000):

# Year is the sixth element of the localtime list
$YEAR = 2000 + (( localtime )[5] % 100);

Redirecting STDOUT temporarily to a scalar (string)

# Open a filehandle on a string
my $scalar_file = '';
open my $scalar_fh, '>', \$scalar_file
		or die "Can't open scalar filehandle: $!";

# Select scalar filehandle as default, save STDOUT
my $ostdout = select( $scalar_fh );

# Unbuffered output
$| = 1;

# Now, close scalar filehandle and bring back STDOUT
close( $scalar_fh );

print "ABC\n";
print "DEF\n";
print "GHI...\n";

# Bring STDOUT back
select( $ostdout );

Slurp an entire file into a scalar

open my $text_fh, '<', 'myfile.txt' or die $!;
my $contents = do { local $/;  <$text_fh> };

Run a function inside of a double-quoted string

my $input = qq|<input type="text" name="email"
	value="<b>${ \escapeHTML( $email ) }</b>">|;

Use Data::Dumper to display a data structure

use Data::Dumper;
warn "KAC:", Data::Dumper->Dump( [$data_structure], ['*main::data_structure'] );

Installation instructions can be found at the beginning of the file. Installing the syntax file is fairly straightforward, especially for anyone who has used Vim before. Feel free to experiment with the colors, and let me know if you come up with better.

View or download handhistory.vim hand history syntax coloring for the Vim editor.

The best (and possibly only) useful source of information about writing modules for Apache 2 is Nick Kew’s The Apache Modules Book.

The first thing you’ll want to do is buy this book. It is well written, provides lots of examples, and is very thorough. But even with The Apache Modules Book, you will need to do some exploring in the Apache code to understand how everything works and to apply it to your work.

Currently, RHEL5 ships with Apache 2.2.3 and APR 1.2.7. APR, the Apache Portable Runtime, provides all the tools and libraries you’ll need to build your own module. You’ll also need apxs, which is used to compile and install modules. The apxs binary is the httpd-devel package. You’ll want to install that before anything else:

# yum install httpd-devel

Also, get the source code directly from apache.org (http://archive.apache.org/dist/httpd/httpd-2.2.3.tar.gz).
You don’t need to rebuild and reinstall apache, but you definitely will want to peruse the included modules. I went so far as to add debug statements in some of the modules to learn more about them. There’s also a terrific template at modules/experimental/mod_example.c to help you get started.

Finally, download the source code for the latest versions of apr and apr-util (1.3.3 and 1.3.4 as of this writing) at http://apr.apache.org/download.cgi.

Like the httpd, you’ll want to be able to view and modify the source code. But more importantly, the ODBC DBD driver is built-in as of version 1.3.1 of apr-util. This is a big enough enhancement to upgrade the stock versions used by RHEL 5.2.

Install apr-1.3.3 using

# ./configure --prefix=/usr
# make
# make install

Install apr-util-1.3.4 using

# ./configure --with-ldap --with-apr=/usr/ --prefix=/usr
# make
# make install

You now have everything you need to start writing your own Apache module! Use the apxs utility to compile and install it:

Compile and install mod_safeca:

# apxs -c mod_safeca.c
# apxs -i mod_safeca.la 

Restart httpd

# service httpd restart

Creating the Package

A Perl module is a collection of code stored in a single file. By definition, all variables and function names in the module are stored in a package with the same name as the file.

For example, a module named Cgiutils would be stored in the file Cgiutils.pm. Any functions or variables in the module belongs to the Cgiutils package namespace. The beginning of the file should look like this:

#!/usr/bin/perl
#
# Cgiutils.pm - various debugging utilities for the CGI.pm module and other
#		CGI scripts.
#

package Cgiutils;

use strict;
use warnings;

Accessing Package Variables

To access a function or a variable declared in Cgiutils, you first need to tell the main program to import the module with the use command. Then, the module’s code can be accessed by prefixing the module name, Cgiutils:: to its functions and variables.

use Cgiutils;

my $ip = $Cgiutils::ip_address();

If it is too much of a hassle to type out the fully qualified names of the module’s functions and variables, you can export them from the module with the following code:

#!/usr/bin/perl
#
# Cgiutils.pm
#

package Cgiutils;

use strict;
use warnings;

use Exporter;
our (@ISA, @EXPORT);
@ISA = qw( Exporter );
@EXPORT = qw( &ip_address $cgi_debug %cgi_hash );

our $cgi_debug;
our %cgi_hash;

Note that any variables that are exported must be declared using our instead of my. This gives the main program access to those variables.

Using Global Variables in the Package

Global variables are the bane of neat and organized code, but they can be unavoidable in Perl. Here’s how to use them in modules.

In your main program, declare the variables to be global:

use strict;

# Define these variables as globals... **DO NOT USE 'my $xxx' with these!**
use vars qw($chosen_sort $sort_order $sort_type);

The module can now use these variables with the default main package.
Warning: make sure that you don’t try to declare them again with ‘my’ later on in the program!.

print "The chosen sort is ", $::chosen_sort, "\n";

Adding sort routines to modules

Finally, a word on moving sort routines into modules. If the sort routine inside the module will be used in the main program, make sure you use $::a and $::b for the sort variables.

# Sort subroutine located in a module and called from the main program
sub sort_by_field()
 {
  my $a_val = $::a->{ $::chosen_sort };
  my $b_val = $::b->{ $::chosen_sort };
  my $ret_val;

  # Determine how to compare based on the chosen field's type
  $ret_val = $a_val <=> $b_val;

  return ($ret_val);
 }

Sample Programs

Here’s a sample main program and module:

modmain.pl

#!/usr/bin/perl
#
# modmain.pl
#
# Testing main program for relationship between a
# main program and a module
#
use strict;
use warnings;

use modmodule;

use vars qw( $global_var );

$global_var = 5;

print "The module_var = $module_var\n\n";
module_func();

modmodule.pm

#!/usr/bin/perl
#
# modmodule.pm
#
# Testing module for relationship between a main program
# and a module
#

package modmodule;

use strict;
use warnings;

use Exporter;
our ( @ISA, @EXPORT );
@ISA = qw( Exporter );
@EXPORT = qw( &module_func $module_var );

our $module_var = 34;           # NOTE: Can't use 'my' here!

sub module_func {
	print "Calling module_func()...\n";
	print "The global_var = $::global_var\n";
}

From Moby Dipsum…

Call me Ishmael. Some years ago—never mind how long precisely—having little or no money in my purse, and nothing particular to interest me on shore, I thought I would sail about a little and see the watery part of the world. It is a way I have of driving off the spleen and regulating the circulation.

Whenever I find myself growing grim about the mouth; whenever it is a damp, drizzly November in my soul; whenever I find myself involuntarily pausing before coffin warehouses, and bringing up the rear of every funeral I meet; and especially whenever my hypos get such an upper hand of me, that it requires a strong moral principle to prevent me from deliberately stepping into the street, and methodically knocking people’s hats off—then, I account it high time to get to sea as soon as I can. This is my substitute for pistol and ball. With a philosophical flourish Cato throws himself upon his sword; I quietly take to the ship. There is nothing surprising in this. If they but knew it, almost all men in their degree, some time or other, cherish very nearly the same feelings towards the ocean with me.

This is an H3 Heading

So far as what there may be of a narrative in this book; and, indeed, as indirectly touching one or two very interesting and curious particulars in the habits of sperm whales, the foregoing chapter, in its earlier part, is as important a one as will be found in this volume; but the leading matter of it requires to be still further and more familiarly enlarged upon, in order to be adequately understood, and moreover to take away any incredulity which a profound ignorance of the entire subject may induce in some minds, as to the natural verity of the main points of this affair.

I care not to perform this part of my task methodically; but shall be content to produce the desired impression by separate citations of items, practically or reliably known to me as a whaleman; and from these citations, I take it—the conclusion aimed at will naturally follow of itself.

This is an H4 Heading

“Nothing, sir.”

“Nothing! and noon at hand! The doubloon goes a-begging! See the sun! Aye, aye, it must be so. I’ve oversailed him. How, got the start? Aye, he’s chasing me now; not I, him—that’s bad; I might have known it, too. Fool! the lines—the harpoons he’s towing. Aye, aye, I have run him by last night. About! about! Come down, all of ye, but the regular look outs! Man the braces!”

Steering as she had done, the wind had been somewhat on the Pequod’s quarter, so that now being pointed in the reverse direction, the braced ship sailed hard upon the breeze as she rechurned the cream in her own white wake.

“Against the wind he now steers for the open jaw,” murmured Starbuck to himself, as he coiled the new-hauled main-brace upon the rail. “God keep us, but already my bones feel damp within me, and from the inside wet my flesh. I misdoubt me that I disobey my God in obeying him!”