» Home » Docs » Release notes » Release notes for 1.4.0

Release notes for 1.4.0

This is a new minor release, adding new features, upgrade is recommended for all users.

As you may have noticed, the last release of Rex is around 6 months old. A lot has happened since then, and due to bigger internal changes, we wanted to be extra careful about the release, giving it as much time to bake as it needed.

The result is so far the biggest release of Rex, containing 4353 insertions and 1655 deletions happened to the lines of 118 files on 271 commits from 16 contributors, closing 93 PRs and 48 issues. To put it in perspective, this means ~30% of all files, and ~20% of all lines were touched in one way or another.

We couldn't have done this without the support of our fantastic community, keeping us busy with feedback, discussions both on- and offline, reporting issues, and sending pull requests. Thank you!

We are especially happy to have Eric Johnson (kablamo) to join the project as a new core developer. Most of the big things in this release were originally his ideas, and he was restless implementing them.

We would like to welcome H. Daniel Cesario, Harm Müller, Hiroki Matsuo, Jose Luis Perez Diez, Ken Crowell, Patrick Lauer, petersonchen, Prajithp as new contributors.

So as they say:

Good things come to those who wait

Highlights

Rex can now manage Linux Mint, Raspian and Arch Linux systems
tasks return exit codes
summary output at the end of run
unified firewall management via a new resource
ParamLookup has been merged into core from rex-recipes
values from CMDB and task parameters are now available directly as template variables
better late, than never: official debian 8 packages
experimental vim highlight support (see this gist)
rex can use development versioning on CPAN correctly now (e.g. for BETA or RC releases) to provide early access

New features

New Mkfs command

Now there's a dedicated Mkfs command, split out from LVM and partition handling modules, to deal with file system creation.

start of refactor mkfs/mkswap into their own pm - Peter H. Ezetta
starting to add option parsing - Peter H. Ezetta
setting up logic for lvm vs raw disk - Peter H. Ezetta
error checking on $lvname - Peter H. Ezetta
code cleanup and addition of POD - Peter H. Ezetta
refactoring swap logic into mkfs logic, silly to separate. - Peter H. Ezetta
refactor Partition.pm, fix $lv_name in Mkfs.pm - Peter H. Ezetta
Refactoring LVM.pm to use mkfs(); - Peter H. Ezetta
Ensuring that labels can be added to swap partitions - Peter H. Ezetta
Add Rex::Commands::Mkfs to rexify and pp.args - Ferenc Erki
Check for fstype existence at the start - Ferenc Erki
Skip creating filesystems for some reserved words - Ferenc Erki
Don't fail mkfs if label is undefined - Ferenc Erki
Pass options to mkfs (#fix 905) - Ferenc Erki

Improved argument parsing

As a preparation of another planned feature, it is now possible to pass arguments and CLI options to rex tasks, even if multiple ones are asked to be run. Please be aware this might not be a fully backwards compatible change for your use case. This feature is enabled by the 1.4 feature flag, but you can disable it with no_task_chaining_cmdline_args.

For example:

rex lib:task1 --option1=value1 lib:task2 --option2=value2
rex lib:task1 --option=value1 lib:task2 --option=value2
rex lib:task1 <arg> --option=value

Don't parse @ARGV on import. - Eric Johnson
added feature flag for 1.4 and removed deprecated warning of Rex::RunList - Jan
fixed a logic bug, that prevents batch creation - Jan
default to empty task list - Jan
Exit with an error message when a bad task name is used. - Eric Johnson
Add missing Data::Dumper import - Ken Crowell
Fix regression in deprecated static call to Rex::Task->run() - Eric Johnson
added Rex::Args->get method again. fixes #860 - Jan
don't clone task object. When cloned before_task_start/after_task_finished hooks doesn't work - fix #919 - Jan
check if task is no object, and get the task object then - Jan Gehring
fixed issue of calling Rex::Args->get inside before_task_start - RexOps/Rex#934 - Jan

New supported OSes

Managing Arch Linux is now supported via a completely new module, while Linux Mint and Raspbian are being recognized as Debian clones.

Recognize LinuxMint as a Debian clone (close #506) - Ferenc Erki
Recognize Raspbian as a Debian clone - Ferenc Erki
added archlinux support - Harm Müller
Remove purge package action on arch (fix #867) - Ferenc Erki

ParamLookup

Rex::Ext::ParamLookup from the RexOps/rex-recipes repository is now living in core as Rex::Helper::Rexfile::ParamLookup.

param_lookup is a command to manage task parameters.

my $var = param_lookup "param_name", "default_value";

First param_lookup checks the task parameters for param_name. If it's not found, it tries to look it up from CMDB (if CMDB is used). If it's still not found, it falls back to default_value. Additionally it registers the parameters as template values.

added ParamLookup and registering task parameter as template variables. - Jan
fixed a bug that was introduced by b0416b1710611b900284ba5a2d9d74991fc4351b - Jan
also lookup task name in cmdb - Jan
don't concat caller package anymore - Jan
fixed call of task as functions with normal parameters, #827 - Jan

Summary output

Now there's a short summary displayed at the end of each run, containing success/failure info of the executed tasks. If there were any errors, they are listed by remote machine (ordered by hostname).

Summarize success/failure at the end of a run - Eric Johnson
Summary tests for all permutations of Parallel::ForkManager, Base, and exec_autodie - Eric Johnson
fixed calling of exit hooks after summary - Jan Gehring
Fix summary tests for Mac OS X - Ferenc Erki
print the error message in the summary when a task die()d - Jan
Use error level output in the summary if some tasks failed - Ferenc Erki
Refactor summary output - Ferenc Erki
Further simplify summary output - Ferenc Erki
Sort failure list - Ferenc Erki
Fix summary test tasks for Windows - Ferenc Erki
don't print summary on rex -T - Jan
Exit codes now work when using 'rex -e' (fix #761, close #840) - Eric Johnson
Exit codes need to be bit shifted by 8. Do it in one place. - Eric Johnson
fix error code in $? after a run command which was introduced by 6c8441 - Jan
remove error message, we don't need it for the tests - Jan
try to fix failing tests on freebsd - #901 - Jan
Avoid bitshifting error code twice (fix #901) - Ferenc Erki
Don't confess upon failure of a non-resource (fix #936) - Ferenc Erki

Firewall handling

There's a new firewall resource to manage firewall rules.

firewall "open-port-82-for-1.2.3.4",
  ensure => "present",
  action => "accept",
  port   => 82,
  source => "1.2.3.4";

The UFW firewall provider has been moved from rex-recipes to core too, and it is now available on top of the default iptables provider.

On top of that, IPv6 support has been added.

open_port, [ 22, 80 ];     # defaults to IPv4 rule
open_port - 4, [ 22, 80 ]; # IPv4 rule
open_port - 6, [ 22, 80 ]; # IPv6 rule

Added a firewall module to manage different firewall models with one interface. - Jan
Add support for ip6tables (close #795) - Dmitry Kopytov
fixed firewall resource to work with ipv6 patch - Jan
detect iptables version and choose syntax for ip adresses - Jan

CMDB

Now CMDB can contain template code too.

YAML cmdb can now contain template code - Jan
make environment available inside cmdb template - Jan
allow config (set) vars in cmdb templates - Jan
Fix template handling in YAML CMDB files - Ferenc Erki

Enhancements

Crontab handling

Unprivileged users can now manage their own crontabs.

Add _whoami to determine remote effective username - Ferenc Erki
Fix managing a user's own crontab via cron_entry - Ferenc Erki
Suppress STDERR during crontab listings - Ferenc Erki

Templates

resource and task now have own namespace for template variables - Jan

Rexfile/module loading

The way of loading Rexfile and modules, has completely been refactored, making it easier to detect potential errors, and giving more useful feedback about them.

Simplify loading the Rexfile and improve error handling. - Eric Johnson
Put Rexfile exists check inside load_rexfile() - Eric Johnson
Refactor out logic for loading server ini file into load_server_ini_file(). - Eric Johnson
Refactor out some code into handle_lock_file() - Eric Johnson
Fix indentation - Eric Johnson
load a rexfile also if it doesn't return a true value. This fixes a regression - RexOps/Rex#513 - Jan
fixed loading of rexfiles with -f - Jan
fix loading of libraries in lib directory when use -f path/to/Rexfile - Jan
Avoid __Rexfile__ as dependency - Ferenc Erki
readd close of stderr - Jan
fix loading of Rexfile on perl 5.8.9 - RexOps/Rex#922 - Jan
debug output now gets displayed again. - fix RexOps/Rex#910 - Jan
Reopen STDERR after loading Rexfile - Dmitry Kopytov

Guest VM options

Now it is possible to set CPU and memory options for (test) VMs.

Allow tests so set Box memory and cpus (close #839) - Patrick Lauer
Fix passing memory/cpus options to constructor - Ferenc Erki
defaults for cpus and memory - Jan
add default values for cpu and memory - Jan

Documentation

Fix GetRex link url - hiroraba
Add docs on glob usage with file command (close #712, fix #738) - Jose Luis Perez Diez
Updated documentation in bin/rex (pod/man page) - Eric Johnson
Updated pod in bin/rex and lib/Rex.pm - Eric Johnson
Slightly reword synopsis - Ferenc Erki
Fix examples - Ferenc Erki
Update docs for partition command - Ferenc Erki
Update sudo docs (fix #918) - Ferenc Erki
Add docs for Rex::Shared::Var (fix #920) - Ferenc Erki
Update docs for Rex::Task - Ferenc Erki

General

A bunch of general enhancements happened:

host-specific sudo settings now take precedence
colored log output on Windows (if Win32::Console::ANSI is present)
supporting key_url and key_file for Red Hat repositories
shell detection has been improved
we can use development versioning now to make BETA and RC releases available early
do_task now accepts and handles parameters passed to it
less verbose connection output

Full list of commits:

Avoid unnecessary package rebuilds on Gentoo (close #809) - Patrick Lauer
Added option for rebase during pull. - Eivin Giske Skaaren
Mute noisy tests - Ferenc Erki
Use static comments to avoid alignment change later - Ferenc Erki
Fix static side comments - Ferenc Erki
give server configuration precedence - Jan
Custom storage path for kvm - Prajithp
use can_run to detect arp command - Jan Gehring
fixed lookup path for arp command - Jan
allow nested resources in reports - Jan
ignore eclipse files - Jan
Prefer uid over getlogin as fallback username (close #793, fix #792) - Ferenc Erki
allow mix of modules.rexify.org modules and git modules - Jan
Tests for needs() were not useful. Now they are. - Eric Johnson
color for windows - Jan
Output full task names (fix #825, close #842) - Daniel Cesario
support key_url and key_file also on redhat systems - Jan
fixed tests for windows - Jan
added tmp to gitignore - Jan
Refactor Rex::TaskList. No functional change. - Eric Johnson
fixed reporting of nested resources (changed resource) - Jan
only update chained resource when the upper resource wasn't changed before. Also removed 'inside resource check' because this is not needed anymore due to stacked resources - Jan
moved detect method to base, because it is everywhere the same, created direct_exec method for all interfaces - Jan
fixed inheritance problem, moved shell() method to base class. - Jan
remove unnecessary new lines - Jan
run shell detection a second time if first time failed - Jan
Use development versioning - Ferenc Erki
Enable more feature flags for development releases - Ferenc Erki
Add tests for task hooks - Ferenc Erki
added parameters to do_task and fixed LOCAL block - Jan Gehring
removed connecting to mesages (change to debug level) - Jan Gehring
can't connect message must be ::info logging - Jan
Let service tests be silent - Ferenc Erki

Bugfixes

Shared variables

Previously there was a race condition while using Rex::Shared::Var to share variables across tasks.

Test for race condition problem in Rex::Shared::Var. - Eric Johnson
Avoid race condition when doing IPC. - Eric Johnson
Refactor child/parent pid logic + comments to be more readable - Eric Johnson

Templates

use empty string to prevent uninitialized warning messages - Jan
Test rendering of false values in templates - Ferenc Erki
Fix rendering of false values in templates - Ferenc Erki

AWS handling

We now use AWS::Signature4 to talk to Amazon services.

added AWS::Signature4 directly to code, due to problems with Centos5 dependencies (#737) - Jan
Unbundle AWS::Signature4 - Ferenc Erki

Draining IO in parallel

The previous way to drain STDOUT/STEDERR output of a remote (or for that matter, local) connection proved to be, let's say suboptimal, leading to potential hangs, and other weirdness. This now has been fixed, moved to a common codebase, and we cover it with tests.

Read STDOUT and STDERR in parallel (fix #756) - Ferenc Erki
use sysread to read from filehandle. fixes #887 - Jan
Use sysread for local exec interface too (fix #887) - Ferenc Erki
fix continuous read bug - RexOps/#951 - Jan
Add initial tests for draining IO - Ferenc Erki
Fix draining IO - Ferenc Erki
Handle partial last chunks from STDOUT and STDERR - Ferenc Erki
removed read_buffer reduction - Jan Gehring
Update read_buffer test for Windows - Ferenc Erki
Test with a bit less output - Ferenc Erki
Increase timeout for read_buffer test (fix #954) - Ferenc Erki
cleanup newline(s) at the end. fix for #957 - Jan
remove chr(13) everywhere - fix #957 - Jan
fixed variable usage - Jan
Normalize EOL characters earlier (fix #957) - Ferenc Erki

General

The reporting feature had some troubles when using bulk_install, so until a proper fix, Rex now simply falls back to sequential package installation as a workaround.

On top of that, the most notable bug fixes are realted to task hooks and continuous reading of output.

Full list of commits:

Fatal exceptions if git switching fail - alex1line
fix for Net::OpenSSH can't set initialize_options for the constructor - #796 - Jan Gehring
fixes the problem of run_task connecting to the wrong host #788 - Jan Gehring
fixed loading of Rex::Commands via require. RexOps/Rex#789 - Jan Gehring
lower exitecode to less than 255 because some os have problems with larger ones - Jan
Fix undef warnings if calling get() without param - Andy Beverley
Map Debian clones correctly - Ferenc Erki
iptables_clear should clear only loaded tables - Dmitry Kopytov
Checking for a running rex process failed if the pidfile contains a pid that partially matches a running process. - petersonchen
Ensure proper test-time connection type (fix #821) - Ferenc Erki
use md5 binary if perl module is not available - fix for #834 - Jan
report all packages - Jan
fixed some unitialized warnings - Jan
fix after hook call for connections that wasn't successfull - Jan
call after task hook, even if connection to system failed - Jan
Fix links in README (fix #853) - Ferenc Erki
The after hook now runs even if the ssh connection fails. - Eric Johnson
load right cron module if os is *bsd and shell is csh - #883 - Jan
return always shell object - Jan
some windows fixes for rexify command - Jan
Fix POD - Ferenc Erki
fixed loading of resource module in perl 5.8 - RexOps/Rex#917 - Jan
Make CPAN Testers happy for unsupported OSes - Ferenc Erki
Fix copy-pasta POD errors - Ferenc Erki

(R)?ex the friendly automation framework

News

Events

Release notes for 1.4.0

Highlights

New features

New Mkfs command

Improved argument parsing

New supported OSes

ParamLookup

Summary output

Firewall handling

CMDB

Enhancements

Crontab handling

Templates

Rexfile/module loading

Guest VM options

Documentation

General

Bugfixes

Shared variables

Templates

AWS handling

Draining IO in parallel

General