JJ5-bin

From Sixsigma
Jump to: navigation, search

jj5-bin is where John keeps his Linux scripts. For other projects see projects.

Status

Released, but there's stuff TODO.

Administration

Contributors

People who have contributed to this project. Newest on top.

Copyright

Copyright 2012, Contributors.

License

Licensed under the New BSD license.

Resources

Source code

The repository can be browsed online:

The latest stable released version of the code is available via Subversion from:

Or if you want the latest version for development purposes:

Links

Specifications

Functional specification

Utilities

archive

Moves the files specified as arguments to an archive directory. Your archive directory is in ~/archive and it should only be accessible by you. Archive directories are named based on the date in yyyy-MM-dd-HHmmss format.

archive-copy

Copies the files specified as arguments to an archive directory. Your archive directory is in ~/archive and it should only be accessible by you. Archive directories are named based on the date in yyyy-MM-dd-HHmmss format.

backup-mysql-db

Backs up a MySQL database on the local host. Accepts two arguments. The first argument is the backup type, typically one of 'hourly', 'daily', 'weekly', 'monthly' or 'adhoc'. The second argument is the name of the database to backup.

Database backups are stored in the /var/backups/mysql/$database_name/$backup_type directory which is only accessible by root.

In order to run the database backup script needs access to the file /home/jj5/.passwd/mysql_root which must contain a single line containing the password for the MySQL root user.

backup-svn-repo

Backs up a Subversion repository. Accepts two arguments. The first argument is the name of the repository being backed up, e.g. 'bkrepo', 'pcrepo', 'jjrepo', etc. The second argument is the file path to the repository, e.g. /var/svn/export/svn/public/jjrepo.

Subversion backups are stored in the /var/backups/svn/$repo directory which is only accessible by root.

bkts

The 'bkts' command is short for 'Blackbrick Timestamp'. A Blackbrick Timestamp is in the form yyyy-MM-dd-hhmmss using 24 hour time and in the Australia/Sydney timezone.

To get a Blackbrick Timestamp run the following command:

$ bkts

To create a directory in the current directory named after the current date/time run the following command:

$ bkts md

To create a directory in the current directory named after the current date/time and to automatically pushd into that new directory run the following command:

$ bkts pd

Note that after running 'bkts' the $TS environment variable is updated with the latest/last timestamp.

See bkts technicalities for technical details concerning implementation.

bkxfer

The 'bkxfer' command is for Blackbrick staff who have access to the Blackbrick bkwork repository. This command will pushd into your user transfer directory. Typical use would be:

$ bkxfer
$ bkts md

That would put you into your transfer directory and then create a timestamped directory for you to put your files in.

See bkxfer technicalities for technical details concerning implementation.

ckdir

The 'ckdir' command is a contraction of 'cd' and 'mkdir'. The command issues a 'mkdir' command to create a new directory, and then a 'pushd' command to change directory into the newly created directory.

See ckdir technicalities for technical details concerning implementation.

cron-configure-backups

Installs the hourly, daily, weekly and monthly cron backup scripts for a host that has said scripts available in the lib directory.

fw-restart

Reloads the iptables firewall rules from the standard location /etc/iptables.up.rules.

fw-stop

Disables the iptables firewall allowing (and logging) all traffic. This is in case you need to do some debugging, but use with caution.

keep-only

Keeps only the specified number of files (typically backup files) in a directory. Accepts two arguments. The first argument is the path to the directory being processed. The second argument is the number of files that should be kept.

If the number of files in the specified directory exceeds the number of files to be kept, then the oldest files are deleted so only the specified number of files remain.

ls-all

Lists all files in a directory, or if no argument is given, all files on the system. This script is used by the backup scripts to get a full list of files and their user/group/permissions.

pd

The 'pd' command is an alias for the Bash built-in 'popd'. It pops the top directory off $DIRSTACK returning you to the directory that is the new top of the stack.

See pd technicalities for technical details concerning implementation.

reconnect

Reconnects your screen session.

take-ownership-jj5

Grants ownership of an entire directory (recursively) to jj5:jj5 and makes sure the files are owner accessible.

colorize-web-logs.sh

The colorize-web-logs.sh (download) script works in tandem with customised Apache2 web logs and lib.sh to colourise output. To work with this script you need to copy the config file x-log.conf (download) into /etc/apache2/conf-available/:

# cp x-log.conf /etc/apache2/conf-available/

Then enable the custom log config:

# a2enconf x-log

Then configure your virtual hosts to use the custom config (called "sm_log") by adding these lines or similar to your config files in /etc/apache2/sites-available/:

ErrorLog ${APACHE_LOG_DIR}/error.log
CustomLog ${APACHE_LOG_DIR}/access.log sm_log

Then restart Apache:

# apache2ctl graceful

Next you need to download your web logs in an appropriate fashion so that you can colourise them, here is a bare bones script to do that:

main() {

  # 2018-03-05 jj5 - prepare our working directory...
  #
  quiet rm -rf /tmp/web-spy;
  quiet mkdir /tmp/web-spy;
  quiet pushd /tmp/web-spy;

  # 2018-03-05 jj5 - disable the console screen saver (optional)...
  #
  setterm -blank 0 -powerdown 0;

  # 2018-03-05 jj5 - monitor web logs for hosts 'trust', 'orac' and 'tact'...
  #
  web_spy 201 jj-srv-1 J trust;
  web_spy 202 jj-wrk-1 T orac;
  web_spy 203 jj-wrk-2 T tact; 

  # 2018-03-05 jj5 - we're racing the creation of the log files so give them
  # a chance...
  #
  sleep 3;

  # 2018-03-05 jj5 - now that our weblogs are being downloaded into
  # /tmp/web-spy (the current directory) we can tail them locally and pipe
  # into our colouriserer...
  #
  tail -f *.log | /home/jj5/bin/colorize-web-logs.sh;

  return 0;

}

# 2017-07-06 jj5 - If you need a copy of this library:
# 2017-07-06 jj5 - SEE: jj5-bin standard library:
# https://svn.jj5.net/jjrepo/jj5-bin/branches/0.2/bin/inc/lib.sh
source /home/jj5/bin/inc/lib.sh;

Note: the above script relies on the web_spy() function from the lib.sh library in addition to its runtime helpers to support e.g. the main() function.

Now you can run the above script to colourise your web logs.

Have fun!

Session management

session-info

Prints info about your session that is being recorded.

session-login

The login script run when creating a new recorded session.

session-replay

Replays a recorded session. Accepts one argument, being the path to the session to replay without its file extension (either .timing or .script).

session-start

Starts a new recorded session.

session-stop

Stops recording the current session.

Configuration management

makeslave

The makeslave.sh script calls the makeslave.js script which builds a config file for the slave DNS server in /home/jj5/bin/bind/slave.conf.

Project management

bkrepo-release

Releases a Blackbrick Development project.

pcrepo-release

Releases a ProgClub project.

jjrepo-release

Releases a project in John's public repository.

jprepo-release

Releases a project in John's private repository.

svn-release

Releases a project in a Subversion repository. Accepts three arguments. The first argument is the name of the project being released. The second argument is the release notes. The third argument is the repository root URL.

This script is called from the bkrepo-release, pcrepo-release, jjrepo-release and jprepo-release scripts.

sixsigma

devotion

empathy

orac

Technical specification

The facilities available in jj5-bin are generally Bash shell scripts, functions or aliases.

bkts technicalities

The bkts command is implemented as a function in .bashrc. It temporarily sets the timezone to 'Australia/Sydney' and uses the 'date' command with a format string to build a timestamp.

bkxfer technicalities

The bkxfer command is implemented as a function in .bashrc. It just uses the Bash built-in command 'pushd' to change directory into a user's bkwork xfer directory.

ckdir technicalities

The ckdir command is implemented as a function in .bashrc. It issues a 'mkdir' command and if that is successful a 'pushd' into the newly created directory, returning a zero error level on success or non-zero error level on failure.

pd technicalities

The pd command is implemented as an alias in .bashrc. It just maps the new 'pd' command to the existing Bash built-in command 'popd'.

Notes

Notes for implementers

If you are interested in incorporating this software into your project, here's what you need to know:

Just copy-and-paste your way to victory! Feel free to ask questions.

Notes for developers

If you're interested in contributing to this software, here's what you need to know:

Send your patch to jj5@jj5.net along with a statement that you are willing to be listed in the contributors section of the documentation and willing to license your contribution under the license.

For security reasons write access to jjrepo is only available for John. It's basically my private repository. If you'd like to hack on projects where you do have checkin permissions on the Subversion repository, along with infrastructure for running your code, then why not head over to ProgClub and register?

Release notes

Release notes are (supposed to be) added after notable releases are made. New release notes are added to the bottom of the release notes section and have a level 3 title based on the release date.

Tasks

TODO

Things to do, in rough order of priority:

  • implement the stub scripts (e.g. session-info).

Done

Stuff that's done. Latest stuff goes on the bottom of the list.

2012-03-01

  • JE: created project page and initial documentation.