Linux Tools

Intro

This page is a collection of (hopefully) useful information and trivia which may be required to build a Web service based on Django/Apache/PostgreSQL and to manage a small pool of machines for testing purposes.

Python

At the time of writing the system version of Python is often 2.7, whereas newer applications benefit from using Python 3.*. One way to deal with that is to include "env" in hashbang pointing to the exact version you want to use. Apache/WSGI deployments may require additional footwork to ensure the correct version of Python runtime is used in mod_wsgi etc.

Debian "Alternatives" - Debian has a way to specify the default version of an app. For example, if more than one version of Python is present on the system, the command "update-alternatives" can be used to activate any of the available choices.

Caution - it's not a good idea to switch from the version of Python which came with your distro, since there documented and undocumented dependencies in various places, on that particular version. Random things may break such as software update, applications like Dropbox etc. Caveat Emptor.

Remove an alternative version:

sudo update-alternatives --remove python /usr/bin/python3

Example above allows to fall back on the previous version, such as Python 2.7.

It is recommended that instead of replacing the default, relevant scripts contain explicit reference to version 3+ if possible.

Django

There are a few ways to install Django, perhaps the cleanest and easiest is by using pip. With Python 3+ you will need to install pip3 first, like

apt-get install python3-pip

After that, Django is obtained by

pip3 install django==1.10

...and other versions available instead of 1.10 can be specified if needed. An important and popular Django add-on package "tables2" can be added likewise:

pip3 install django-tables2

To check which version of Django you are using at the moment, start interactive Python and use this:

import django
django.VERSION

Databases

Postgres

Installation

sudo apt-get update
sudo apt-get install postgresql postgresql-contrib

You will likely need an additional package if using PostgreSQL as the Django backend:

sudo pip3 install psycopg2

Running

If a restart of the DB engine is required:

sudo service postgresql restart

PSQL

Log in/out on localhost

The "psql" client requires a semicolon after each string you enter on the command line. It won't report if it's missing and it's easy to forget.

Switch over to the postgres account on your server by typing:

sudo -i -u postgres
psql

Same without switching accounts (just switching for one session):

sudo -u postgres psql

After having created a user and making sure authentication method is set correctly in the configuration file (path may be system and version dependent and named something like /etc/postgresql/9.5/main/pg_hba.conf), one can connect to PostgreSQL not as the default "postgres" user but for example as "p3s" or any other userID of choice:

psql -U p3s -d tst

The "-d" option is important because otherwise psql will assume a default database name which may not in fact exist. In the above example, the "tst" database was created beforehand by the user "postgres" to enable testing.

Example of getting help:

testdb=# \h create table

Exit out of the PostgreSQL prompt by typing: \q

Remote Access

Add or edit the following line in your postgresql.conf, in order to enable access from any host (edit accordingly for more selective access rights):

listen_addresses = '*'

To enable authentication from remote hosts for user "foo", edit pg_hba.conf to add

host    all             foo             0.0.0.0/0               md5

Users

Create a user/role:

createuser --interactive

Another example:

create user FOO with SUPERUSER

Databases and Tables

Creation of DB

From the OS prompt:

sudo -u postgres createdb foo

Also can be done from within psql.

postgres=# create database testdb;

Tables

testdb=# create table people (
testdb(# name char(50) primary key not null,
testdb(# age int not null
testdb(# );

Changing a table:

ALTER TABLE foo ADD last_maint date;

Info

List of DBs:

\l

List of schemas:

\d

Apache

Installation

On Ubuntu:

sudo apt-get install apache2

Start-Stop-Restart

Ubuntu

To start/stop/restart Apache 2 web server, enter one of the commands in each category:

### START
/etc/init.d/apache2 start
sudo /etc/init.d/apache2 start
sudo service apache2 start
### STOP
/etc/init.d/apache2 stop
sudo /etc/init.d/apache2 stop
sudo service apache2 stop
### RESTART
/etc/init.d/apache2 restart
sudo /etc/init.d/apache2 restart
sudo service apache2 restart

System status:

systemctl status apache2.service

CentOS/RH

On RedHat Linux, the name of the daemon is httpd. Also, "service" command may be aliased to systemctl.

systemctl status -l httpd.service

Apache Configuration

General Items

KeepAlive sets the tradeoff between memory and CPU usage by Apache.

Serving static files: https://docs.djangoproject.com/en/1.10/howto/deployment/wsgi/modwsgi/#serving-files

Official Layout of the Config Files

https://wiki.apache.org/httpd/DistrosDefaultLayout This, however, is not written in stone.

Ubuntu

/etc/apache2/apache2.conf

Deploying Django

mod_wsgi

• When using mod_wsgi one has to make sure the version matches the Python version, this needs to be specified when mod_wsgi is installed
• https://www.sitepoint.com/deploying-a-django-app-with-mod_wsgi-on-ubuntu-14-04/
• Methods of setting up the environment for wsgi described in the current Django documentation may or may not work on a particular installation of Apache due to a few bugs and general complexity of *.conf and related files

Ubuntu Example

Snippet from 000-default.conf on Ubuntu:

        ServerName promptproc
        ServerAlias promptproc


        WSGIScriptAlias / /home/maxim/projects/p3s/promptproc/promptproc/wsgi.py

        Alias /static/ /var/www/static/
        <Directory /var/www/static>
        Require all granted
        </Directory>

        <Directory /home/maxim/projects/p3s/promptproc/promptproc>
        <Files wsgi.py>
        Require all granted
        </Files>
        </Directory>

The "static directory must contain static content such as themes for the tables2 package. Keep in mind that while this is served automatically by the Django development server, it's not the case under Apache.

The file wsgi.conf needs to contain a reference to Python runtime like:

WSGIPythonPath /home/maxim/.local/lib/python3.5/site-packages

Database Deployment

Permissions

Assuming you are using sqlite, the file permissions on the DB file do matter if when you deploy under Apache. So you either need to set wide permissions (may not be a good idea depending on the security situation) or change the owner to "www-data" (on Ubuntu) or "apache" (on CentOS). Other OS may require similar tweaks.

PostgreSQL

An example of the "settings.py" clause:

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql',
        'NAME': 'foo',
        'USER': 'bar',
        'PASSWORD': '***',
        'HOST': '',
        'PORT': '',
    }
}

Misc Tools

ssh, telnet and other access methods

It is convenient to control a few machines from a single host. Typically ssh is used for this purpose, but if security is not a concern (e.g. then the network is strictly local) telnet can be also used as a quick solution. It will also server to "bootstrap" ssh connectivity i.e. debug ssh configuration remotely to make it operational.

Among advantages of ssh is X11 forwarding, which functionality telnet does not have.

ssh

You'll need to run the sshd service on every machine you want to connect to. On Linux, this is most frequently openssh-server and it can be trivially installed. Make sure there is a ssh entry in /etc/services, with the desired port number.

To be used productively, private and public keys will need to be generated or imported as necessary. For the private/public key pair to work, public keys should be added to the file ".ssh/authorized_keys". A matching private key must be loaded to an identity managing service (e.g. ssh-agent in case of Linux) on the machine from which you are going to connect. If it's not cached, you will likely be prompted to enter the passphrase for the key.

Typically (this depends on the flavor of your sshd) you will get a message specifying which public key is used during the login that you are attempting. This is useful to know if you have many keys and forget which was used for what connection.

Restarting the service:

sudo systemctl restart ssh

Adding a key to the agent:

eval "$(ssh-agent -s)"
ssh-add key_file

Gateways such as one operating at BNL and other Labs typically require that your public key would be uploaded and cached on their side in advance. The exact way this can be done is site-dependent. Some sites require to verify the upload by providing the public key's fingerprint. Example of how to get it:

ssh-keygen -E md5 -lf my_public_key_file

If you lost your public key (while still having your private one) you can re-create it:

ssh-keygen -yf my_private_key_file

Once it's done, a connection becomes possible, for example:

ssh username@atlasgw.usatlas.bnl.gov

The '-X' option is needed to enable X11 forwarding in a connection established in this manner.

Tunneling at BNL:

ssh -L 8080:130.199.23.54:3128 yourAccount@your.gateway.bnl.gov

The port 8080 is chosen as an example - by rules it must be a number larger than a certain low limit. On your local machine, you would need to specify a proxy which looks like this:

localhost:8080

Another example when going from one Linux box to another:

ssh -L 8000:localhost:8000 myRemoteHost

The above gives you access to the remote port 8000 on the local machine via localhost:8000.

telnet

While using ssh is in general preferable for many reasons and foremost due to security concerns, sometimes there is a chicken and an egg problem where you need to establish access fast in order to debug ssh on a remote machine. In these cases, and if security is not a concern (rare, but could happen on an entirely internal network), one may opt to use telnet.

On Ubuntu one can install the software necessary to run the telnet service in the following manner:

sudo apt-get install xinetd telnetd

Make sure there is an entry in /etc/services which looks like

telnet        23/tcp

Also, create a file /etc/xinetd.d/telnet with contents similar to this:

service telnet {    
        disable         = no
        flags           = REUSE
        socket_type     = stream
        wait            = no
        user            = root
        server          = /usr/sbin/in.telnetd
        log_on_failure  += USERID HOST
        log_on_success  += PID HOST EXIT
        log_type        = FILE /var/log/xinetd.log
}

...and start the service as follows:

sudo /etc/init.d/xinetd start

pdsh

This is an advanced parallel shell designed for cluster management. It often uses ssh as the underlying protocol although there are other options as well. Configuration is defined by files residing in /etc/pdsh. For example, the file "machines" needs to contain the list of computers to be targeted by pdsh. Optionally, this is also the place for a file that can be sourced for convenience of setup, cf

# setup pdsh for cluster users
export PDSH_RCMD_TYPE='ssh'
export WCOLL='/etc/pdsh/machines'

This of course can be done from the command line anyway, cf

export PDSH_RCMD_TYPE=ssh

Using ssh as the underlying protocol for pdsh implies that you have set up private and public keys just like you normally would for ordinary ssh login. Once this is done, you should be able to do something like this as a basic test of your setup:

pdsh -w targetHost "ls"

If the targetHost is omitted, the command will be run against all machines listed in the "machines" file as explained above. Should a command fail on a particular machine, this will be indicated (with an error code) in the output of the command, with the name of the machine listed. Redirection of stderr with something like "2>/dev/null" included with the command you run won't work with pdsh.

Misc

"nslookup" is a useful network information utility with diverse functionality. One simple function is to translate qualified host names to IP addresses and back.

Version Control

Notify git of your identity:

git config --global user.email "yourname@yoursite.yourdomain"

To avoid entering git userID and password:

git config --global credential.helper 'cache --timeout 7200'

LaTeX

One can choose to install all of tex packages or just a few:

apt install texlive texlive-humanities texlive-science

To see what is installed

dpkg -l

The little two-leter code at the front of each line says the status of the package. "ii" means installed and "rc" means removed but with config files still around ("dpkg --purge" or "apt-get remove --purge" gets rid of the "rc" but they are just harmless cruft).