rsync over SSH

sshMany of us use SSH multiple times on a daily basis times to do simple, complicated, and often redundant tasks. Often times the tasks are those which could be scripted and automated. For instance, if you have to synchronize files with a server often throughout the day, a cron job would be the ideal way because then it will be done automatically and you don’t have to worry about it. If you use SSH keys without a password to access a server, you can expand on it by using rsync to synchronize those local and remote directories.

Here is the command you would use to make this happen:

rsync -e 'ssh -i ~/.ssh/id_rsa' -rulvhtpz /Users/user/file_to_sync user@host.com:~/

rsync options used

-r, recursive throughout directories
-u, skip files that are newer at the destination (meaning only update old files)
-l, copy symlinks as symlinks
-v, verbose; show all output as it happens
-h, display output in human readable format
-t, preserve times of files
-p, preserve permissions
-z, compress files during transfer to preserve bandwith

Making rsync convenient:

rsync is really nice when it comes to automation. Adding rsync to a crontab entry comes really handy. There are all kinds of options for cron – to view them, check out my knowledge base article on it.

If we want rsync to run automatically at 12pm and 4pm, this is what we would do:

Open up your terminal app and type the following:

crontab -e

Add the following lines to the file:

00 12 * * * rsync -e 'ssh -i ~/.ssh/id_rsa' -rulvhtpz /Users/user/file_to_sync user@host.com:~/
00 16 * * * rsync -e 'ssh -i ~/.ssh/id_rsa' -rulvhtpz /Users/user/file_to_sync user@host.com:~/

If you’re really clever

If you are a programmer and want your code to automatically synchronize to a remote server, add a macro to your IDE that somehow that adds the rsync code to a button in. For instance, if you add the rsync command to the save button command, maybe it will kill two birds with one stone.

For more information

Go into your terminal and type man rsync or rsync -h

Installing Dell V515w Printer on Arch Linux

Dell printers can be a challenge to install on some Linux distributions…

However, once you figure out the key files, it makes it a little bit easier to make things happen. The steps here are basically hacks that I went through until I could figure out what was going on. Here are some KEY bits of information to get things going:

Summary of Requirements

Install dpkg (Debian Package Manager)

To install dpkg, you need to get it from the AUR (Arch User Repository). The easiest way is to follow these steps:

echo "[archlinuxfr]" >> /etc/pacman.conf
 echo "Server = http://repo.archlinux.fr/x86_64" >> /etc/pacman.conf
 pacman -Syu yaourt

At this point, yaourt should now be installed. Now to install dpkg (NEVER EVER RUN YAOURT AS ROOT!!!):

When you run the following command, select “dpkg” – at the time of this article, it was number 3 on the list:

yaourt dpkg

No need to modify the makepkg or anything – just let it do its thing.

Getting the .deb file from the Dell .sh installer

This part is a little tricky. Do these steps (as regular user – not root) and it “should” work:

  • sh dell-inkjet-09-driver-1.0-1.i386.deb.sh
  • Open up your user’s home directory (use the GUI for this)
  • Go into the lua* directory that was created by the installer
  • As you press next in the wizard, you will notice some files that pop up in that directory. When they do, quickly copy the .deb files to your desktop or somewhere that you will remember
  • I know it’s tricky, but keep trying. It worked for me

Run the .deb file

dpkg -i --force-architecture dell-inkjet-09-driver-1.0-1.i386.deb

Check if “dlnet” exists in /usr/lib/cups/backend

ls /usr/lib/cups/backend/dlnet

If the dlnet file does exist, then you can now run system-config-printer and add your printer (use Dell Network Backend as the socket when the list comes up:

dlnet://192.168.1.6 <-- Change that with the IP of your printer

When it asks for a driver file, select the Dell_V310V510_Series.ppd file (if you can’t find it, run this command):

find / -name Dell_V310V510_Series.ppd

If the dlnet file is not in /usr/lib/cups/backend…

Then you will have to extract the .deb files and

bsdtar -xf *.i386.deb; tar -xzvf data.tar.gz; cd usr/local/dell/dell09/bin; cp dlnet /usr/lib/cups/backend/

Now proceed to the step above this one and configure your printer.

If something didn’t work for you, or if I missed something…

I wrote this article after spending about 20 hours trying to get this printer to work. Because of this, there are probably some things I forgot – but I believe the most important stuff is written here. If you notice anything off or that needs to be added, please just reply using the comments area below.

Mod your Php installation to handle larger uploads

Though PHP presents a very versatile and user friendly interface for handling file uploads, the default installation is not geared for working with files in excess of 2 Mega Bytes. This article will help you configure your PHP engine for handling such large file transfers.

The php.ini File

All the configuration settings for your installation are contained in the php.ini file. Sometimes these setting might be overridden by directives in apache .htaccess files or even with in the scripts themselves. However you cannot over ride the settings that effect file uploads with .htaccess directives in this way. So let’s just concentrate on the ini file.

You can call the phpinfo() function to find the location of your php.ini file, it will also tell you the current values for the following settings that we need to modify

  • file_uploads
  • upload_max_filesize
  • max_input_time
  • memory_limit
  • max_execution_time
  • post_max_size

The first one is fairly obvious if you set this off, uploading is disabled for your installation. We will cover the rest of the configuration settings in detail below.

  • upload_max_filesize and post_max_size
  • memory_limit
  • max_execution_time and max_input_time

Files are usually POSTed to the webserver in a format known as ‘multipart/form-data’. The post_max_size sets the upper limit on the amount of data that a script can accept in this manner. Ideally this value should be larger than the value that you set for upload_max_filesize.

It’s important to realize that upload_max_filesize is the sum of the sizes of all the files that you are uploading. post_max_size is the upload_max_filesize plus the sum of the lengths of all the other fields in the form plus any mime headers that the encoder might include. Since these fields are typically small you can often approximate the upload max size to the post max size.

According to the PHP documentation you can set a MAX_UPLOAD_LIMIT in your HTML form to suggest a limit to the browser. Our understanding is that browsers totally ignore this directive and the only solution that can impose such a client side restriction is our own Rad Upload Applet

When the PHP engine is handling an incoming POST it needs to keep some of the incoming data in memory. This directive has any effect only if you have used the –enable-memory-limit option at configuration time. Setting too high a value can be very dangerous because if several uploads are being handled concurrently all available memory will be used up and other unrelated scripts that consume a lot of memory might effect the whole server as well.

These settings define the maximum life time of the script and the time that the script should spend in accepting input. If several mega bytes of data are being transfered max_input_time should be reasonably high. You can override the setting in the ini file for max_input_time by calling the set_time_limit() function in your scripts.

  • Additonal Comments
  • Apache Settings
  • Other Options

The apache webserver has a LimitRequestBody configuration directive that restricts the size of all POST data regardless of the web scripting language in use. Some RPM installations sets limit request body to 512Kb. You will need to change this to a larger value or remove the entry altogether.

If you expect to handle a large number of concurrent file transfers on your website consider using a perl or java server side component. PHP happens to be our favourite web programming language as well but perl and Java are just slightly ahead when it comes to file upload.

Most installations of perl as an apache module can accept in excess of 32 megabytes out of the box. Compare this against the 2MB default for PHP. The downside is that perl coding takes just a bit more effort than PHP but it’s worth it. You can try our sample scripts to get started.

Using .htaccess files

By Katherine Nolan
Expert Author
Article Date: 2003-04-16

If your site is hosted on a Unix or Linux server which runs Apache, you may already be familiar with your .htaccess file.

We referred to it in an earlier tutorial on Creating Custom Error Messages, where we showed how you can configure it to instruct the browser to display custom error messages rather than the dull and unhelpful generic ones.

But that is far from the whole story! In this article we will look at some of the other things that this powerful little file can do. In part two we have 7 Magic Tricks that you can perform with .htaccess, but first let’s have a look at the file itself.

What is the .htaccess file? 

The .htaccess file is a text file which resides in your main site directory and/or in any subdirectory of your main directory. There can be just one, there can be a separate one in each directory or you may find or create one just in a specific directory.

Any commands in a .htaccess file will affect both the directory it is in and any subdirectories of that directory. Thus if you have just one, in your main directory, it will affect your whole site. If you place one in a subdirectory it will affect all the contents of that directory.

Some Important Points 

Windows does not use the .htaccess system. I believe there are ways of doing the things .htaccess does on Windows servers but that is a story for another day and I am afraid I will not be telling it – it just isn’t as simple or as elegant as the way Apache manages things in my humble opinion! So unless you are on a Linux/Unix server, this article is no good to you. Sorry.

A warning you will commonly see is that changing the .htaccess file on a server that has FrontPage extensions installed will at best not work and at worst make a complete mess of your extensions. I have to say that this has not been my experience and I have done a fair bit of messing with .htaccess files on FrontPage sites, including using .htaccess for authentication. However do any of these things at your own risk – I cannot be responsible for any harm they might cause.

Your host may not support alteration of the .htaccess file; either contact them first and ask before you make changes or proceed with caution and be sure you have a backup of the original file in case of problems.

Oh! And none of the ‘Magic Tricks’ described in this article are either magic or tricks. They just seem that way!

Working With Your .htaccess File 

Sometimes the first problem is finding it! When you FTP to your site the .htaccess file is generally the first one displayed in a directory if it exists.

Some servers are configured to hide files whose names begin with a period. Your FTP client allows you to chose to display these files. In WS_FTP you can do this by entering -la or -al as indicated in the image on the left and then clicking Refresh. Other clients may use a different method – check the help files in yours.

Editing should be done in a text editor, such as NotePad. You should not edit .htaccess files in editors such as FrontPage. The best thing to do is download a copy of your .htaccess file to your computer, edit it, and upload again, remembering to save a copy of the original in case of errors.

If you do not already have a .htaccess file you can create one in NotePad, it is just a simple text file. However when saving it to the server you may need to rename it from .htaccess.txt to just .htaccess. The two are NOT the same. In fact .htaccess is an extension – to a file with no name!

It is very important when entering commands in your file that each is entered on a new line and that the lines do not wrap. If you find that when you paste any of the commands in this article into your file that the lines are not breaking or are wrapping you will need to correct this.

You must upload and download your .htaccess file in ASCII mode, not BINARY.

So, What about the Magic Tricks? Read on! 

Well, first another boring bit! To prevent people from being able to see the contents of your .htaccess file, you need to place the following code in the file:

<Files .htaccess> order allow,deny deny from all </Files>

Be sure to format that just as it is above, with each line on a new line as shown. There is every likelihood that your existing .htaccess file, if you have one, includes those lines already.

Magic Trick No. 1: Redirect to Files or Directories

You have just finished a major overhaul on your site, which unfortunately means you have renamed many pages that have already been indexed by search engines, and quite possibly linked to or bookmarked by users. You could use a redirect meta tag in the head of the old pages to bring users to the new ones, but some search engines may not follow the redirect and others frown upon it.

.htaccess leaps to the rescue! 

Enter this line in your .htaccess file:

Redirect permanent /oldfile.html http://www.domain.com/filename.html

You can repeat that line for each file you need to redirect. Remember to include the directory name if the file is in a directory other than the root directory:

Redirect permanent /olddirectory/oldfile.html http://www.domain.com/newdirectory/newfile.html

If you have just renamed a directory you can use just the directory name:

Redirect permanent /olddirectory http://www.domain.com/newdirectory

(Note:The above commands should each be on a single line, they may be wrapping here but make sure they are on a single line when you copy them into your file.)

This has the added advantage of preventing the increasing problem on the Internet, as people change their sites, of ‘link rot’. Now people who have linked to pages on your site will still have functioning links, even if the pages have changed location.

Magic Trick No. 2: Change the Default Directory Page 

In most cases the default directory page is index.htm or index.html. Many servers allow a range of pages called index, with a variety of extensions, to be the default page.

Suppose though (for reasons of your own) you wish a page called honeybee.html or margarine.html to be a directory home page?

No problem. Just put the following line in your .htaccess file for that directory:

Directory Index honeybee.html

You can also use this command to specify alternatives. If the first filename listed does not exist the server will look for the next and so on. So you might have:

Directory Index index.html index.htm honeybee.html margarine.html

(Again, the above should all be on a single line)

Magic Trick No. 3: Allow/Prevent Directory Browsing 

Most servers are configured so that directory browsing is not allowed, that is if people enter the URL to a directory that does not contain an index file they will not see the contents of the directory but will instead get an error message. If your site is not configured this way you can prevent directory browsing by adding this simple line to your .htaccess file:

IndexIgnore */*

But there may be times when you want to allow browsing, perhaps to allow access to files for downloading or for whatever reason, on a server configured not to allow it. You can override the servers settings with this line:

Options +Indexes

Easy!

Magic Trick No. 4: Allow SSI in .html files 

Most servers will only parse files ending in .shtml for Server Side Includes. You may not wish to use this extension, or you may wish to retain the .htm or .html extension used by files prior to your changing the site and using SSI for the first time.

Add the following to your .htaccess file:

AddType text/html .html AddHandler server-parsed .html AddHandler server-parsed .htm

You can add both extensions or just one.

Remember though that files which must be parsed by the server before being displayed will load more slowly that standard pages. If you change things as above, the server will parse all .html and .htm pages, even those that do not contain any includes. This can significantly, and unnecessarily, slow down the loading of pages without includes.

Magic Trick No 5: Keep Unwanted Users Out 

You can ban users by IP address or even ban an entire range of IP addresses. This is pretty drastic action, but if you don’t want them, it can be done very easily.

Add the following lines:

order allow,deny deny from 123.456.78.90 deny from 123.456.78 deny from .aol.com allow from all

The second line bans the IP address 123.456.78.90, the third line bans everyone in the range 123.456.78.1 to 123.456.78.999 and so is much more drastic. The fourth line bans everyone from AOL. A somewhat excessive display of power perhaps!

One thing to bear in mind here it that banned users will get a 403 error – “You do not have permission to access this site”, which is fine unless you have configured a custom error for this page which in fact appears to let them in. So bear that in mind and if you are banning users for whatever reason make sure your 403 error message is a dead end.

Magic Trick No. 6: Prevent Linking to Your Images 

The greatest and most irritating bandwidth leech is having someone link to images on your site. You can foil such thieves very easily with .htaccess. Copy the following into your .htaccess file:

RewriteEngine on RewriteCond %{HTTP_REFERER} !^$ RewriteCond %{HTTP_REFERER} !^http://(www.)?domain.com/.*$ [NC] RewriteRule .(gif|jpg)$ - [F]

You don’t need to understand any of that! Just change ‘domain.com’ to the name of your domain.

(Again each command should be on a single line. There are 4 lines above, each starting with ‘Rewrite’)

If you want to really let them know they have been rumbled why not make an image like the one below (or take this one if you like)

call it stealing.gif, save it to your images file and add the following line after the code abov

RewriteRule .(gif|jpg)$ http://www.domainname.com/images/stealing.gif [R,L]

(The above command should be on a single line)

Magic Trick No 7: Stop the Email Collectors 

While you positively want to encourage robot visitors from the search engines, there are other less benevolent robots you would prefer stayed away. Chief among these are those nasty ‘bots that crawl around the web sucking email addresses from web pages and adding them to spam mail lists.

RewriteCond %{HTTP_USER_AGENT} Wget [OR]
RewriteCond %{HTTP_USER_AGENT} CherryPickerSE [OR]
RewriteCond %{HTTP_USER_AGENT} CherryPickerElite [OR]
RewriteCond %{HTTP_USER_AGENT} EmailCollector [OR]
RewriteCond %{HTTP_USER_AGENT} EmailSiphon [OR]
RewriteCond %{HTTP_USER_AGENT} EmailWolf [OR]
RewriteCond %{HTTP_USER_AGENT} ExtractorPro RewriteRule ^.*$ X.html [L]

Note that at the end of each line for a named robot there appears an ‘[OR]’ – don’t forget to include that if you add any others to this list.

This is by no means foolproof. Many of these sniffers do not identify themselves and it is almost impossible to create an exhaustive list of those that do. It’s worth a try though if it even keeps some away. The above is as many as I could find.

….and Finally 

There is one very important area of the .htaccess file’s use that we have not really mentioned and that is its use for user authentication. It is perfectly possible to configure your .htaccess files by hand to control access to directories on your site, but this is rarely necessary.

In most cases your host will provide a method to allow you to much more easily configure the file from your hosting control panel and there are a myriad of Perl scripts that will allow you to set up full user management systems by harnessing the power of .htaccess.

If you do want to go it alone there is a tutorial here that will get you there: http://www.apacheweek.com/features/userauth

If you are looking for scripts there a many here: http://www.hotscripts.com/Perl/Scripts_and_Programs/Password_Protection/

Two scripts that I have used and can recommend are:

1. Locked Area 
The free version will be adequate for many situations, though both versions will give you control over access to one directory and its contents only.
http://www.locked-area.com/html/

2. Password Manager 
Allows you very sophisticated control over access to multiple directories. Not cheap but very good value for money.
http://www.cgi-world.com/password_manager.html

Have fun!

Article reprinted with permission from Thomas Brunt’s OutFront, a Microsoft FrontPage learning community
http://www.outfront.net

About the Author:
Katherine Nolan
OutFront Moderator

http://www.inkkdesign.com/

Upload/Download using your Private Key in Linux or OSX

If you use Linux or Mac OSX, this is a tutorial for transferring files between our server and your computer

First of all, if you are a web hosting client of kaslnetwork.com and have not received your private key by email, then please contact me and I will email it to you.

Now that you have your private key (called something like coyote_kaslit_key), put it on your Desktop, open up your terminal and run these commands (replace coyote with your username (if your key is named something completelydifferent, then use the name of your key):

cd ~
mkdir .ssh
cp ~/Desktop/coyote_kaslit_key .ssh/
chmod 700 .ssh
chmod 600 .ssh/*
ssh-add .ssh/coyote_kaslit_key
sftp coyote@kaslnetwork.com

If you get a prompt after that last command that looks like sftp> then it worked and you are now on our server in an encrypted file transfer program.

You can now use your favorite sftp client (some ftp clients include sftp connection methods) to connect. Be sure to use your private key (coyote_kaslit_key) to connect to our server on port 22