make a new directory:
mkdir testfolder

change directory:
cd testfolder

Create an empty file:
touch test.txt

command line editors:
nano, vi, emacs

count number of words in a file:
wc filename

get manual on a command:
man wc

go to home folder:
cd ~

show current folder:
pwd

go to previous folder:
cd -

TAB is autocomplete

go to the start of a line:
ctrl+a

go to the end of a line:
ctrl+e

the up arrow shows your previous command.

go up one folder:
cd ..

delete directory (it must be empty):
rmdir testfolder

delete contents of a folder:
rm testfolder/*.*

move a file:
mv filename testfolder

move and rename:
mv  filename testfolder/newfilename

copy a file:
cp folder1/filename folder2/

copy a contents of folder
cp -r folder1/* folder2/

copy a whole folder:
cp -r folder1 folder2/

recursively delete folder AND contents:
rm -rf testfolder

show all files:
ls -a

show details, human readable:
ls -lh

get fifth column from results using awk:
ls -lh | awk '{print $5}'

show in order of creation:
ls -ltr

show command history:
history

remove the line numbers:
history | cut -c 8-

Creating an alias:
edit the .bashrc file in your home folder
add the command with no spaces around the equals sign:
alias cmd="nano ~/commands.txt"
alias rm='rm -v -i'
alias open='gvfs-open'

Search a directory for a file:
find . -name *.py

weird thing about find, you must use "" if there is something with that name in the folder.

find a word in a file:
grep 'def' *.*

the filesystem structure
/	The slash / character alone denotes the root of the filesystem tree.
/bin Stands for binaries and contains certain fundamental utilities, such as ls or cp
/boot Contains all the files needed for successful booting process.
/dev  Stands for devices. Contains file representations of peripheral devices and pseudo-devices.
/etc Contains system-wide configuration files and system databases
/home Contains user home directories on Linux and some other systems. In the original version of Unix, home directories were in /usr instead.[15] Some systems use or have used different locations still: OS X has home directories in /Users, older versions of BSD put them in /u, FreeBSD has /usr/home.
/lib Originally essential libraries: C libraries, but not Fortran ones. On modern systems, it contains the shared libraries needed by programs in /bin, and possibly loadable kernel module or device drivers. Linux distributions may have variants /lib32 and /lib64 for multi-architecture support.
/media Default mount point for removable devices, such as USB sticks, media players, etc.
/mnt Stands for mount. Empty directory commonly used by system administrators as a temporary mount point.
/opt Contains locally installed software.
/proc procfs virtual filesystem showing information about processes as files.
/root The home directory for the superuser root - that is, the system administrator. This account's home directory is usually on the initial filesystem, and hence not in /home (which may be a mount point for another filesystem) in case specific maintenance needs to be performed, during which other filesystems are not available. Such a case could occur, for example, if a hard disk drive suffers physical failures and cannot be properly mounted.
/sbin Stands for "system (or superuser) binaries" and contains fundamental utilities, such as init, usually needed to start, maintain and recover the system.
/srv Server data (data for services provided by system).
/sys In some Linux distributions, contains a sysfs virtual filesystem, containing information related to hardware and the operating system.
/tmp A place for temporary files not expected to survive a reboot. Many systems clear this directory upon startup or use tmpfs to implement it.
/usr The "user file system": originally the directory holding user home directories,[15] but already by the Third Edition of Research Unix, ca. 1973, reused to split the operating system's programs over two disks (one of them a 256K fixed-head drive) so that basic commands would either appear in /bin or /usr/bin.[17] It now holds executables, libraries, and shared resources that are not system critical, like the X Window System, KDE, Perl, etc. In older Unix systems, user home directories might still appear in /usr alongside directories containing programs, although by 1984 this depended on local customs.[13]
/include Stores the development headers used throughout the system. Header files are mostly used by the #include directive in C language, which historically is how the name of this directory was chosen.
/lib Stores the needed libraries and data files for programs stored within /usr or elsewhere.
/libexec Holds programs meant to be executed by other programs rather than by users directly. E.g., the Sendmail executable may be found in this directory.[18] Not present in the FHS until 2011;[19] Linux distributions have traditionally moved the contents of this directory into /usr/lib, where they also resided in 4.3BSD.
/local Resembles /usr in structure, but its subdirectories are used for additions not part of the operating system distribution, such as custom programs or files from a BSD Ports collection. Usually has subdirectories such as /usr/local/lib or /usr/local/bin.
/share Architecture-independent program data. On Linux and modern BSD derivatives, this directory has subdirectories such as man for manpages, that used to appear directly under /usr in older versions.
/var Stands for variable. A place for files that may change often - especially in size, for example e-mail sent to users on the system, or process-ID lock files.
/log Contains system log files.
/mail The place where all incoming mails are stored. Users (other than root) can access their own mail only. Often, this directory is a symbolic link to /var/spool/mail.
/spool Spool directory. Contains print jobs, mail spools and other queued tasks.
/tmp The /var/tmp directory is a place for temporary files which should be preserved between system reboots.

cat /proc/meminfo
cat /proc/cpuinfo
cat /proc/version

devices:
cat /dev/input/mouseX
cat /dev/input/byid/keyboard

run previous command as sudo:
sudo !!

The standard streams (stdout, stderr, stdin):
stdin(0): keyboard
stdout(1): normal output to screen
stderr(2): error output to screen

output something to a file:
ls > out.txt (this is the same as 1>)

append to a file:
ls >> out.txt

redirect errors to a file:
ls 2> err.txt

search the whole filesystem and redirect errors to nowhere
find / -name "*.py" 2> /dev/null

redirect stderr to stdout:
ls > out.txt 2>&1

2>&1 redirects the standard error to the standard output so they appear i
together and can be jointly redirected to a file. (Writing just 2>1 would 
write it out to a file called 1


pipes: | take the output of one command and pipe it to the input of another

translate byte values into ascii
cat keyboard.txt | tr -cd 'a-f0-9'

use tee to redirect to a file AND stdout:
ls | tee all.txt

get a random number from /dev:
cat /dev/urandom | tr -cd 'a-f0-9' | head -c 32

writing a script:
#!/bin/bash
var="5"
echo "hello, $1, the value of variable is $var"
if [ $var -gt 8 ]; then
   echo "var is greater than 8"
fi


another sample script:
#!/bin/bash
echo "converting $1 to $2"
for file in *.$1; do
    filename=`echo $file | sed s/$1/$2/`
    echo "converting $file to $filename"
    #convert $file $filename
done 

chmod +x script.sh
./script.sh

The & makes the command run in the background.

make command run in the background after executing:
ctrl+z
bg


creating a symlink
ln -s ~/filename

debugging stuff:
logs are located in /var/log/

show all driver messages:
dmesg

use less when opening a big file ( it prints to the screen as it reads):
less filename

use tail to read the end of a logfile (-n 100 reads the last 100 lines):
tail -n 100 filename

read a file and keep reading it (follow):
tail -n 100 -f filename

ssh, rsync and scp:
if you set up ssh server on your machine then install fail2ban

ssh user@machine.movidius.com

debugging it:
ssh user@machine.movidius.com -v
tcptraceroute github.com 22


scp filename user@machine.movidius.com:

if it fails to copy then you can resume it, the -u only copies new files:
rsync -u --partial --progress --rsh=ssh user@host:remote_file local_file

screen, a must for dodgy ssh connections:
screen - start a new screen session
Ctrl a c - Creates a new screen session so that you can use more than one screen session at once.
Ctrl a n - Switches to the next screen session (if you use more than one).
Ctrl a p - Switches to the previous screen session (if you use more than one).
Ctrl a d - Detaches a screen session (without killing the processes in it - they continue).

list all screens:
screen -ls
reattach to a screen:
screen -d

Edit your crontab file:
crontab -e
minute hour  day-of-month month day-of-week command

every minute:
* * * * *
every day at 7pm:
0 19 * * *
run something at reboot:
@reboot root /var/kiosk/btest.sh
the cron jobs are logged to /var/log/syslog:
grep CRON /var/log/syslog

----------------------------cool commands----------------------------

show all the environment variables.
printenv | sort -h

show all processes:
top
show it with nice colors:
htop

list all processes:
ps -ef

kill <processnumber>

maximum killage:
kill -9 <psNo>

kill all processes of this type:
killall <psName>

get exif data from images:
exiftool imagename
or
exiv2 imagename

combine mp3s together:
cat song1.mp3 song2.mp3 > song3.mp3

AWK and SED!
ls -l | awk '{n+=$5}END{print n}'  <-add up all the file sizes
cat *.dat | awk '{print$2}'  <-concatenate files and read the second column
svn status | awk '{print $2}' | xargs svn add
sed s/day/night/ < moo.txt > new.txt <- replace day with night, write to file
sed -e 's/day/night/g' moo.txt > blah.txt <- s(ubsitiute) g(lobal) e(add script)
sed -i 's/foo/bar/' * <- i(use same file)
sed -i 's/foo/bar/g'  <- global, all matches in a line replaced

show differences between two files:
diff file1 file2
or
meld file1 file2

create a video from images:
ls *.jpg > frames.txt
mencoder mf://@frames.txt -mf w=1920:h=1080:fps=20:type=jpg -ovc x264 -x264encopts subq=6:partitions=all:8x8dct:me=umh:frameref=5:bframes=3:b_pyramid=normal:weight_b -o a.avi

convert 4k video to 1080p:
mencoder <filename> -ovc x264 -x264encopts subq=6:partitions=all:8x8dct:me=umh:frameref=5:bframes=3:b_pyramid=normal:weight_b -vf scale=1920:1080 -oac copy -o reduced<filename>

tar and untar:
tar -cvpf file.tar archive folder
c-reate v-erbose p-reservepermission f-ile

tar -xzvf file.tar.gz
x-tract z-ip v-erbose f-ile take out the z for .tar


make a gif:
convert *.jpg pictures.gif

ping a website:
ping reddit.com

get network details:
ifconfig (ipconfig in windows)


download a website:
wget https://reddit.com

pull all links out of a webpage:
grep -Po '(?<=href=")[^"]*' index.html

wget -q -O- reddit.com/r/thedonald | grep -o -i 'Trump' | wc -w

get word count from pdf
pdftotext filename.pdf - | tr -d '.' | wc -w

restart from the commandline:
sudo shutdown -r now


backing up a database:
/usr/bin/mysqldump --opt -u <username> -p<pass> <database> > /tmp/backup.sql

echo "This is the body."| mailx -s "mailx Test1" jonathanbyrn@gmail.com

:w !sudo tee %   <- save a write protected file in VIM

show disk usage for a folder and subfolders, sorted by size:
du -h --max-depth=1 | sort -h

find all executable files:
find . -type f -executable -print

Add a prefix to all files in a folder:
rename 's/^/wide/' *

see where package contents are installed to:
dpkg -L <packagename>

remove all trailing white space
sed -i 's/[ \t]*$//' "$1"

trim the first 50 lines off a file:
sed -i '1,50d' filename
truncate last 50 lines:
head -n -50 myfile.txt

grep and replace using sed:
grep -l add_gene *.py | xargs sed -i 's/add_gene/add_extra/g'


Automatically pep8 all your code:
find . -iname "*.py" | xargs -I filename autopep8 --in-place --aggressive --aggressive filename
also use yapf -i

change ownership of a folder, its contents and all subfolders:
              user  group
sudo chown -R byrnj:byrnj public_html/

encrypt a file
gpg -c moo.txt
gpg moo.gpg

check which process is hogging the ram:
ps -e -o pid,vsz,comm= | sort -n -k 2
sort by second column:
sort -k 2

Profile a piece of python code:
python -m cProfile -o profile.pyprof -s 'time' myScript.py
pyprof2calltree -i profile.pyprof -k


No password ssh (from a to b):
if you have not generated a key then (ONLY IF NO KEY):
Be careful of DSA vs RSA
ssh-keygen -t rsa <- generate key for autologin on machine a
otherwise just do this
ssh-copy-id -i ~/.ssh/id_rsa.pub name@server <- copy file across from machine a
cat .ssh/id_rsa.pub | ssh b@B 'cat >> .ssh/authorized_keys' <- alternative way
if it doesnt work immediately then type:
ssh-add on machine a to register the keys