redirect

show stdout but redirect all to file

[!NOTE] references:

  • redirect overview
              || visible in terminal ||   visible in file   || existing
      Syntax  ||  StdOut  |  StdErr  ||  StdOut  |  StdErr  ||   file
    ==========++==========+==========++==========+==========++===========
        >     ||    no    |   yes    ||   yes    |    no    || overwrite
        >>    ||    no    |   yes    ||   yes    |    no    ||  append
              ||          |          ||          |          ||
       2>     ||   yes    |    no    ||    no    |   yes    || overwrite
       2>>    ||   yes    |    no    ||    no    |   yes    ||  append
              ||          |          ||          |          ||
       &>     ||    no    |    no    ||   yes    |   yes    || overwrite
       &>>    ||    no    |    no    ||   yes    |   yes    ||  append
              ||          |          ||          |          ||
     | tee    ||   yes    |   yes    ||   yes    |    no    || overwrite
     | tee -a ||   yes    |   yes    ||   yes    |    no    ||  append
              ||          |          ||          |          ||
     n.e. (*) ||   yes    |   yes    ||    no    |   yes    || overwrite
     n.e. (*) ||   yes    |   yes    ||    no    |   yes    ||  append
              ||          |          ||          |          ||
    |& tee    ||   yes    |   yes    ||   yes    |   yes    || overwrite
    |& tee -a ||   yes    |   yes    ||   yes    |   yes    ||  append
    
  • Redirect terminal output to file
  • Bash: Redirect stdout and stderr
  • show Command Output Redirection.pdf
  • 3.6 Redirections
  • * Beyond Linux From Scratch

tips:

  • echo to stderr
    $ echo 'abcdefg' >/dev/null
    $ echo 'abcdefg' >/dev/null >&2
    abcdefg
    
$ bash -c "echo a;bahs;echo b;bhas" >>file 2> >( tee -a file >&2 )
bash: line 1: bahs: command not found
bash: line 1: bhas: command not found

$ cat file
a
bash: line 1: bahs: command not found
b
bash: line 1: bhas: command not found
  • stderr output with filter

    $ bash -c "echo a;bahs;echo b;bhas" >>file 2> >( tee -a file 2>&1 | grep -v bahs >&2 )
    bash: line 1: bhas: command not found
    # or
    $ rm -rf file; bash -c "echo a;bahs;echo b;bhas" >>file 2> >( tee -a file | grep -v bahs >&2 )
    bash: line 1: bhas: command not found
    
    $ cat file
    a
    bash: line 1: bahs: command not found
    b
    bash: line 1: bhas: command not found
    
  • or

    $ bash -c "set -e; echo a;bahs;echo b;bhas" >>cmd.out 2> >( tee -a cmd.out >&2 )
    bash: line 1: bahs: command not found
    
    $ cat cmd.out
    a
    bash: line 1: bahs: command not found
    

time & date

show date with timezone

# with quotes
$ TZ=':Asia/Shanghai' date

# or without quotes
$ TZ=America/Los_Angeles date

show cal

$ cal -y | tr '\n' '|' | sed "s/^/ /;s/$/ /;s/ $(date +%e) / $(date +%e | sed 's/./#/g') /$(date +%m | sed s/^0//)" | tr '|' '\n'
                             2014
      January               February               March
Su Mo Tu We Th Fr Sa  Su Mo Tu We Th Fr Sa  Su Mo Tu We Th Fr Sa
          1  2  3  4                     1                     1
 5  6  7  8  9 10 11   2  3  4  5  6  7  8   2  3  4  5  6  7  8
12 13 14 15 16 17 18   9 10 11 12 13 14 15   9 10 11 12 13 14 15
19 20 21 22 23 24 25  16 17 18 19 20 21 22  16 17 ## 19 20 21 22
26 27 28 29 30 31     23 24 25 26 27 28     23 24 25 26 27 28 29
                                            30 31

       April                  May                   June
Su Mo Tu We Th Fr Sa  Su Mo Tu We Th Fr Sa  Su Mo Tu We Th Fr Sa
       1  2  3  4  5               1  2  3   1  2  3  4  5  6  7
 6  7  8  9 10 11 12   4  5  6  7  8  9 10   8  9 10 11 12 13 14
13 14 15 16 17 18 19  11 12 13 14 15 16 17  15 16 17 18 19 20 21
20 21 22 23 24 25 26  18 19 20 21 22 23 24  22 23 24 25 26 27 28
27 28 29 30           25 26 27 28 29 30 31  29 30


        July                 August              September
Su Mo Tu We Th Fr Sa  Su Mo Tu We Th Fr Sa  Su Mo Tu We Th Fr Sa
       1  2  3  4  5                  1  2      1  2  3  4  5  6
 6  7  8  9 10 11 12   3  4  5  6  7  8  9   7  8  9 10 11 12 13
13 14 15 16 17 18 19  10 11 12 13 14 15 16  14 15 16 17 18 19 20
20 21 22 23 24 25 26  17 18 19 20 21 22 23  21 22 23 24 25 26 27
27 28 29 30 31        24 25 26 27 28 29 30  28 29 30
                      31

      October               November              December
Su Mo Tu We Th Fr Sa  Su Mo Tu We Th Fr Sa  Su Mo Tu We Th Fr Sa
          1  2  3  4                     1      1  2  3  4  5  6
 5  6  7  8  9 10 11   2  3  4  5  6  7  8   7  8  9 10 11 12 13
12 13 14 15 16 17 18   9 10 11 12 13 14 15  14 15 16 17 18 19 20
19 20 21 22 23 24 25  16 17 18 19 20 21 22  21 22 23 24 25 26 27
26 27 28 29 30 31     23 24 25 26 27 28 29  28 29 30 31
                      30

synchronize date and time with another server over ssh

[!NOTE] inspired from

$ date --set="$(ssh [username]@[sshserver] date)"

download and extract

  • *.gz

    $ wget -O - http://example.com/a.gz | tar xz
    
  • *.zip

    $ curl -fsSL https://services.gradle.org/distributions/gradle-4.7-all.zip | bsdtar xzf - -C <EXTRACT_PATH>
    
    • with zip password
      $ curl -fsSL \
             -u<user>:<passwd> \
             https://path/to/file.zip \
             | bsdtar -xzf- --passphrase <PASSWD_OF_ZIP> - -C <EXTRACT_PATH>
      
  • *.tar.gz

    $ curl -fsSL https://path/to/file.tar.gz | tar xzf - -C <EXTRACT_PATH>
    
    • example
      $ curl -fsSL \
             -j \
             -k \
             -L \
             -H "Cookie: oraclelicense=accept-securebackup-cookie" \
             http://download.oracle.com/otn-pub/java/jdk/8u181-b13/96a7b8442fe848ef90c96a2fad6ed6d1/jdk-8u181-linux-x64.tar.gz \
             | tar xzf - -C '/opt/java'
      

check file without extract

$ tar -Oxvf myfile.tgz path/to/my.sh | less

extract jar

$ unzip <jar-name>.jar -d <target-folder>
  • Delete files from JAR without unzip
    $ zip -d <jar-name>.jar <path/to/file.txt>
    

recursive download

[!TIP] references:

params shortcuts :

SHORT PARAMS LONG PARAMS
-r --recursive
-m --mirror
-l --level
-k --convert-links
-K --backup-converted
-P --directory-prefix
-nv --no-verbose
-nc --no-clobber
-nd --no-directories
-nH --no-host-directories
-np --no-parent
-x --force-directories
-b --background
-v --verbose
-p --page-requisites
$ wget --recursive                       \     # -r
       --user=admin                      \
       --password=admin                  \     # --ask-password
       --auth-no-challenge               \     # optional
       --no-host-directories             \     # -nH
       --no-parent                       \     # -np
       --reject '*.html*'                \
       --directory-prefix=./             \     # -P
       --include-directories=local-dir   \     # -I
  http://example.com/remote-dir

# or
$ wget --recursive                                   \    # -r
       --no-parent                                   \    # -np : will not crawl links in folders above the base of the URL
       --convert-links                               \    # -k  : convert links with the domain name to relative and uncrawled to absolute
       --random-wait --wait 3 --no-http-keep-alive   \    #       do not get banned
       --no-host-directories                         \    # -nH : do not create folders with the domain name
       --execute robots=off --user-agent=Mozilla/5.0 \    #       I AM A HUMAN!!!
       --level=inf --accept '*'                      \    # -l  : do not limit to 5 levels or common file formats
       --reject="index.html*"                        \    #       use this option if you need an exact mirror
       --cut-dirs=0                                  \    #       replace 0 with the number of folders in the path, 0 for the whole domain
  $URL
  • mirror whole website
    $ wget -m https://www.baeldung.com/
    

download directly

$ wget -r -np -nH --cut-dirs=1 https://www.baeldung.com/linux
# or
$ wget -r --no-parent --no-host-directories --cut-dirs=1 https://www.baeldung.com/linux
  • with level

    $ wget -r -np -l 2 https://www.baeldung.com/linux/
    # or
    $ wget -r --no-parent --level=2 https://www.baeldung.com/linux/
    
  • with credentials

    • .wgetrc
      $ cat ~/.wgetrc
      user=admin
      password=admin
      
    • from cmd

      $ wget --user=admin --password=admin
      
      # or
      $ wget --user=admin --ask-password
      Password for user 'admin': admin
      
    • ignore credentials
      $ wget --no-check-certificate
      
  • converting links for local viewing

    $ wget -r --no-parent --convert-links https://www.baeldung.com/linux/category/web
    
  • switching off robot exclusion

    $ wget -r --level=1 --no-parent --convert-links -e robots=off -U="Mozilla"
    

compress

zip package with dot-file

  • .[^.]*

    $ zip name.zip * .[^.]*'
    
  • shopt -s dotglob

    [!NOTE|label:references:]

    $ shopt -s dotglob
    $ zip name.zip *
    
  • .

    $ zip -r name.zip .
    

remove dot-file without skipping '..' '.' issue

[!NOTE|label:references:]

echo 256 colors

[!TIP] see also:

$ for i in {0..255}; do echo -e "\e[38;05;${i}m${i}"; done | column -c 80 -s ' '; echo -e "\e[m"
# or
$ yes "$(seq 1 255)" | while read i; do printf "\x1b[48;5;${i}m\n"; sleep .01; done

commands

ls

[!TIP|label:references]

PWD's secrets

$ l | grep bc
lrwxrwxrwx 1 marslo marslo   37 Mar  4 00:25 bc -> /home/marslo/Tools/Git/BrowserConfig//
$ cd bc/
$ pwd -L
/home/marslo/bc
$ pwd -P
/home/marslo/Tools/Git/BrowserConfig

list the command startsWith

$ compgen -c "system-config-"
system-config-authentication
system-config-authentication
system-config-date
system-config-firewall
system-config-firewall-tui
system-config-kdump
system-config-keyboard
system-config-keyboard
system-config-network
system-config-network
system-config-network-cmd
system-config-network-cmd
system-config-network-tui
system-config-printer
system-config-printer-applet
system-config-services
system-config-services
system-config-users

fuzzy find for commands

$ apropos editor | head
Git::SVN::Editor (3pm) - commit driver for "git svn set-tree" and dcommit
INIFILE (1)          - OpenLink Virtuoso Opensource ini File Editor
atobm (1)            - bitmap editor and converter utilities for the X Window System
bitmap (1)           - bitmap editor and converter utilities for the X Window System
bmtoa (1)            - bitmap editor and converter utilities for the X Window System
ed (1)               - line-oriented text editor
editor (1)           - Nano's ANOther editor, an enhanced free Pico clone
editres (1)          - a dynamic resource editor for X Toolkit applications
ex (1)               - Vi IMproved, a programmers text editor
gedit (1)            - text editor for the GNOME Desktop

batch commands

batch rename

$ l
total 4.0K
-rw-r--r-- 1 marslo marslo 10 Feb 21 00:43 a.b
$ rename -v 's/\./_/g' *
a.b renamed as a_b
$ l
total 4.0K
-rw-r--r-- 1 marslo marslo 10 Feb 21 00:43 a_b

xargs rename

$ shopt -s extglob
$ ls git-+([0-9])*.png
git-0.png  git-1.png  git-2.png  git-3.png  git-4.png  git-5.png

$ ls --color=none git-+([0-9])*.png | xargs rename -v 's/git-/git-for-windows-/'
'git-0.png' renamed to 'git-for-windows-0.png'
'git-1.png' renamed to 'git-for-windows-1.png'
'git-2.png' renamed to 'git-for-windows-2.png'
'git-3.png' renamed to 'git-for-windows-3.png'
'git-4.png' renamed to 'git-for-windows-4.png'
'git-5.png' renamed to 'git-for-windows-5.png'

batch move

[!NOTE] -I replace-str

$ mkdir backup-folder && ls | grep -Ze ".*rar" | xargs -d '\n' -I {} mv {} backup-folder

batch copy

reference:

$ ls -1 a/b/* 11 12 | xargs cp -t copy-target-folder/

copy single file to multipule folders

$ echo dir1 dir2 dir3 | xargs -n 1 cp file1

# or
$ echo dir{1..10} | xargs -n 1 cp file1

ldapsearch

[!NOTE] enhaanced script

  • marslo/mytools reference:
  • ldapsearch Examples
  • The ldapsearch Tool
  • Querying AD with ldapsearch
    -LLL                                          # just a particular way to display the results
    -H ldap://wspace.mydomain.com                 # the URL where the LDAP server listens
    -x                                            # use simple authentication, not SASL
    -D 'user1'                                    # the account to use to authenticate to LDAP
    -w 'user1password'                            # the password that goes with the account on the previous line
    -E pr=1000/noprompt                           # ask the server for all pages, don't stop after one
    -b 'ou=mydomain,dc=wspace,dc=mydomain,dc=com' # the base of the search. We don't want results from e.g. 'ou=blah,dc=wspace,dc=mydomain,dc=com'
    '(&(objectClass=person)(uidNumber=*))'        # Ask for any entry that has attributes objectClass=person and uidNumber has a value
    SAMAccountName uid uidNumber                  # Show only these attributes
    

search specific user

[!NOTE] info :

  • ldap url : ldaps://ldap.mydomain.com:636
  • base search base : dc=mydomain,dc=com
  • login user : user1 / user1password
  • search : user2

remove #refldaps://..

  • remove #.*
    $ ldapsearch ... | sed -r '/^(#.*)$/d'
    
  • remove empty lines
    $ ldapsearch ... | sed -r '/^\s*$/d'
    
  • remove all

    $ ldapsearch ... | sed -r '/^(#.*)$/d;/^\s*$/d'
    
    # or
    $ ldapsearch ... | sed -r '/(^#.*)|(^\s*)$/d'
    
$ ldapsearch \
    -LLL \
    -x \
    -H 'ldaps://ldap.mydomain.com:636' \
    -b 'dc=mydomain,dc=com' \
    -D 'user1' \
    -w 'user1password' \
    CN='user2'
  • or insert password via interactive mode ( -W )
    $ ldapsearch \
        -LLL \
        -x \
        -H 'ldaps://ldap.mydomain.com:636' \
        -b 'dc=mydomain,dc=com' \
        -W \
        -D 'user1' \
        CN='user2'
    

filter DN field only

$ ldapsearch \
    [-LLL \]
    -H 'ldaps://ldap.mydomain.com:636' \
    -b 'dc=mydomain,dc=com' \
    -x \
    -D 'user1' \
    -w 'user1password' \
    CN='user2' \
    DN

filter SAMAccountName, uid and uidNumber only

[!TIP] filter base on base DN (OU=Person,DC=mydomain,DC=com)

$ ldapsearch \
    -LLL \
    -x \
    -H 'ldaps://ldap.mydomain.com:636' \
    -b 'ou=Workers,dc=mydomain,dc=com' \
    -D 'user1' \
    -w 'user1password' \
    -E 'pr=1000/noprompt' \
    '(&(objectClass=user)(sAMAccountName=*))' \
    SAMAccountName uid uidNumber DN

filter particular group

$ ldapsearch \
    -x \
    -H 'ldaps://ldap.mydomain.com:636' \
    -b 'OU=DL,OU=Groups,OU=GLOBAL,OU=Sites,dc=mydomain,dc=com' \
    -D 'user1' \
    -w 'user1password' \
    -E 'pr=1000/noprompt' \
    '(&(objectClass=group)(CN=*))'
  • search particular group (cn=DL-name-group)
    $ ldapsearch \
        -x \
        -H 'ldaps://ldap.mydomain.com:636' \
        -b 'OU=DL,OU=Groups,OU=GLOBAL,OU=Sites,dc=mydomain,dc=com' \
        -D 'user1' \
        -w 'user1password' \
        -E 'pr=1000/noprompt' \
        CN='DL-name-group'
    

get userCertificates

[!NOTE|label:references:]

  • Problems with ldap userCertificate attribute
  • 9.2. Certificate Publishing

    # convert a pem certificate into der
    openssl x509 -outform DER -in incert.pem  -out outcert.der
    
    # created LDIF file
    ldif -b "usercertificate;binary" < outcert.der  > cert.ldif
    
    # creates an usercertificate attribute encoded in base64
    ldapmodify -x -W -D "cn=Manager,dc=yourorg,dc=com" -f cert.ldif
    
  • get cert info

    $ ldapsearch marslo userCertificate |
                 awk '{print $NF}' |
                 xargs -i bash -c "echo {} | base64 -d -w0 | openssl x509 -noout -dates -subject -issuer"
    
    # or
    $ while read -r crt; do
        echo "${crt}" | base64 -d -w0 | openssl x509 -noout -dates -subject -issuer;
      done < <(ldapsearch marslo userCertificate | awk '{print $NF}')
    notBefore=Apr 28 18:05:13 2023 GMT
    notAfter=Apr 27 18:05:13 2025 GMT
    subject=DC = com, DC = example, OU = Workers, CN = marslo, emailAddress = marslo@example.com
    issuer=DC = com, DC = example, CN = example SC Issuing CA V1
    
  • save local

    • der
      $ while read -r n c; do
          echo "-- ${n} --";
          echo "${c}" | base64 -d -w0 > cert_${n}.der;
        done < <(ldapsearch marslo userCertificate | awk '{print $NF}' | cat -n)
      
    • crt
      $ while read -r n c; do
          echo "-- ${n} --";
          echo "${c}" | base64 -d -w0 > cert_${n}.der;
          openssl x509 -in cert_${n}.der -inform DER -out cert_${n}.crt;
        done < <(ldapsearch marslo userCertificate | awk '{print $NF}' | cat -n)
      

others

directory diff

$ diff --suppress-common-lines -y <(cd path_to_dir1; find .|sort) <(cd path_to_dir2; find .|sort)

show some command periodically

$ watch --interval 1 ls -alt
  • watch with pipe
    $ watch -n 1 'ls -Altrh | grep <keywords>'
    

clear

$ printf "\ec"

use less as tail -f

$ less +F <filename>

netcat & nmap-ncat

[!NOTE] references:

  • install

    $ sudo yum -y install epel-release [yum-utils]
    # or via url
    $ sudo dnf [re]install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
    $ sudo dnf [re]install https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm
    
    $ sudo yum update -y
    $ yum list | grep netcat
    netcat.x86_64      1.219-2.el8       @epel
    $ sudo yum install -y netcat.x86_64
    
    # check
    $ sudo yum repolist
    
  • switch with nmap & netcat

    $ ls -altrh /usr/local/bin/nc
    lrwxrwxrwx 1 root root 22 Mar 14 03:14 /usr/local/bin/nc -> /etc/alternatives/nmap
    
    $ sudo update-alternatives --config nmap
    
    There are 2 programs which provide 'nmap'.
    
      Selection    Command
    -----------------------------------------------
    *+ 1           /usr/bin/ncat
       2           /usr/bin/netcat
    
    Enter to keep the current selection[+], or type selection number: 2
    
  • check

    # by using netcat
    $ nc -zv google.com 443
    Connection to google.com (142.251.214.142) 443 port [tcp/https] succeeded!
    
    # by using nact
    $ ncat -zv google.com 443
    Ncat: Version 7.70 ( https://nmap.org/ncat )
    Ncat: Connected to 142.251.214.142:443.
    Ncat: 0 bytes sent, 0 bytes received in 0.07 seconds.
    
  • check package

    $ rpm -ql netcat.x86_64
    /usr/bin/nc
    /usr/bin/netcat
    /usr/lib/.build-id
    /usr/lib/.build-id/f3
    /usr/lib/.build-id/f3/3de6290429f99a8d8f5fe646a93bcc952dafdd
    /usr/share/man/man1/nc.1.gz
    /usr/share/man/man1/netcat.1.gz
    
    $ rpm -ql nmap-ncat.x86_64
    /usr/bin/nc
    /usr/bin/ncat
    ...
    

alternatives & update-alternatives

  • install

    $ sudo update-alternatives --install /usr/bin/java java /opt/java/jdk1.8.0_121/bin/java 999
    $ sudo update-alternatives --auto java
    $ sudo update-alternatives --install /usr/bin/javac javac /opt/java/jdk1.8.0_121/bin/javac 999
    $ sudo update-alternatives --auto javac
    
  • modify

    $ ls -altrh $(which -a nc)
    lrwxrwxrwx 1 root root 22 Jun  1 04:02 /usr/bin/nc -> /etc/alternatives/nmap
    
    $ sudo alternatives --config nmap
    There are 2 programs which provide 'nmap'.
    
      Selection    Command
    -----------------------------------------------
    *+ 1           /usr/bin/netcat
       2           /usr/bin/ncat
    
    Enter to keep the current selection[+], or type selection number: 1
    
Copyright © marslo 2020-2023 all right reserved,powered by GitbookLast Modified: 2024-03-12 15:01:28

results matching ""

    No results matching ""