Ciberterminal Wiki

User Tools

Site Tools

Trace:
marfeel:test

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Next revision		Previous revision
Last revisionBoth sides next revision
marfeel:test [2020/03/16 12:11] – created dodgermarfeel:test [2020/03/17 07:58] – [Things I forgot] dodger
Line 7: Line 7:
  
   * Particular answer to proxy cache:   * Particular answer to proxy cache:
-    * I've configured nginx proxy_cache on /dev/shm for performance (on ram cache). 1 minute for proxy_cache should be fine if the content is changing constantly, cause nginx will "rebuild/compile" the cache after 1 minute but inside that minute, it will use the already compiled version of the file.+    * I've configured nginx proxy_cache on ''/dev/shm'' for performance (on ram cache). 1 minute for proxy_cache should be fine if the content is changing constantly, cause nginx will "rebuild/compile" the cache after 1 minute but inside that minute, it will use the already compiled version of the file.
   * Particular answer to static content:   * Particular answer to static content:
     * Static content to 1 hour maybe is aggressive, normally I set it up to 1day or more, also depends on the type of application.     * Static content to 1 hour maybe is aggressive, normally I set it up to 1day or more, also depends on the type of application.
Line 16: Line 16:
 ===== Ami and UserData ===== ===== Ami and UserData =====
 I did some mistakes, so I create some versions of it: I did some mistakes, so I create some versions of it:
-  * name : dodger_marfeel_test_003 
-  * id : ami-0bc1be25784321fc1 
  
 +| **name** | ''dodger_marfeel_test_003'' | 
 +| **id** | ''ami-0bc1be25784321fc1'' |
 +
 +\\
 +==== nginx ====
 +I've configured nginx with a bare minimal setup:
 +<file config /etc/nginx/nginx.conf>
 +user www-data;
 +worker_processes auto;
 +pid /run/nginx.pid;
 +events {
 +        worker_connections 768;
 +}
 +http {
 +        server_tokens off;
 +        sendfile on;
 +        tcp_nopush on;
 +        tcp_nodelay on;
 +
 +        ## Start: Timeouts ##
 +        client_body_timeout   10;
 +        client_header_timeout 10;
 +        keepalive_timeout     5 5;
 +        send_timeout          10;
 +        ## End: Timeouts ##
 +
 +        types_hash_max_size 2048;
 +        include /etc/nginx/mime.types;
 +        default_type application/octet-stream;
 +        ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # Dropping SSLv3, ref: POODLE
 +        ssl_prefer_server_ciphers on;
 +        access_log /var/log/nginx/access.log;
 +        error_log /var/log/nginx/error.log;
 +
 +        # compress everything
 +        gzip on;
 +        # disabled by marfeel request:
 +        #       Enable gzip for all the requests (proxied request included)
 +        #       you have chosen it :-)
 +        #gzip_disable "msie6";
 +
 +        #include /etc/nginx/conf.d/*.conf;
 +        include /etc/nginx/conf.d/proxy_cache.conf;
 +        include /etc/nginx/sites-enabled/*;
 +}
 +</file>
 +
 +Nginx proxy cache setup:
 +<file config  /etc/nginx/conf.d/proxy_cache.conf>
 +proxy_cache_path /dev/shm levels=1:2 keys_zone=marfeel:1m;
 +proxy_cache                     marfeel;
 +#proxy_cache_background_update  on;
 +proxy_cache_key $scheme$proxy_host$request_uri;
 +proxy_cache_revalidate          on;
 +</file>
 +
 +Default site:
 +<file config /etc/nginx/sites-available/default.conf>
 +server {
 +        listen 80 default_server;
 +        listen [::]:80 default_server;
 +        #root /opt/test/www;
 +        root /opt/test/Marfeel-appserverpythontestapp-2937d4f8673c;
 +        index index.html index.htm ;
 +        server_name _;
 +
 +        # compress all proxy requests
 +        include conf.d/proxy_compression.conf;
 +        include conf.d/static_files.conf;
 +
 +        location ^~ /cgi-bin {
 +                proxy_pass http://localhost:8080;
 +                proxy_set_header Host $http_host;
 +                proxy_cache_valid any      1m;
 +                expires 10m;
 +        }
 +}
 +</file>
 +
 +Proxy compression config:
 +<file config /etc/nginx/conf.d/proxy_compression.conf>
 +# Enable gzip but do not remove ETag headers
 +gzip on;
 +gzip_vary on;
 +gzip_comp_level 4;
 +gzip_min_length 256;
 +gzip_proxied any;
 +#gzip_types application/atom+xml application/javascript application/json application/ld+json application/manifest+json application/rss+xml application/vnd.geo+json application/vnd.ms-fontobject application/x-font-ttf application/x-web-app-manifest+json application/xhtml+xml application/xml font/opentype image/bmp image/svg+xml image/x-icon text/cache-manifest text/css text/plain text/vcard text/vnd.rim.location.xloc text/vtt text/x-component text/x-cross-domain-policy;
 +gzip_types *;
 +</file>
 +
 +Static files ''expires'':
 +<file config /etc/nginx/conf.d/static_files.conf>
 +# serve static files directly
 +# The ?: prefix is a 'non-capturing' mark, meaning we do not require
 +# the pattern to be captured into $1 which should help improve performance
 +location ~* ^.+\.(?:jpg|jpeg|gif|css|png|js|ico|txt)$ {
 +    #access_log        off;
 +    expires           1h;
 +}
 +</file>
 +
 +
 +==== Python http server ====
 +I've setup a ''.service'' for systemd named ''marfeel_test'' which is inside the AMI:
 +<file yaml /etc/systemd/system/marfeel_test.service>
 +[Unit]
 +Description=marfeel test service
 +After=auditd.service systemd-user-sessions.service time-sync.target
 + 
 +[Service]
 +User=marfeel
 +TimeoutStartSec=0
 +Type=simple
 +KillMode=control-group
 +WorkingDirectory=/opt/test/Marfeel-appserverpythontestapp-2937d4f8673c
 +ExecStart=/usr/bin/python3 -m http.server --cgi 8080
 +Restart=no
 + 
 +[Install]
 +WantedBy=multi-user.target
 +</file>
 ===== Which code have you added to the user-data on launching the instance? ===== ===== Which code have you added to the user-data on launching the instance? =====
 <code yaml> <code yaml>
Line 32: Line 152:
  
 ===== Security group ===== ===== Security group =====
-I create the 001 group with https, then I remove it in the 2nd version: +I create a security group (ending with 001) with https, then I remove it in the 2nd version: 
-  * name: dodger_launch_002+| **name** | ''dodger_launch_002'' |
  
 ===== auto-scaling Group ===== ===== auto-scaling Group =====
 +
 +| **name** | ''marfeel_scalinggroup_001'' |
 +
 +
 +====== Load balancer ======
 +
 +===== Dynamic set up =====
 +
 +Done as required with a very simple bash script.\\
 +Dependencies for running the script:
 +  * ''aws-cli''
 +  * ''jq''
 +
 +
 +<WRAP center round info 60%>
 +''aws-cli'' must be configured!!!
 +</WRAP>
 +
 +\\
 +This script should be run by root.
 +
 +Code:
 +<file bash /root/marfeel_auto_updater/make_it_cool.sh>
 +#!/bin/bash
 +
 +# Exit codes:
 +#  1 :
 +#  2 :
 +#  3 :
 +#  4 :
 +
 +
 +########################################################################
 +# INIT
 +########################################################################
 +CONFIGFILE="$(dirname $(readlink -f $0))/$(basename $(readlink -f $0) .sh).config"
 +
 +########################################################################
 +#/INIT
 +########################################################################
 +
 +########################################################################
 +#
 +# CONSTANTS
 +#
 +########################################################################
 +
 +# colors
 +BOLD="\e[1m"
 +GREEN="\e[32m"
 +LIGHTGREEN="${BOLD}${GREEN}"
 +RED="\033[1;31m"
 +LIGHTRED="\033[1;31m"
 +BLUE="\e[34m"
 +LIGHTBLUE="${BOLD}${BLUE}"
 +YELLOW="\e[33m"
 +LIGHTYELLOW="${BOLD}${YELLOW}"
 +WHITE="\033[0;37m"
 +RESET="\033[0;00m"
 +
 +NOW="$(date +%Y%m%d%H%M%S)"
 +
 +
 +########################################################################
 +#
 +# / CONSTANTS
 +#
 +########################################################################
 +
 +
 +
 +
 +########################################################################
 +#
 +# VARIABLES
 +#
 +########################################################################
 +
 +
 +SCRIPTLOG="$(dirname `readlink -f $0`)/logs/$(basename $0 .sh)_script_${NOW}.log"
 +SCRIPTLOGERR="$(dirname `readlink -f $0`)/logs/$(basename $0 .sh)_script_${NOW}.err"
 +
 +TMPFILE=$(mktemp)
 +HADYNAMICCFG=$(mktemp)
 +
 +########################################################################
 +#
 +# / VARIABLES
 +#
 +########################################################################
 +
 +
 +########################################################################                                                                                                                                [49/5875]
 +#                                                                                                                                                                                                                
 +# FUNCTIONS
 +#
 +########################################################################
 +
 +
 +usage()
 +{
 +    printf "%s${LIGHTRED}USAGE:${RESET}
 +    $0
 +     
 + Read documentation here:
 + https://wiki.ciberterminal.net/doku.php?id=marfeel:test#dynamic_set_up"
 +    # VERY INITIAL CHECKS
 +}
 +
 +printmsg()
 +{
 +   echo -e "$*"
 +}
 +
 +output_log()
 +{
 +    if [[ "${QUIETOUTPUT}" == true ]]; then
 +        printmsg "$*" >> ${OUTPUTFILE}
 +    else
 +        printmsg "$*" | tee -a "${OUTPUTFILE}"
 +    fi
 +}
 +
 +abort_message()
 +{
 +    printmsg "ERROR: $*"
 +    exit 1
 +}
 +
 +# debug_me uses variable ${DEBUGME}
 +debug_me()
 +{
 +    if [[ "${DEBUGME}" && ${DEBUGME} -eq 0 ]] ; then 
 +        echo -e "${LIGHTBLUE}DEBUG: ${RESET}$*"
 +    fi
 +}
 +
 +
 +########################################################################
 +#
 +# / FUNCTIONS
 +#
 +########################################################################
 +
 +########################################################################
 +#
 +# MAIN
 +#
 +########################################################################
 +
 +[[ ! -d $(dirname ${SCRIPTLOG}) ]] && mkdir -p $(dirname ${SCRIPTLOG})
 +
 +# DETECTING if the script is run by cron
 +if [[ "$(tty)" = "not a tty" ]] ; then
 + set -x
 + exec > ${SCRIPTLOG}
 + exec 2> ${SCRIPTLOGERR}
 +fi
 +
 +if [[ ${DEBUG} -eq 0 ]] ; then
 +    echo -e "${BLUE}DEBUGMODE${RESET} is on"
 +    echo -e "\t SCRIPTLOG will be ${SCRIPTLOG}"
 +    echo -e "\t SCRIPTLOGERR will be ${SCRIPTLOGERR}"
 +    set -x
 +    exec 2> ${SCRIPTLOGERR}
 +fi
 +
 +
 +[[ ! -f ${CONFIGFILE} ]] && echo -e "${LIGHTRED} CONFIGFILE ${CONFIGFILE} NOT FOUND${RESET}" && exit 1
 +. ${CONFIGFILE}
 +
 +[[ ${DEBUG} -eq 0 ]] && DEBUGME="bash -x"
 +
 +${AWSCLI} ec2 describe-instances --filters Name=instance-type,Values=t2.nano Name=instance-state-name,Values=running > ${TMPFILE}
 +let x=0
 +for PRIVATEIP in $(cat ${TMPFILE} | ${JQ} '.Reservations[] | .Instances[] | "\(.PrivateIpAddress) \(.PublicIpAddress) \(.Tags)"' | egrep "${SCALINGTAG}" | awk 'BEGIN{FS="[ \"]"}{print $2}') ; do
 + echo -e "\tserver marfeel_nginx_${x} ${PRIVATEIP}:80 check maxconn 512" >> ${HADYNAMICCFG}
 + let x++
 +done
 +
 +if [[ $(cat ${HADYNAMICCFG} | wc -l) -ge ${MINBACKENDS} ]] ; then
 + # ok, overwriting config
 + echo -e "#### WARNING THIS CONFIG WILL BE REWRITTEN BY CRONJOB" > ${HAPROXYCONFIG}
 + cat ${HAPROXYTEMPLATE} >> ${HAPROXYCONFIG}
 + cat ${HADYNAMICCFG} >> ${HAPROXYCONFIG}
 + echo -e "#### WARNING THIS CONFIG WILL BE REWRITTEN BY CRONJOB" >> ${HAPROXYCONFIG}
 + systemctl reload haproxy
 +fi
 +
 +rm -f ${TMPFILE}
 +rm -f ${HADYNAMICCFG}
 +
 +exit ${EXITCODE}
 +
 +########################################################################
 +#
 +# / MAIN
 +#
 +########################################################################
 +
 +</file>
 +\\
 +Config file:
 +<file config /root/marfeel_auto_updater/make_it_cool.config>
 +
 +# Our template for re-generate the config
 +HAPROXYTEMPLATE=/etc/haproxy/haproxy.cfg.TMPL
 +# The configfile itself
 +HAPROXYCONFIG=/etc/haproxy/haproxy.cfg
 +# Minimum amount of backends that must be running, less than this, the config file won't be changed
 +MINBACKENDS=2
 +
 +# Tag for the scaling group (backend servers will be filtered by this tag)
 +SCALINGTAG="marfeel_scalinggroup_001"
 +
 +# generic setup
 +AWSCLI=/usr/bin/aws
 +JQ=/usr/bin/jq
 +
 +# SET to 0 for DEBUG
 +DEBUG=0
 +</file>
 +
 +\\
 +Setup crontab with desired frequency for refresh, for example:
 +<code crontab>
 +*/3 *   * * *   root    /root/marfeel_auto_updater/make_it_cool.sh
 +</code>
  
  
 +====== Things I forgot ======
  
 +  * Purge log files from ''make_it_cool.sh'' in HA proxy node
 +  * A lot of additional checks for the script...