marfeel:test
Differences
This shows you the differences between two versions of the page.
Next revision | Previous revisionNext revisionBoth sides next revision | ||
marfeel:test [2020/03/16 12:11] – created dodger | marfeel:test [2020/03/16 15:28] – dodger | ||
---|---|---|---|
Line 7: | Line 7: | ||
* Particular answer to proxy cache: | * Particular answer to proxy cache: | ||
- | * I've configured nginx proxy_cache on /dev/shm for performance (on ram cache). 1 minute for proxy_cache should be fine if the content is changing constantly, cause nginx will " | + | * I've configured nginx proxy_cache on '' |
* Particular answer to static content: | * Particular answer to static content: | ||
* Static content to 1 hour maybe is aggressive, normally I set it up to 1day or more, also depends on the type of application. | * Static content to 1 hour maybe is aggressive, normally I set it up to 1day or more, also depends on the type of application. | ||
Line 16: | Line 16: | ||
===== Ami and UserData ===== | ===== Ami and UserData ===== | ||
I did some mistakes, so I create some versions of it: | I did some mistakes, so I create some versions of it: | ||
- | * name : dodger_marfeel_test_003 | ||
- | * id : ami-0bc1be25784321fc1 | ||
+ | | **name** | '' | ||
+ | | **id** | '' | ||
+ | |||
+ | \\ | ||
+ | ==== nginx ==== | ||
+ | I've configured nginx with a bare minimal setup: | ||
+ | <file config / | ||
+ | user www-data; | ||
+ | worker_processes auto; | ||
+ | pid / | ||
+ | events { | ||
+ | worker_connections 768; | ||
+ | } | ||
+ | http { | ||
+ | server_tokens off; | ||
+ | sendfile on; | ||
+ | tcp_nopush on; | ||
+ | tcp_nodelay on; | ||
+ | |||
+ | ## Start: Timeouts ## | ||
+ | client_body_timeout | ||
+ | client_header_timeout 10; | ||
+ | keepalive_timeout | ||
+ | send_timeout | ||
+ | ## End: Timeouts ## | ||
+ | |||
+ | types_hash_max_size 2048; | ||
+ | include / | ||
+ | default_type application/ | ||
+ | ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # Dropping SSLv3, ref: POODLE | ||
+ | ssl_prefer_server_ciphers on; | ||
+ | access_log / | ||
+ | error_log / | ||
+ | |||
+ | # compress everything | ||
+ | gzip on; | ||
+ | # disabled by marfeel request: | ||
+ | # | ||
+ | # you have chosen it :-) | ||
+ | # | ||
+ | |||
+ | #include / | ||
+ | include / | ||
+ | include / | ||
+ | } | ||
+ | </ | ||
+ | |||
+ | Nginx proxy cache setup: | ||
+ | <file config | ||
+ | proxy_cache_path /dev/shm levels=1:2 keys_zone=marfeel: | ||
+ | proxy_cache | ||
+ | # | ||
+ | proxy_cache_key $scheme$proxy_host$request_uri; | ||
+ | proxy_cache_revalidate | ||
+ | </ | ||
+ | |||
+ | Default site: | ||
+ | <file config / | ||
+ | server { | ||
+ | listen 80 default_server; | ||
+ | listen [::]:80 default_server; | ||
+ | #root / | ||
+ | root / | ||
+ | index index.html index.htm ; | ||
+ | server_name _; | ||
+ | |||
+ | # compress all proxy requests | ||
+ | include conf.d/ | ||
+ | include conf.d/ | ||
+ | |||
+ | location ^~ /cgi-bin { | ||
+ | proxy_pass http:// | ||
+ | proxy_set_header Host $http_host; | ||
+ | proxy_cache_valid any 1m; | ||
+ | expires 10m; | ||
+ | } | ||
+ | } | ||
+ | </ | ||
+ | |||
+ | Proxy compression config: | ||
+ | <file config / | ||
+ | # Enable gzip but do not remove ETag headers | ||
+ | gzip on; | ||
+ | gzip_vary on; | ||
+ | gzip_comp_level 4; | ||
+ | gzip_min_length 256; | ||
+ | gzip_proxied any; | ||
+ | #gzip_types application/ | ||
+ | gzip_types *; | ||
+ | </ | ||
+ | |||
+ | Static files '' | ||
+ | <file config / | ||
+ | # serve static files directly | ||
+ | # The ?: prefix is a ' | ||
+ | # the pattern to be captured into $1 which should help improve performance | ||
+ | location ~* ^.+\.(?: | ||
+ | # | ||
+ | expires | ||
+ | } | ||
+ | </ | ||
+ | |||
+ | |||
+ | ==== Python http server ==== | ||
+ | I've setup a '' | ||
+ | <file yaml / | ||
+ | [Unit] | ||
+ | Description=marfeel test service | ||
+ | After=auditd.service systemd-user-sessions.service time-sync.target | ||
+ | |||
+ | [Service] | ||
+ | User=marfeel | ||
+ | TimeoutStartSec=0 | ||
+ | Type=simple | ||
+ | KillMode=control-group | ||
+ | WorkingDirectory=/ | ||
+ | ExecStart=/ | ||
+ | Restart=no | ||
+ | |||
+ | [Install] | ||
+ | WantedBy=multi-user.target | ||
+ | </ | ||
===== Which code have you added to the user-data on launching the instance? ===== | ===== Which code have you added to the user-data on launching the instance? ===== | ||
<code yaml> | <code yaml> | ||
Line 33: | Line 153: | ||
===== Security group ===== | ===== Security group ===== | ||
I create the 001 group with https, then I remove it in the 2nd version: | I create the 001 group with https, then I remove it in the 2nd version: | ||
- | | + | | **name** | '' |
===== auto-scaling Group ===== | ===== auto-scaling Group ===== | ||
+ | | **name** | '' | ||
+ | ====== Load balancer ====== | ||
+ | |||
+ | ===== Dynamic set up ===== | ||
+ | |||
+ | Done as required with a very simple bash script.\\ | ||
+ | Dependencies for running the script: | ||
+ | * '' | ||
+ | * '' | ||
+ | |||
+ | |||
+ | <WRAP center round info 60%> | ||
+ | '' | ||
+ | </ | ||
+ | |||
+ | \\ | ||
+ | This script should be run by root. | ||
+ | |||
+ | Code: | ||
+ | <file bash / | ||
+ | #!/bin/bash | ||
+ | |||
+ | # Exit codes: | ||
+ | # 1 : | ||
+ | # 2 : | ||
+ | # 3 : | ||
+ | # 4 : | ||
+ | |||
+ | |||
+ | ######################################################################## | ||
+ | # INIT | ||
+ | ######################################################################## | ||
+ | CONFIGFILE=" | ||
+ | |||
+ | ######################################################################## | ||
+ | #/INIT | ||
+ | ######################################################################## | ||
+ | |||
+ | ######################################################################## | ||
+ | # | ||
+ | # CONSTANTS | ||
+ | # | ||
+ | ######################################################################## | ||
+ | |||
+ | # colors | ||
+ | BOLD=" | ||
+ | GREEN=" | ||
+ | LIGHTGREEN=" | ||
+ | RED=" | ||
+ | LIGHTRED=" | ||
+ | BLUE=" | ||
+ | LIGHTBLUE=" | ||
+ | YELLOW=" | ||
+ | LIGHTYELLOW=" | ||
+ | WHITE=" | ||
+ | RESET=" | ||
+ | |||
+ | NOW=" | ||
+ | |||
+ | |||
+ | ######################################################################## | ||
+ | # | ||
+ | # / CONSTANTS | ||
+ | # | ||
+ | ######################################################################## | ||
+ | |||
+ | |||
+ | |||
+ | |||
+ | ######################################################################## | ||
+ | # | ||
+ | # VARIABLES | ||
+ | # | ||
+ | ######################################################################## | ||
+ | |||
+ | |||
+ | SCRIPTLOG=" | ||
+ | SCRIPTLOGERR=" | ||
+ | |||
+ | TMPFILE=$(mktemp) | ||
+ | HADYNAMICCFG=$(mktemp) | ||
+ | |||
+ | ######################################################################## | ||
+ | # | ||
+ | # / VARIABLES | ||
+ | # | ||
+ | ######################################################################## | ||
+ | |||
+ | |||
+ | ######################################################################## | ||
+ | # | ||
+ | # FUNCTIONS | ||
+ | # | ||
+ | ######################################################################## | ||
+ | |||
+ | |||
+ | usage() | ||
+ | { | ||
+ | printf " | ||
+ | $0 | ||
+ | |||
+ | Read documentation here: | ||
+ | https:// | ||
+ | # VERY INITIAL CHECKS | ||
+ | } | ||
+ | |||
+ | printmsg() | ||
+ | { | ||
+ | echo -e " | ||
+ | } | ||
+ | |||
+ | output_log() | ||
+ | { | ||
+ | if [[ " | ||
+ | printmsg " | ||
+ | else | ||
+ | printmsg " | ||
+ | fi | ||
+ | } | ||
+ | |||
+ | abort_message() | ||
+ | { | ||
+ | printmsg " | ||
+ | exit 1 | ||
+ | } | ||
+ | |||
+ | # debug_me uses variable ${DEBUGME} | ||
+ | debug_me() | ||
+ | { | ||
+ | if [[ " | ||
+ | echo -e " | ||
+ | fi | ||
+ | } | ||
+ | |||
+ | |||
+ | ######################################################################## | ||
+ | # | ||
+ | # / FUNCTIONS | ||
+ | # | ||
+ | ######################################################################## | ||
+ | |||
+ | ######################################################################## | ||
+ | # | ||
+ | # MAIN | ||
+ | # | ||
+ | ######################################################################## | ||
+ | |||
+ | [[ ! -d $(dirname ${SCRIPTLOG}) ]] && mkdir -p $(dirname ${SCRIPTLOG}) | ||
+ | |||
+ | # DETECTING if the script is run by cron | ||
+ | if [[ " | ||
+ | set -x | ||
+ | exec > ${SCRIPTLOG} | ||
+ | exec 2> ${SCRIPTLOGERR} | ||
+ | fi | ||
+ | |||
+ | if [[ ${DEBUG} -eq 0 ]] ; then | ||
+ | echo -e " | ||
+ | echo -e "\t SCRIPTLOG will be ${SCRIPTLOG}" | ||
+ | echo -e "\t SCRIPTLOGERR will be ${SCRIPTLOGERR}" | ||
+ | set -x | ||
+ | exec 2> ${SCRIPTLOGERR} | ||
+ | fi | ||
+ | |||
+ | |||
+ | [[ ! -f ${CONFIGFILE} ]] && echo -e " | ||
+ | . ${CONFIGFILE} | ||
+ | |||
+ | [[ ${DEBUG} -eq 0 ]] && DEBUGME=" | ||
+ | |||
+ | ${AWSCLI} ec2 describe-instances --filters Name=instance-type, | ||
+ | let x=0 | ||
+ | for PRIVATEIP in $(cat ${TMPFILE} | ${JQ} ' | ||
+ | echo -e " | ||
+ | let x++ | ||
+ | done | ||
+ | |||
+ | if [[ $(cat ${HADYNAMICCFG} | wc -l) -ge ${MINBACKENDS} ]] ; then | ||
+ | # ok, overwriting config | ||
+ | echo -e "#### WARNING THIS CONFIG WILL BE REWRITTEN BY CRONJOB" | ||
+ | cat ${HAPROXYTEMPLATE} >> ${HAPROXYCONFIG} | ||
+ | cat ${HADYNAMICCFG} >> ${HAPROXYCONFIG} | ||
+ | echo -e "#### WARNING THIS CONFIG WILL BE REWRITTEN BY CRONJOB" | ||
+ | systemctl reload haproxy | ||
+ | fi | ||
+ | |||
+ | rm -f ${TMPFILE} | ||
+ | rm -f ${HADYNAMICCFG} | ||
+ | |||
+ | exit ${EXITCODE} | ||
+ | |||
+ | ######################################################################## | ||
+ | # | ||
+ | # / MAIN | ||
+ | # | ||
+ | ######################################################################## | ||
+ | |||
+ | </ | ||
+ | \\ | ||
+ | Config file: | ||
+ | <file config / | ||
+ | |||
+ | # Our template for re-generate the config | ||
+ | HAPROXYTEMPLATE=/ | ||
+ | # The configfile itself | ||
+ | HAPROXYCONFIG=/ | ||
+ | # Minimum amount of backends that must be running, less than this, the config file won't be changed | ||
+ | MINBACKENDS=2 | ||
+ | |||
+ | # Tag for the scaling group (backend servers will be filtered by this tag) | ||
+ | SCALINGTAG=" | ||
+ | |||
+ | # generic setup | ||
+ | AWSCLI=/ | ||
+ | JQ=/ | ||
+ | |||
+ | # SET to 0 for DEBUG | ||
+ | DEBUG=0 | ||
+ | </ | ||
+ | |||
+ | \\ | ||
+ | Setup crontab with desired frequency for refresh, for example: | ||
+ | <code crontab> | ||
+ | */3 * * * * | ||
+ | </ | ||