Hello!
Recently we published guides how to push WordPress sites with Jenkins or how to push WordPress sites with a simple shell script.
We thought it might be useful to give an overview of how to streamline your code integration process with Jenkins , GitHub and Bash shell scripting. The script I will outline below is definitely a starting point. There are of course efficiencies that could be made within this script, or perhaps that could be found with porting the logic within the script to Python.
The idea behind this process is to automate and streamline code integration across your local development environment and potentially to a “staging” server where unit and other tests can be performed. Lastly a “production” push is included in this script. The strategy is to create a separate git branch for staging and production and to either push code directly or create a pull request from staging to production. Either pushing or creating a pull request will trigger a Jenkins webhook that will process the request and pass the variables, such as which branch you pushed, over to the shell script which I will detail below.
Set up Jenkins with your Laravel Git Repository
In your Jenkins project, you want to configure the repository that Jenkins will be monitoring under the “Source code management” section. You simply enter the repository URL as well as the pre-defined credentials that should be created to allow Jenkins to access the project. This is especially true if your repository is private or on a service like Bitbucket.
Further to the repository configuration, you can also see in the screenshot above that we are monitoring two branches : staging and production. This works for our development cycle, but really you can define whatever works best for you here.
Set up a build trigger in Jenkins to push your Laravel site
In the “Build” section, the key element is that we are triggering our shell script “jenkins-laravel.sh” with two arguments : project name and the Jenkins variable GIT_BRANCH When a commit is actually made, the GIT_BRANCH variable will be whatever the branch is, such as “origin/production”. Since jenkins is only monitoring two branches, this trigger simply wont happen if you dont push to the configured branches within Jenkins.
Configuration for your Jenkins Laravel push script
Before we get into the push script itself, we want to define a “config” file that the script will read. This will be useful if you have multiple Laravel projects that you want to integrate Jenkins and this script with.
#!/bin/sh
#check command input
if [ "$#" -ne 2 ];
then
echo "JENKINS LARAVEL PUSH"
echo "--------------------"
echo ""
echo "Usage : ./jenkins-laravel.sh project-name"
echo ""
exit 1
fi
# Declare variables
currentdate=`date "+%Y-%m-%d"`
scriptpath="/usr/local/bin/jenkins"
destination_project="$1"
destination_branch=`echo "$2" | awk -F "/" '{printf "%s", $2}'`
# Get configuration variables
source ${scriptpath}/config/laravel/${destination_project}.conf
echo "Pushing to $destination_branch .. "
You can see on the second last line in the above snippet of code that we are executing a source command against the configuration file for the project. What the source command will do is load all the variables defined in the project configuration file. You can see the layout of the configuration file below :
#GLOBAL item_rootdir="/var/lib/jenkins/jobs/your-project/workspace" alert_email="alerts@yourdomain.com" user_perm="uuser" group_perm="nginx" # STAGING dest_user_staging="staginguser" dest_host_staging="0.0.0.0" dest_dir_staging="https://dmgbuscdkh46d.cloudfront.net/usr/share/nginx/html/staging.yoursite.com" # PRODUCTION git_repo="git@github.org:whatever/whatever.git" dest_user_prod="produser" dest_host_prod="0.0.0.0" dest_dir_prod="/data/www/your-project" dest_dir_root="/data/www/your-project" gen_docs_prod="FALSE" pre_prod="/home/prod-push/sfs"
Most of the variables are self explanatory. Basically we are defining paths for folders for the staging and production environments. We are also defining users that we will be connecting as in order to execute commands over SSH on the staging and production servers.
Most of the executions in this file will be done over SSH. Going back to my efficiency comment at the beginning of this post, there is definitely more efficient ways of accomplishing the same thing with things like Python as an alternative. Bash is chosen for this exercise because it is simple and has been tested as reliable and stable for the tasks at hand. The key thing to remember, which will be explained in further detail below, is that for some key elements of the production push we will want to check and make sure there are no errors. We want the script to alert us if npm run production fails. Most importantly we want to halt the production push process if any key element fails prior to actually launching the code.
Common functions for alerting and sanity checks
Before we get into the staging / production push elements of this script, we want to define two common functions that will be used multiple times. One will be an alert notification function that uses the mail command to send an alert email and the next one will be a sanity_check function that will check the error runlevel of the previously run command in order to determine if it exited cleanly or if there was an error in execution.
# Declare functions
alert_notification() {
echo "Push script failure : $2" | mail -s "Push script Failure" $1
}
sanity_check() {
if [ $1 -ne 0 ]
then
echo "$2"
alert_notification $alert_email "$2"
exit 1
fi
}
Push to staging and run unit tests for Laravel with Jenkins
For the staging push portion of our Jenkins script, it is fairly simple. We want to reset the repo, fetch and pull changes from the remote branch, then run a bunch of artisan commands to migrate and clear caches. Next we want to install any newly added npm modules and run npm run dev to rebuild the assets. Lastly we want to run the phpunit tests on staging and save the results in a directory that we will use later.
################
# STAGING PUSH #
################
if [ "$destination_branch" == "staging" ]
then
destination_user="$dest_user_staging"
destination_host="$dest_host_staging"
destination_dir="$dest_dir_staging"
# Push command over ssh
ssh -l $destination_user $destination_host \
"cd $destination_dir;\
rm -rf composer.lock;\
git reset --hard;\
git fetch --all;\
git checkout -f $destination_branch;\
git reset --hard;\
git fetch --all;\
git pull origin $destination_branch;\
/usr/local/bin/composer update --no-interaction --prefer-dist --optimize-autoloader;\
php artisan clear-compiled;\
php artisan migrate --force;\
php artisan cache:clear;\
php artisan route:clear;\
php artisan view:clear;\
php artisan config:clear;\
php artisan config:cache;\
npm i;\
npm run dev;\
php artisan config:clear;\
/usr/bin/php ./vendor/bin/phpunit --log-junit ${destination_dir}/tests/results/${destination_project}_test1.xml"
# Get test results
ssh -l $destination_user $destination_host \
"cat ${destination_dir}/tests/results/${destination_project}_test1.xml" > ${item_rootdir}/tests/results/${destination_project}_test1.xml
Production push for your Laravel project with Jenkins
This is the most interesting portion of our script! It took a lot of testing to get this right, but once its working and fine tuned to your environment, it will save you time and headaches.
Typically to push to production we will be creating a pull request on Git from staging to production. This will keep a clear and concise audit trail of what was pushed from staging as well as keeping in line with the QA processes for unit and other testing the code by the time it reaches production.
###################
# PRODUCTION PUSH #
###################
elif [ "$destination_branch" == "production" ]
then
destination_user="$dest_user_prod"
destination_host="$dest_host_prod"
destination_dir="$dest_dir_prod"
pre_prod_dir="$pre_prod"
# Get current latest commit running on prod
current_local_commit=`ssh -l $destination_user $destination_host "cd $destination_dir;git rev-parse --short HEAD"`
current_remote_commit=`ssh -l $destination_user $destination_host "cd $destination_dir;git rev-parse --short origin/${destination_branch} "`
# Make sure local and remote arent the same because then theres no reason to push
if [ "$current_local_commit" == "$current_remote_commit" ]
then
alert_msg="Remote HEAD : $current_remote_commit matches Local HEAD : $current_local_commit, exiting..."
echo "$alert_msg"
alert_notification $alert_email "$alert_msg"
exit 1
fi
The only thing to note so far before we delve further with the above snippet is that before the push even starts, we check to see if the local commit matches what is on the remote branch. If they are the same commit, we dont bother going any further. We dont need to push if they are the same.
# Prep the pre prod folder
check_clear_folder=`ssh -l $destination_user $destination_host "rm -rf $pre_prod_dir"`
sanity_check $? "Error with cleaning pre prod folder : $check_clear_folder"
# Clone files from the repo in prod prep folder, set permissions and rsync files from live site
ssh -l $destination_user $destination_host \
"mkdir $pre_prod_dir &&\
cd $pre_prod_dir &&\
git clone $git_repo . &&\
rsync --ignore-existing -razp --progress --exclude '.git' --exclude '.npm' --exclude 'node_modules' --exclude 'vendor' --exclude '.cache' ${destination_dir}/ ${pre_prod_dir} &&\
chown -R ${user_perm}:${group_perm} ${pre_prod_dir}"
# Sanity checks
check_composer_update=`ssh -l $destination_user $destination_host "cd $pre_prod_dir;/usr/local/bin/composer update --no-interaction --prefer-dist --optimize-autoloader"`
sanity_check $? "Error with composer update on production : $check_composer_update"
# Sanity checks before actually pushing live
check_npm_install=`ssh -l $destination_user $destination_host "cd $pre_prod_dir;npm i"`
sanity_check $? "Error with NPM install pacakge on production : $check_npm_install"
check_npm_run=`ssh -l $destination_user $destination_host "cd $pre_prod_dir;npm run dev"`
sanity_check $? "Error with NPM run dev on production : $check_npm_run"
check_move_preprod=`ssh -l $destination_user $destination_host "mv $pre_prod_dir ${destination_dir}_${current_remote_commit}"`
sanity_check $? "Error with moving pre-prod folder to cluster folder : $check_move_preprod"
ssh -l $destination_user $destination_host \
"cd ${destination_dir}_${current_remote_commit};\
$gen_docs_cmd;\
php artisan clear-compiled;\
php artisan cache:clear;\
php artisan route:clear;\
php artisan view:clear;\
php artisan config:clear;\
php artisan config:cache"
check_force_symlink=`ssh -l $destination_user $destination_host "ln -sf ${destination_dir}_${current_remote_commit} ${$destination_dir}"`
sanity_check $? "Error with creating symlink to newly pushed folder : $check_force_symlink"
# Remove all folders except the current and previous commit folders as well as the symlink
ssh -l $destination_user $destination_host \
"find ${dest_dir_root} -type d -not \( -name '${destination_dir}' -or -name '${destination_dir}_{$current_remote_commit}' -or -name '${destination_dir}_${current_local_commit}' \) -delete"
# We dont run unit tests on production
echo "" > ${item_rootdir}/tests/results/${destination_project}_test1.xml
echo "" > ${item_rootdir}/tests/results/${destination_project}_test2.xml
The remainder of the script above goes through the following steps :
- Prep a “pre production” folder outside of the current running live site production folder by clearing out and re-creating the folder
- git clone the production branch into the pre production folder
- run composer update on the pre production folder, as well as npm install and npm run production
Everything in step 3 is run against the sanity_check function detailed above. This is so that if any of those key steps fail, we missed a problem or something caused the process to fail. This will generate an alert email and the push process will halt without any changes to production being executed.
The next part of the production push process, if those items detailed above run without errors, would be to clear out all the laravel caches :
ssh -l $destination_user $destination_host \
"cd ${destination_dir}_${current_remote_commit};\
php artisan clear-compiled;\
php artisan cache:clear;\
php artisan route:clear;\
php artisan view:clear;\
php artisan config:clear;\
php artisan config:cache"
One of the last steps, after the above is executed, would be to move the pre production folder to the live site folder and switch the symlink to point to the new folder. Switching a symlink in this scenario is a much quicker and safer method for “launching” the new code. If there are problems at this point you can manually fix the symlink to point to the old folder.
check_force_symlink=`ssh -l $destination_user $destination_host "ln -sf ${destination_dir}_${current_remote_commit} ${$destination_dir}"`
sanity_check $? "Error with creating symlink to newly pushed folder : $check_force_symlink"
One of the last steps of the script is to clear out any previous backup folders that are older than the recently replaced folder. Each time this production push script runs it creates a new folder in your destination directory with the naming convention of projectname_commit-hash. We keep the previous project folder and the newly pushed project folder. Anything older will be cleared out with the following command :
# Remove all folders except the current and previous commit folders as well as the symlink
ssh -l $destination_user $destination_host \
"find ${dest_dir_root} -type d -not \( -name '${destination_dir}' -or -name '${destination_dir}_{$current_remote_commit}' -or -name '${destination_dir}_${current_local_commit}' \) -delete"
Thats about it! I hope this was helpful! This script can be fine tuned to accommodate different environments, additional repository branches and other procedural things that might suit your specific requirements. Our script is around 157 lines total. You can find a Github link to the full Jenkins Laravel push script here.