Strapi media gallery syncing with a JAM site

The web has changed we no longer have the admin and website on the same server with the images sitting on a CDN. We don't even use servers anymore everything just lives in a serverless environment this leads to several granular issues one of which we had to solve recently at cryptoskilz HQ and thought we would share our solution with everyone.

We used to have a pretty standard setup:

  • hosting on S3 of the static frontend websites
  • custom coded cms on an EC2 box
  • code in Github
  • deployment via Codeship
  • assets in a CloudFront CDN
  • DNS managed by Cloudflare.

Firstly we replaced the backend withStrapiand upgraded the frontend to a JAM stack usingeleventyafter this we moved the hosting toCloudflare pages. This has simplified our setup a lot, no more AWS, no more Codeship just Github,Herokuto host Strapi, and Cloudflare to host the static site.

This worked great until we published our first blog post and realized that the images we used in the post were kept in the uploads directory for Strapi which lead to a problem as eleventy did not have access to this uploads folder. To fix this issue we did it in a couple of ways

Attempt 1

Our first attempt was to modify our build script with the following code

copyImages () {
    #mkdir -p output/templates/_includes
    cp ../cms/public/uploads/* output/HTML/assets/images/uploads
 }

This copies all the images from the Strapi uploads folder to the assets images folder. This is a solution that works great for the local deployment where the admin and frontend are both running locally but it does not work in production as the admin is on Heroku and the frontend is on Cloudflare pages so we required another solution.

Attempt 2

Copying the files using the API.

The first thing we have to do is expose the files via the Strapi API and you do this in the roles and permission settings as shown in the image below.

Screenshot from 2021-06-23 10.21.29.png

next in eleventy, in the data directory we add files to deal with data globally this is what we are going to use to connect to the Strapi CMS. Create a file called API.js and add the following code. Of course, tweak this to your purpose.

//API Data preprocessing can happen here 

const superagent = require('superagent');
let env = require('./env')


let getImages = async () => {
    console.log('getting images')
    try {
        var res = await superagent.get(env.API_URL+ "/upload/files/").query({})
        //check we got a response.
        if (res.ok) {
            let images = res.body;
            for (var i = 0; i < images.length; i++) {
                //console.log(images[i].url);
                var http = require('http')     
                var https = require('https');
                var usehttpmethod = 0; // default to http                                          
                Stream = require('stream').Transform                               
                fs = require('fs');                                                    
                let url;
                let imagename;


                if(images[i].url.indexOf("https://cryptoskillz.com/assets/images/uploads/undefined) >= 0) 
                {
                    //console.log('found')
                    usehttpmethod =1;
                    url = images[i].url  
                    imagenamearr = images[i].url.split("/");
                    //console.log(imagenamearr[7])
                    imagename = imagenamearr[7]
                }
                else
                {
                    //console.log('not found')
                    usehttpmethod =0;
                    url = env.API_URL+images[i].url
                    imagename = images[i].url;
                }
                
                //console.log(url)   
                //console.log(imagename)
                if (usehttpmethod == 1)
                {

                    https.request(url, function(response) {                                        
                      var data = new Stream();                                                    

                      response.on('data', function(chunk) {                                       
                        data.push(chunk);                                                         
                      });                                                                         

                      response.on('end', function() {     
                        //console.log('hhh output/HTML/assets/images/uploads/'+imagename )                                        
                        fs.writeFileSync('output/HTML/assets/images/uploads/'+imagename   , data.read());                               
                      });                                                                         
                    }).end();
                }
                else
                {
                    http.request(url, function(response) {                                        
                        var data = new Stream();                                                    

                        response.on('data', function(chunk) {                                       
                            data.push(chunk);                                                         
                        });                                                                         

                        response.on('end', function() {     
                            //console.log('hhh output/HTML/assets/images'+imagename )                                        
                            fs.writeFileSync('output/HTML/assets/images'+imagename   , data.read());                               
                        });                                                                         
                    }).end();
                }
            }
        }
    } catch (err) {
        console.log('Error getting something:')
        console.error(err)
    }
}

let getSomething = async () => {
    console.log('getting articles')
    try {
        var res = await superagent.get(env.API_URL+ "/blog-articles/").query({})
        //console.log(res.body)
        //check we got a response.
        if (res.ok) {
            let articles = res.body;
            
            for (var i = 0; i < articles.length; i++) {
                const regex = /upload\/(.*?)\)/gm;
                const str = articles[i].content
                let m;
                //we can clean this regex up but it is good enough for now
                while ((m = regex.exec(str)) !== null) {
                    // This is necessary to avoid infinite loops with zero-width matches
                    if (m.index === regex.lastIndex) {
                        regex.lastIndex++;
                    }
    
                    // The result can be accessed through the `m`-variable, we can clean th
                    m.forEach((match, groupIndex) => {
                        let tmp = "https://res.cloudinary.com/dfesv3sqc/image/upload/";
                        if (groupIndex == 1)
                        {
                            let tmpnamearr = match.split("/");
                            let tmpimagename = tmpnamearr[1]
                            //console.log(tmpimagename)
                            articles[i].content = articles[i].content.replace(tmp+match,env.ROOT_URL+"/assets/images/uploads/"+tmpimagename)
                            //console.log(tmp2)
                        }
                        //console.log(`Found match, group ${groupIndex}: ${match}`);
                    });

                }
            }
            return res.body;
        }
    } catch (err) {
        console.log('Error getting something:')
        console.error(err)
    }
}


module.exports = async () => {
    let functionResults = [];
    if (functionResults.length === 0) functionResults = await getSomething();

    getImages();

    //console.log(functionResults)
    return {
        articlesArray: functionResults
    }
    
}

You can view the codehereand read more about how it workshere

this gets a list of files from the API URLs and downloads the images to the assets directory of our eleventy app which we just pass through during the build phase.

Then we ran into one final issue as we used Heroku to host our CMS then we cannot use the uploads dir to store the images as it is not ephemeral in this case we used a 3rd party to store our images. We usedCloudinaryand you can set this up very easily using the first part of thistutorial, we don't use next or graphQL so we ignored that part. However, this begs the question if the image is now on a 3rd party and the CMS links directly to it why bother with the remaining steps?

It is a good question and the reason is the speed we like to remove every bottleneck and seeing as our entire site lives in a CDN it makes little sense to access another to pull down the images so (for us anyway) we want all the files to be as close to one another as possible.

If you view any of the images in this blog post you will see they are being loaded from our assets/images/uploads but they live in Cloudinary and are managed with our Strapi CMS instance.

2de1994d25f287b4b8720cc3b66da53e.png

You can view the source could along with the eleventy files, build scripts, etc from our git repohere